When we published the Agenda for Open Science in Communication roughly one year ago, I expected to receive pushback. After all, we called for a sweeping change of several well-established research practices. Transparency instead of confidentiality, research openly accessible but not hidden behind paywalls, showing instead of telling, quality instead of quantity.
I expected to receive pushback coming from the established quantitative scholars. From the p-hackers, the grinders, the paper manufacturers, the data-dredgers, the beneficiaries of the old and closed system. However, I didn’t expect to receive pushback from the more qualitative-oriented scholars.1
Well, turns out I was wrong.
Background
The Journal of Communication is currently in the process of publishing a special issue on Open Science, of which our agenda was the opening piece. Last Sunday, the first (indirect) response was published by JOC, titled Open Science, Closed Doors? Countering Marginalization through an Agenda for Ethical, Inclusive Research in Communication.
Let’s call a spade a spade: The piece is very critical of open science in general and our agenda piece in particular. Not to misrepresent, here’s the abstract:
The open science (OS) movement has advocated for increased transparency in certain aspects of research. Communication is taking its first steps toward OS as some journals have adopted OS guidelines codified by another discipline. We find this pursuit troubling as OS prioritizes openness while insufficiently addressing essential ethical principles: respect for persons, beneficence, and justice. Some recommended open science practices increase the potential for harm for marginalized participants, communities, and researchers. We elaborate how OS can serve a marginalizing force within academia and the research community, as it overlooks the needs of marginalized scholars and excludes some forms of scholarship. We challenge the current instantiation of OS and propose a divergent agenda for the future of Communication research centered on ethical, inclusive research practices.
Abstract of Fox et al. (2021)
Below some more strident quotes from the paper:
„Collectively, OS’s gamification reflects the broader cultural values of its creators (competition, capitalism) and is designed to reward similar researchers (see Dutta, 2020).
„Although OS considers diversifying samples less effective for conducting replications, researchers should recognize the value to scientific and societal progress.“
„Samples will be less representative, data will be more biased, and scientific findings will be diminished in both accuracy and generalizability.“
How to react?
Let’s be honest, reading the paper was challenging. I could not have a more different view on the motivations behind open science and on the effects to which open science likely leads. I believe the paper contains several straw men and that the argumentation is almost exclusively one-sided.
I was also a bit disappointed, because although the paper discusses open science and addresses inequalities, it was published behind a paywall. More people could engage in this conversation and weigh the arguments by Fox et al. if the paper was open access or published together with a publicly available preprint (which Oxford University Press allows).
(Update: In the meantime, the authors have reacted and have posted a preprint on researchgate.)
At the same time, I wholeheartedly believe in an open discussion. We asked for a conversation on open science, and here it is. So I tell myself: Let’s be open and fair. Some aspects seem to be misunderstandings, and an ongoing discussion might help solve them. And, of course, some points are valid and important.
At all events, the negative reactions clearly show that several people don’t perceive open science to be open. How can we improve open science and make it truly open? Hence, in what follows, some thoughts on the paper, including some further soul-searching.
Good points
I actually share many of the points put forward by Fox et al. (2021). For example: Data should never be shared if this puts participants or researchers at risk. We should bridle the capitalization, mass production, and metrification of research output. We need to empower marginalized and less privileged researchers. Barriers have to be erased, research has to be made more inclusive and accessible. Power structures need to be made transparent, and many need to be revisited. Without hesitation, I’d sign the proposed alternative research agenda (Table 1).
I think the stronger parts of the paper address open science’s unknown unknowns or unintended consequences. Yes, of course open science wants to make science more accessible and diversified. Software should not be expensive but open source. Papers not behind expensive paywalls but at least published as preprints so everyone can read them. Researchers should be able to reuse others’ data, because not everyone has the resources to collect high quality data. Research belongs to the public, not the individual.
However, in making sharing data the norm, this indeed might put those who cannot do so at a disadvantage. At least, we have to make sure that this is not the case, so the warning is valid. We should not publicly audit and parameterize individuals, as proposed by Curate Science. Communication shall not be aggressive — and I personally really hope I’m not coming across as bropensciency. We should always protect minorities and marginalized people, and if marginalized people express fear this has to be taken extremely seriously. One should not overgeneralize calls for transparency, both participants and researcher need and require protection. Open access doesn’t equal accessibility, and researchers should not have to pay to publish their work.
Thus, the paper contains many relevant and correct points. And as we state in our agenda, “as we collectively move towards more open science practices in Communication, the agenda will be revisited, challenged, and expanded.” (p. 19). I’m glad we’re in the midst of this process.
Straw men
The main problem I have with the paper is that while there are some new and legitimate points of criticism, many of these points are held, expressed, discussed, and deeply shared also by the open science community.
Above all is the primary interest of protecting the participants’ privacy. In the agenda, we outline several strategies on how data can be shared while maintaining privacy, such as creating entirely artificial synthetic data sets. We clearly state that “[t]here are cases in which the full data cannot be shared because participants can or could be identified” and that “[w]hether or not the sharing of data is possible always needs to be evaluated in the context of each individual research project” (p. 17).
Again. Some research data cannot be shared. Full stop.
As we state in the agenda, “[i]n general, when sharing their data researchers should be as restrictive as necessary and as open as possible” and that “it is necessary to implement an appropriate informed consent process” (p. 17). So far, I’ve yet to meet an open science proponent advocating “unbridled openness”. I believe the statement that open science shows a “lack of consideration for participants”, “rarely mentioning associated participant risks” (p. 4) is a straw man.
There are other arguments I perceive to be misrepresentations. When discussing badges and rankings such as Curate Science or the TOP guidelines, Fox et al. conclude that “[c]ollectively, OS’s gamification reflects the broader cultural values of its creators (competition, capitalism) and is designed to reward similar researchers (see Dutta, 2020)” (p. 10). Yes, the pros and cons of badges can be discussed. Several open science proponents explicitly reject badges2, and also in our group some were against their inclusion into the agenda. Likewise, the use of rankings is always open to criticism, and rightly so. In my book, Curate Science’s ranking of individual researchers and the communication during this process is atrocious. Rating journals according to how many open science practices they implement, however, I believe is informative. That said, using these examples to infer that the general values of the creators of Open Science are “competition” and “capitalism” is irritating. Open science is about fighting the power of publishers, of making research materials, software, and data freely available, of openly sharing teaching materials. Many open science advocates actively address the miserable working conditions of ECRs. FWIW, the agenda explicitly builds on Merton’s (1974) principles of science, which explicitly include “communism”.3
As another example, in their paper the authors write that “[l]abeling those who practice OS “high quality” and “credible” scholars (e.g., Dienlin et al., 2021; LeBel et al., 2018) implies that other scholars are deficient or untrustworthy“ (p. 15). Although I of course see the point, I feel open science is misrepresented when in our paper we explicitly write that “[t]ransparency doesn’t guarantee credibility” and that “studies employing open science practices need to be evaluated just as carefully as traditional studies” (p. 18). Of course, also open science can be gamed, and I’m well aware of that.
I also don’t see how open science “considers diversifying samples less effective for conducting replications” (p. 15). Instead, for example when referring to the generalizability crisis4, open science advocates routinely call for more representative samples, in order to fight the widespread WEIRD problem. In the agenda we explicitly call for larger samples. But yes, important to add, also more inclusive samples.
Power and hierarchies
„Notably, the approaches typically overlooked by OS (qualitative, participatory, and critical methods) are central to research that challenges the White, male, elite, and Western bias of academia (Dutta, 2020; McMillan Cottom, 2015).“ (p. 11)
I think there are two interesting points here. First, indeed, the open science community focuses on quantitative research. But — and that’s a big but — that’s because in quantitative research the problem is much bigger! P-hacking, data-dredging and HARKing are genuine problems of quantitative research. As we write in the agenda, “qualitative research cannot really have a replication crisis, because it is not the explicit aim to generalize over an underlying population” (p. 15).
I genuinely believe quant folks are not disregarding qualitative researchers. Instead, they are often turning to qualitative researchers, for example to help detect biases. To engage in reflexivity. To avoid self-deception. When we included the part on qualitative research and open science into our agenda, our intention was to be more inclusive and open-minded, but not to be prescriptive.
Let me share some very personal impressions with you. It is my own experience that open science folks are critical of the existing hierarchy in academia. It’s about calling out BS-vendors, hackers of the game, people in for the fame, success, and status but not the actual science. Open science is very grassrootsy, driven be ECRs, discussed in journal clubs and by idealists. I personally believe these are sentiments and characteristics often shared by more qualitative folks and by more marginalized researchers. So it’s interesting that open science is perceived as coming from or even supporting a position of power and might. I personally believe that by advocating open science I’m not making many fans among seniors. It often still feels risky.
To me, the criticism coming from more qualitative-oriented scholars always feels a bit like friendly fire.
But that’s just my very personal feeling. Indeed, I might be wrong. I’m white, male, raised in an academic background, and definitely privileged. Intentions are only intentions, not effects. For example, I’m increasingly realizing that it needs profound IT skills such as programming to partake in many open science practices, and these skills are often determined by socioeconomic factors.
So this is a point where I want to learn. Why is open science perceived so negatively by some? I suspect it’s (also) because open science is indeed often too prescriptive. It might be perceived as yet another instance of quant researchers telling qual researchers how to do their job. Arrogance, conceit. Or just generally forcing or overly pushing people to do something. No one like this. So let’s try harder to avoid that, and let’s be more welcoming and open.
Conclusion
I’m happy the authors wrote their paper. If Open Science is perceived as closing doors, then we need to change how we conduct and communicate about open science. Some practices need revisiting.
So let’s cultivate ethical, inclusive research practices early and often. Let’s practice reflexivity. Let’s respect and empower people. Let’s promote ethical transparency. Let’s handle data responsibly. Let’s share resources and knowledge.
Footnotes
- In what follows, I sometimes refer to “more qualitative scholars”. Of course, not everyone voicing criticism regarding open science is “more qualitative”, critics include scholars using more quantitative methods. It’s also a continuum.
- See this blog post by Hilda Bastian
- “All scientists should have common ownership of scientific goods (intellectual property), to promote collective collaboration; secrecy is the opposite of this norm.”
- https://psyarxiv.com/jqw35/