I’m currently at the 70th Annual Conference of the International Communication Assocation, where this year’s theme is Open Communication.
One ongoing discussion is how open science works for research coming from a more qualitative perspective.
Because of COVID-19 the conference is all digital, which is why I’m tweeting even more than usual (sorry ’bout that). In what follows, you can find the transcript of a tweet that was particular long, so that you can read it more easily. Here we go!
🍿
Okay, #ica20, let’s do this. Here are my thoughts on #openscience and #opencomm for the more qualitative approaches in our field!
(Beware, long thread)
(Also, please be kind)
Background: I was VERY curious how our agenda for #opencomm would be received in our field (if you don’t know what I’m talking about, here’s the paper).
Basically, we’re calling for making research more available, transparent, & traceable. We call for the sharing of data (whereever possible) & analysis scripts, and the introduction of preregistration, registered reports, large scale collaborations, the TOP guidelines, etc.
We explicitly criticize QRPs such as p-hacking, harking, story telling, or the general incentive structure that prioritizes quantity over quality.
Although we explicitly focus on quantitative approaches in the hypothetico-deductive context, we also offer some suggestions how #opencomm might work for the more qualitative approaches (see talk by @lindseybier https://www.youtube.com/watch?v=nm-19808jSo&list=PL_wlr6EiCKuXAj0j-EoJnTmfWkbxpFVnn&index=3)
So, again, I was VERY curious re. how our agenda would be received by our community.
Overall, the feedback was overwhelmingly positive, and I’m really happy & thankful for all the nice & kind words we’ve received 🙂
However, both in our reviews from JOC and the subsequent discussion at #ica20, we did get a bit of pushback coming from one particular direction — which, tbh, somewhat surprised me.
(But, to be crystal clear here: Feedback was fair, constructive, good spirited, and self-reflective — just the way it should be.)
Interestingly, our agenda was not criticized by the industrious p-hackers, which was what happened in psych.
(Mb they’ve learned from the mistakes of their psych kin, bc arguments simply don’t hold up?)
Instead, pushback came from the more qualitative approaches. I’ve read the following objections & concerns:
(pls. feel free to correct wrong understandings or to add novel points.)
1. Replicability not relevant, bc. all knowledge is contextual.
2. Through self-reflectively explicating lenses & biases, qual. is already doing most of this anyway.
3. The researcher is part of the knowledge process, hence cannot be separated from the results.
4. Preregistration doesn’t make sense, bc. we don’t know what we’ll find beforehand.
5. Knowledge is co-created together with participants, hence it’s a process that cannot be planned/explicated/replicated.
6. Results need to be interpreted and are never absolute/general.
7. Related, it’s not about replicability or generalizability, but transferability.
8. In a lot / most (?) cases, sharing data not possible/useful: It’s too risky for participants, even when anonymized/deidentified, and too much effort for too little value of what’s left.
Here’s my reply:
Qualitative research is not the problem. Qualitative research is the solution.
Exploratory research is not the problem, either. The problem is selling exploratory research as confirmatory. Exploratory research is CRUCIAL for knowledge building. It’s part of the solution.
Preregistration & Registered Reports do not prevent exploration. Instead, by explicitly labeling unplanned/emergent analyses as exploratory, we make them more prominent. We elevated them.
Preregistration is mainly for hypothetico-deductive, quantitative approaches. If you’re 100% exploratory, just skip preregistration! Preregistration is not mandatory. But most of the time we start with a certain theoretical framework, perspective, assumption, or hunch — this you can preregister.
Yes, you absolutely cannot share specific kinds of data. End of story.
Yes, qualitative research is much more contextual. BUT: It’s not a 100% contextual. Bc if it were a 100% contextual/idiosyncratic, we could, by definition, not transfer it. But we can.
And, granted, we do: Just read the discussion sections, you’ll often find many (important!, relevant!) claims re. general power structures, disadvantage, oppression mechanisms, etc.
BUT: You cannot have the cake and eat it. It’s a truism that for the very same reasons, the transferability of insights coming from small scale qualitative studies is also more limited.
Let’s not forget: Objectivity >= Reliability >= Validity. Let’s also not engage in jingle-jangle fallacies: Transferability ~ generalizability.
Awareness of reduced generalizability is not a problem, it’s again part of the solution. Quant researchers can learn much from qual researchers:
Our knowledge is highly contextual.
Don’t overclaim.
Know yourself.
Know the structure you’re working in.
Be humble.
(The irony.)
I’m perfectly aware that all this is highly delicate: E.g., there are different understandings of what qual. research itself even is. In our agenda, we explicate that. I’m no expert on qual. I’m hence happy to have other jump in here. These are my thoughts, my understanding.
I honestly believe that the replication crisis has shown that we should be more humble, more descriptive, more nuanced. If I’m not mistaken, these are core values of qual research.
I would love to encourage qual research to see the many opportunities #openscience offers. This is really not about paternalizing qual. & exploratory research. This is about empowering qual & exploratory research!
Don’t believe me? Well, then read @talyarkoni magnificient paper The generalizability crisis.
(Background: Tal is a leading figure of the #openscience movement in psych and, without doubt, currently the smartest person living on this earth.)
This is what he writes:
Conclusion: #openscience and #opencomm will lead to knowledge claims that are more robust. In this process, qualitative research is not part of the problem but part of the solution.
Let’s all join this discussion, continue to exchange thoughts and concerns, see what works and what not.
I truly believe that #openscience has much to offer for all of us.
(Please don’t kill me.)