I was excited when last year I saw that International Journal of Information Management (IJIM) published a meta analysis on the privacy paradox. Does the privacy paradox exist? The authors concluded that yes, the “‘privacy paradox’ phenomenon [. . . ] exists in our research model”.
I was surprised, because this finding contradicts another meta-analysis by Baruh et al. (2017), which found significant relations between privacy concerns and information sharing. Also in my own research, I regularly find significant (though often small) effects (for example in this study).
So I carefully read the paper. I quickly realized that there were several statistical errors (RMSEA didn’t make sense; fit cannot decrease when adding paths), problems with inferences (why focus only on direct effect & not total effect of concerns; bivariate relation between concerns and information sharing actually shows small but significant effect) and theory (perceived risks were not discussed when evaluating the existence of the privacy paradox; however, risks were a relevant predictor).
So, does the privacy paradox exist? Bottom line: The paper doesn’t provide evidence that the privacy paradox exists. If anything, it even provides evidence against the privacy paradox.
How to react?
Errors happen, to everyone, including me. That’s not problematic. Instead, errors should be expected and embraced.
Important, however, is how we react to errors.
So I thought, given this is a meta-analysis likely to have a strong impact on our field and a RQ I deeply care about, I should write a response/comment.
To make sure I wasn’t missing anything, I reached out to Ye Sun, who’s an expert on meta-analyses. She agreed with my criticism. We shared our concerns with several colleagues, who likewise agreed that there were several problems.
Publishing the comment
I asked the editor of IJIM, Dr. Yogesh Dwiwedi, if they would publish comments. He declined. He said we should reach out to the authors. We wrote our comment & shared it with the authors. He later agreed that we could submit Technical Note. Authors were part of the reviewers.
We got mixed evaluations, but we were invited to resubmit. We were told that a response wouldn’t be sufficient and that we’d need new data. We didn’t collect new data but added a second part with a research agenda and a reanalysis of Baruh et al. (2017). Our revision was rejected.
Some thoughts: Dr. Dwivedi was critical of our approach. But I appreciate that eventually we could submit a paper. Reviews weren’t enthusiastic either; one even said our response could lead to “legal implications”. To summarize, our response wasn’t particularly welcome.
We then submitted to Meta Psychology: “Traditional journals will not always publish commentaries on articles they have published. We welcome these types of contributions regardless of what journal the original article was published in.”
It was a streamlined review process. Three of four reviewers agreed we could forward their reviews. (If you want to read the reviews, you can find them here.) Based on these reviews, editor Rick Carlson opted for minor revisions. We updated our manuscript. In the last week, it was now finally published — you can read it here.
Conclusion
1. It’s important critically to engage with research. In a fair & constructive way. I hope & believe our comment is fair & constructive.
2. Let’s be more open to critical discussions of research. Please criticize my research and especially this comment. Because: