I think that a very stringent threshold for p values (it's going to be a lot harder to get to p < 0.005 than p < 0.05 and it's going to encourage a lot more gaming of the system) is not the solution. At best, it will lead to smaller numbers of much bigger studies with more authors and fewer interesting results.
I'd suggest that a better, broader, and more practical solution is to provide a public clearing house for studies and associated data -- let's assume it's a website.
a) When you start a study you must describe your study, methodology, and hypothesis, and publish these things on the site. It should be possible to find the study the same way you find any paper that results from it.
b) All data must be stored in the same place (but not made public) as it becomes available, and this must be demonstrable. This data should be available to anyone trying to replicate, review, etc.
c) Final data sets and analyses must also be also be available, ideally with the code used to do the analyses.
All of this should be prerequisite for review and publication, but also for anyone working in the field should no papers result.
This solves a lot of problems, including making it easier to conduct meta-analyses when looking for effects that aren't significant in smaller studies, and also address the lack of availability of negative results, and even help combat theft of methodology (e.g. where someone sees work in progress and is able to publish using the methodology). It would make meta-analysis easier, and allow researchers to better take advantage of existing, unpublished results (e.g. to hone their study methods, etc.)
If we were designing things from scratch, I'd even want t odisaggregate two career tracks for scientists:
The first, theorists, design interesting studies with open ended questions, explain how the study would provide useful scientific results, or theorize about the implications of existing published results.
The second, practitioners, are hired to actually run the studies according to the specifications provided. The practitioners would publish null results of course. Practitioners would also have to acknowledge how many of their results are significant, so it'd be transparent if they were just confirming everything.
A bit like barristers and solicitors in many common law countries, a distinction that isn't strictly necessary for the system to work, but prevents some conflicts of interest.
Yes, and indeed with this system a theorist can look for interesting data out there and publish with attribution, while experimenters can look for interesting failures and home methodology.
One of my favourite comments from a lecturer in my applied science degree: "In theory there's no difference between theory and practice. In practice, there is". We encountered so many times when theory said one thing but it didn't pan out when tried.
It's another way of saying "First, take a spherical cow..."
I'm thinking back to my graduate thesis project, in physics. I started with a fairly nebulous problem, and went down a lot of blind alleys before I finally got something working. And there were some unexpected results that we didn't even have a good way of describing until we saw them happen and puzzled over them. I published a few weeks later, and my advisor moved to another university a few weeks after that.
A rigid rule of pre-registering "study, methodology, and hypothesis" would have prevented me from even launching the project. At the very least, I would have had to register a formal study after my results were already established to a decent degree of confidence. It would have been wasteful and redundant, not to mention risky, since anything can happen to a grad student to prevent them from finishing a degree (illness, loss of faculty advisor, etc). At best, time is money for a grad student who wants to get out of school ASAP and start earning a salary.
My project produced a humble but solid result, and launched a new research program for my advisor, that bore fruit for another 25 years.
Now, I wasn't doing a "study," as in a life science study looking for a statistical correlation. I reported no p-value. So maybe it doesn't apply to all kinds of scientific work. And I appreciate the need to restore come kind of integrity to the medical and behavioral sciences in particular. I don't know how to help those fields, and maybe pre-registration only applies to them. But I'm concerned that declaring rigid rules for science across the board will have unintended consequences.
Edit: Another issue is a student getting "scooped," where some other lab reads the plans and rushes the work to completion. There are labs in chemistry and biology that are notorious for doing this. If a student doesn't at least have an informal guarantee of priority over their own work, they won't take the risk of launching a project in the first place.
assume that, to simplify, that every research paper out of there has findings with p-value < 0.05 - that means one in 20 papers has wrong conclusions in it.
or that one in 20 teams will not be able to reproduce findings.
0.05 is just removed enough from coincidence that, given a solid theory, corroborates the findings. but running across data to find correlation and using 0.05 as threshold is a surefire way to find something that has no actual significance beyond mathematics and zero ground in reality.
Because of conditional probabilities, your analysis is wrong. This xkcd explains the issue. Someone else linked to it already, but it exactly describes the problem -- with p=0.05 >> 5% of published results are incorrect:
You're making the same mistake in reverse. The p-value is not a marker of correctness, but of confidence. If something doesn't reach the p-value cutoff, that doesn't mean that the effect doesn't exist, just that it didn't meet the confidence level. A paper could be entirely correct in its hypothesis, yet not meet the confidence level in its experimentation.
I'd suggest that a better, broader, and more practical solution is to provide a public clearing house for studies and associated data -- let's assume it's a website.
a) When you start a study you must describe your study, methodology, and hypothesis, and publish these things on the site. It should be possible to find the study the same way you find any paper that results from it.
b) All data must be stored in the same place (but not made public) as it becomes available, and this must be demonstrable. This data should be available to anyone trying to replicate, review, etc.
c) Final data sets and analyses must also be also be available, ideally with the code used to do the analyses.
All of this should be prerequisite for review and publication, but also for anyone working in the field should no papers result.
This solves a lot of problems, including making it easier to conduct meta-analyses when looking for effects that aren't significant in smaller studies, and also address the lack of availability of negative results, and even help combat theft of methodology (e.g. where someone sees work in progress and is able to publish using the methodology). It would make meta-analysis easier, and allow researchers to better take advantage of existing, unpublished results (e.g. to hone their study methods, etc.)