After many years of doing medical research it is very clear to me that the vast, vast majority of what we create in terms of research is utter garbage, and it is so a priori, i.e. we know it before we publish.
There are hundreds of meta research papers on this topic, including the Lancet series “Reducing Waste”.
Why is this the case and how can this be resolved?
TLDR: It isn't.
Longer answer: Much medical research is flawed. How much is debatable, but it's a lot. But the flaws do not always (or even usually) make the research "useless" or "utter garbage".
For instance, in much medical research it is impossible to randomly assign people to conditions. E.g. you can't randomly assign people to "overweight" and "not overweight". That does not make the studies of obesity useless.
Medicine isn't physics or chemistry. People are complicated. Disease is (often) very complicated. There is a lot more error (in the statistical sense) in medicine than in physics. But that doesn't mean it is useless. (E.g. a lot of people who smoke don't get cancer, but so?)
Of course, there are ways medical research could be improved. I am a statistical editor at a couple of medical journals, so I do my small part to improve it.
Another issue is an over-reliance on randomized clinical trials, which reduce some kinds of error at the cost of increasing others. For one thing, people act differently when they are part of a trial than they do in real life. And RCTs often look at samples that have no other medical conditions when, in real life, people with one condition often have others. And there are problems with selecting too many White men and not enough women or people of color. And so on. It's by no means perfect. But there's a long way between "perfect" and "garbage".
For evidence, just look at the amazing progress medicine has made, much of it based on research.
Of course the "pressure to publish" doesn't help. But ... that pressure also applies to the people who are writing papers decrying the state of research.
Apparently, there is a debate in the community, since you pointed out to Lancet "Reducing Waste". I guess they also point out to some solutions to problem, not simpy yelling at the produced waste.
The best path forward (best as "minimal effort and penalty for you, maximal long term gain) you can try is to keep on publishing your waste (as required by career, social pressure, contracts, research agreements and so on...), and continuously mention and refer to "Reducing Waste" of Lancet somewhere in your publication (I would suggest either the introduction or discussion section).
Advertising works very well that way, rewiring brains through repetitions, why should not this work in the medical community, too?
Why is 90% of medical research useless?
If that implies 10% of medical journal articles are useful, then I would say that's good performance. There are several fields of research that never produce anything "useful," some of which do not intend to be useful.
Publishing useless journal articles is useful in that it discourages other people from repeating the useless research (0% success rate). That encourages people to do something new (apparently 10% success rate). Some think publishing the useless results is obligatory.
If I simplify it, maybe one of the reasons that many of the doctors publish their papers based on data collection e.g., out of 100 selected patients, in certain age groups certain findings, and so on. Finally giving a conclusion at the end of the paper, which the doctor already knows. This is because many doctors prefer their knowledge-based profession instead of thinking analytically about some disease or ignoring critical thinking. Doctors are trained as practical professionals, not as researchers. They don't go through a PhD. They earn a huge amount of money by treating patients with existing diseases and become ignorant about unsolved problems (diseases in this case). To be precise, medical professionals, especially doctors, do their jobs on existing problems or existing diseases. So they are bound by the nature of their profession.
However, of course, there are some researchers in the medical sciences who don't practice as doctors and instead focus on finding solutions to unsolved diseases. Of course, those medical professionals publish pure, original research papers.
To conclude my answer, I would make a separation between medical doctors and medical researchers. The first category focuses on the treatment of patients and earning a flexible livelihood. While the second category is dedicated to real research to solve mysteries about many unsolved diseases, this second category certainly publishes fundamental research papers. Of course, researchers from biochemistry, bioengineering, applied mathematics, and biophysics also publish some original research works that are important for the medical sciences, as you can see among the Nobel Prize winners in chemistry in the last few years, some of who worked on destroying cancer cells, etc.
„Medical“ is a red herring. Any deep research, if done properly, is useless. The reason is that the term research primarily refers to creation, systematization, and maintenance of knowledge. It's not about being useful to anyone or for any purpose.
If you wish that what you do is useful, do something else. Don't do research. Or at least stop fooling yourself by expecting any kind of application of proper research. Or switch to applied research, which, by its definition, is meant to be applied in some area.
As for garbage in terms of poor knowledge, it's also not specific to medicine. Most of research out there is irreproducible or low-quality in some way. There are many reasons: poor publication standards (including inadequate control in publication), poor/inadequate/small empirical studies (including too much freedom for too few tested individuals), the contents (in some cases) being not interesting to anyone except the researcher and 1-2 other folks on Earth, etc. If you wish to do something higher-quality, go to an area with better control and higher interest in your product.
