In my personal experience the main problem with scientific research today is caused by publication counts representing the sole metric for academic success.
Research is 'optimized' for publications and segmented according to conference schedules. That in itself I consider detrimental to the research effort. However the major damage in terms of advancing resarch in my opinion is caused by the induced tendency of scholars to closely stick to the main stream paradigms, trends and topics. Naturally it is much 'harder' to do research that does not directly build up on the current state of the art. Such research usually takes much longer and therefore results in a lower publication frequency. Secondly it is carrying a much higher risk of not bringing about positive results. Thirdly I deem the chances for acceptance in peer reviewed conferences and journals to be much lower. Such venues are often biased towards current main stream research trends.In my observation there is also an aversion against research that questions the state of the art. People who have spent years or decades to master every aspect of the state of the art have a strong incentive to shoot down anything potentially disruptive.
As such the incentives are misaligned with regard to the aim of science of figuring out new things.
> In my personal experience the main problem with scientific research today is caused by publication counts representing the sole metric for academic success.
It also leads to bloat in the literature: People learn the "least publishable increment" and go publish that.
In their 1970 Annual review chapter on Memory and Verbal learning, Tulving and madigan reported that they had independently rated 540 published articles in terms of their "contribution to knowledge". With "remarkable agreement", they found that they had sorted two-thirds of the articles into a category labelled "utterly inconsequential"."The primary function these papers serve is to give something to do to people who count papers instead of reading them. Future research and understanding of verbal learning and memory would not be affected at all if none of the papers in this category seen the light of day"
About 25 percent of the articles were classified as: "run-of-the-mill .. these articles also do not add anything really new to knowledge... [such articles] make one wish that at least some writers, faced with the decision of whether to publish or perish, should have seriously considered the latter alternative"
Only about 10% of the entire set of published papers received the modest compliment of being classified as "worthwhile".Given that memory and verbal learning was then a popular and relatively 'hard' area of psychological research, attracting some of the brightest students, this is a devastating assessment of the end product.
Hence, of the research ideas generated by these psychologists, who are all card-carrying scientists and who liked these ideas well enough to invest weeks or months of their lives working on them, less than 25% of 40% of 10% = 1% actually appear to make some sort of contribution to the discipline.
Given that this work was published in the 1970s when the "publish or perish" pressure was less strongly developed I expect today's number to be worse. Concerning Information Retrieval the field of research of my PhD I would certainly attest that this applies.
I have been trawling through large numbers of papers recently as part of my computer science Master's background research, and some papers are quite frankly unintelligible garble... I can't even understand how they passed peer review.
I would say that of the articles I've read, maybe 5% are meaningful contributions that advance the science; maybe 25% contribute something to the current understanding, perhaps 60% are "we did something cool", and 10% are "are these reviewers awake".
Thanks for those links. During my own PhD, I was pretty sure that the "least publishable increment" became negative.
The underlying problem is that there is no penalty for publishing non-increments. For some inexplicable reason, citation impact assessments do not divide by the total number of publications.
I have no doubt that such a normalization factor would go a long way to fixing the problem.
There's been an attempt to measure paper quality using the "impact factor", but as with seemingly every measure, it's led to shenanigans. Some publishers encourage authors to cite papers from their journals, and some editors write survey papers that happen to cite a bunch of articles from their journals. A recent AMS Notices covered this in math, and I sort of expect it's worse in other disciplines.
To me it looks a lot like a manipulation of search engine. Maybe Google or Microsoft will not disclose their rank inflation counter-tactics, but what they could do is to provide their solution for this specific verticals.
As much as I'd like open-source effort for that, I understand that knowing the rules allows easy manipulation.
Well, there already is Google Scholar. I don't know precisely how much effort goes into the ranking algorithms used there -- but I do already use it in preference to Pubmed.
Research is 'optimized' for publications and segmented according to conference schedules. That in itself I consider detrimental to the research effort. However the major damage in terms of advancing resarch in my opinion is caused by the induced tendency of scholars to closely stick to the main stream paradigms, trends and topics. Naturally it is much 'harder' to do research that does not directly build up on the current state of the art. Such research usually takes much longer and therefore results in a lower publication frequency. Secondly it is carrying a much higher risk of not bringing about positive results. Thirdly I deem the chances for acceptance in peer reviewed conferences and journals to be much lower. Such venues are often biased towards current main stream research trends.In my observation there is also an aversion against research that questions the state of the art. People who have spent years or decades to master every aspect of the state of the art have a strong incentive to shoot down anything potentially disruptive.
As such the incentives are misaligned with regard to the aim of science of figuring out new things.