IMO this would track, applied mathematics (even e.g. data science, though perhaps calling that applied math is a bit generous / insulting to more serious applied math) is in some ways more exciting now because it is far easier to surface complex / appropriate methods for the task at hand, and you can more confidently explore these methods because the AI sort of "has your back" in catching some of the more obvious beginner errors you make during these explorations. Plus, applied math feels roughly more results- than process-focused, compared to pure math.
IMO the divide here between pure vs. applied math feels a lot like the divide between those who enjoyed coding for the understanding it led to, i.e. the writing itself was the joy, vs. those that primarily coded for the results. I enjoy the creative part of coding, the thought of software jobs just devolving into writing specifications and doing code review very much kills it for me.
Writing is pretty hard for a lot of people, maybe especially so if they are more non-verbal thinkers, and then doubly so again if one must write not in one's native language.
Depends on the writing. Writing a blog post or something is challenging because you want to have an engaging style. But I see people at work using LLMs to fill out tickets, which is the easiest thing in the world. It's just a plain description of things, it requires no skill at writing at all. It baffles me that people are using LLMs for something like that, which should take less than a minute of your time to fill out and will be more pleasant for the recipient than the slop the model produces.
Here I would maybe argue that the simplicity makes these kinds of tasks tedious (like doing taxes), so it also makes a lot of sense for people to want to just throw AI at it. Tedious, easy rote tasks can be much more unpleasant than engaging ones.
I think people massively over-rely on AI and I really worry about the consequences of this. But there is nothing baffling at all about the basic appeal, IMO.
AI is very bad at properly handling statements that make heavy use of vague quantifiers (e.g. "some", "most") and also commits a lot of pretty serious logical fallacies. It is also bad at handling subtle logical negation, generally.
One of the most egregious negation issues I run into a lot is when I (or someone) makes a statement of the form: "not X" or "X is thus not true", and the AI then proceeds to interpret or summarize this as 'whatever is the opposite of X is the case'". This will cause it to go down a useless path investigating or disputing the opposite of X, which generally has no relevance or bearing on anything.
It also often very harmfully will replace your carefully chosen words with weirdly specific academic operationalizations or formalisms, then again waste huge amounts of text refuting / showing "problems" that result from that formalism, all of which again have no bearing or relevance on the original statement. An example would be you saying something like "intelligence, generally, must surely explain some of the differences in X", and then it will go "actually IQ does not correlate with X", unless you specifically tell it not to conflate psychometric IQ with intelligence generally.
Sometimes this is helpful, but the more specific / technical the domain, the more often you specifically have to prevent it from going down stupid paths that should be obvious given the expert context and wording, because it can seem almost hungry to try to catch you in some kind of insipid 'gotcha'. Much of these issues often clearly arise immediately from the first-pass "reword what the user said" part, given the reasoning traces.
And also to think in different directions than you would have gone in isolation. Regardless, this is a great heuristic / rule that I will be sharing and keeping in mind.
This feels very related to the issues re: the presence or absence of world models in LLMs. Insofar as they have world models (or "intuitions"), these would seem to have to be primarily verbal-linguistic (or symbolic, when using math). LLM world models are not likely (currently) very spatial, in contrast to e.g. V-JEPA-2 models, which likely do have some basic spatial models (and perhaps "intuitions").
There's an old joke about funding, goes something like:
"Why you are always demanding more funding? Why can't you be more like the mathematicians, all they need is a desk, some paper, and a pencil, and a garbage can, and they just do fine. Or how about philosophy, for that matter? They don't even need the garbage can"
I mean, obviously with modern computational mathematics, this doesn't hold so simply, but there is this confound about math research also not getting much funding also because much of it isn't that expensive, relatively speaking.
Last I knew, at least in America, universities that want their math prof's to do research also expect those prof's to bring in plenty of outside funding. You could argue about the costs of that desk, paper, pencil, and such - but modern "research" universities have evolved into extremely high-overhead operations, and The Beast Must Be Fed.
This is tricky, because, in fact, hard math having an intelligence floor is one of the nastier realities of the human condition. Anyone who is even quite intelligent but has really pursued the rigorous stuff, unless they are in fact a prodigy, eventually realizes they have an abstraction ceiling (and this term is a common one thrown around in people studying mathematics, because intelligence denial is so obviously false when you do hit your abstraction ceiling).
Most people are correct that they lack the intelligence / mind for a lot of hard math (even epsilon-delta proofs are enough to eliminate the majority of the population, no matter how good a teacher you are, and these are just basic undergrad calc).
And yeah, sure, people have different kinds of intelligence and such, but there is still a g-factor, and people of low intelligence almost universally can't do hard math, whereas most people who can do e.g. advanced undergrad math can generally do almost all other advanced undergrad fields reasonably well. The world isn't fair here.
> Anyone who is even quite intelligent but has really pursued the rigorous stuff, unless they are in fact a prodigy, eventually realizes they have an abstraction ceiling
Eh. I'm a math PhD who fled academia because it was too much for me. But I have never encountered this term "abstraction ceiling" nor did I succumb to it. I simply ran out of motivation to pursue higher math, especially when following through on learning and research became more and more labor. (It was always labor; but it was a labor I used to love.) I am far from a prodigy.
> even epsilon-delta proofs are enough to eliminate the majority of the population, no matter how good a teacher you are
Disagree. It's a notoriously hard subject to teach, and with all the demands placed on e-d in so little time in your average curriculum, it doesn't require appeals to IQ to explain its infamy. With enough motivation and practice, the quantifier alternation is comprehensible to any sound mind. What your average mind (and student) lacks is exposure to formalism, abstraction, and how these things tie in with what they are familiar with, which is symbolic manipulation. With the exception of geometric proofs (another educational bugbear), they have little context for what formalism is or why it matters.
> Eh. I'm a math PhD who fled academia because it was too much for me. But I have never encountered this term "abstraction ceiling" nor did I succumb to it.
This sounds a lot like you may have in fact succumbed to your abstraction ceiling, because in practice, the ceiling manifests as not as it being impossible for you to learn something, but that it would take you years and inordinate effort to master what you notice others mastering easily in just a fraction of the time. You may have not heard the exact term (comes from Douglas Hofstadter), and you may be talking about just the academic busywork, but I find it hard to believe you never encountered discussions about this kind of stuff. I would also politely suggest that unless you are Terry Tao posting under some kind of alt, you most certainly do have an abstraction ceiling (or your own mathematical limits) too.
> It's a notoriously hard subject to teach, and with all the demands placed on e-d in so little time in your average curriculum, it doesn't require appeals to IQ to explain its infamy. With enough motivation and practice, the quantifier alternation is comprehensible to any sound mind
The latter statement is obviously false, but regardless, intelligence explains some of the difficulty, and much other difficulties far more parsimoniously than "everyone could just learn any math if they just tried hard enough and had good enough teachers". E-d is merely an obvious and generally familiar example, and nothing I said really relies on this very specific aspect of maths, obviously. We also shouldn't pretend your (almost certainly false) view of math and intelligence isn't also often harmful to struggling students in its own way.
> Since then I've had the chance, in the world of mathematics that bid me welcome, to meet quite a number of people, both among my "elders" and among young people in my general age group, who were much more brilliant, much more "gifted" than I was. I admired the facility with which they picked up, as if at play, new ideas, juggling them as if familiar with them from the cradle - while for myself I felt clumsy. even oafish, wandering painfully up a arduous track, like a dumb ox faced with an amorphous mountain of things that I had to learn ( so I was assured), things I felt incapable of understanding the essentials or following through to the end.
(Alexander Grothendieck, Recoltes et Semailles)
Amazing that he managed to keep going after hitting his abstract ceiling in graduate school.
You clearly don't understand the meaning of the term. Grothendieck was almost certainly wrong about his gifts here, and even if not, your mathematical ability and output isn't fully explained by your ability ceiling.
Honestly, the pushback on this post is utterly baffling. Clearly the human mind has limits on what it can comprehend and the rate at which it can learn difficult things. Clearly these limits differ among individuals and are related to intelligence broadly.
Huge proportions of the population struggle to ever even grasp simple fractions, and not for a lack of effort from them or society. Fourth-year undergraduate mathematics is another beast entirely. Pretending the world is otherwise is pure fantasy and also plainly harmful, to the world and people that are unfairly pushed beyond their capabilities.
> Grothendieck was almost certainly wrong about his gifts here, and even if not, your mathematical ability and output isn't fully explained by your ability ceiling
lol, see, it's unfalsifiable. No true abstraction ceiling.
Find me the magical teaching method that can make anyone learn any kind of mathematics at nearly the same rate, and you have falsified the idea that anyone has mathematical limits.
Given we as a society can't even figure out how to do this for educating stuff involving simple fractions, my theory is far superior than whatever exactly it is you think.
That's far different than telling a math PhD whom you've never met that they hit their "abstraction ceiling" based on a 2/3 paragraph comment on hacker news. I'm sure you'd have said a similar thing to a young Grothendieck if he were describing his early struggles in graduate school. You're getting pushback because you were being rude and presumptuous.
Work on your reading comprehension, I made it clear that increased effort is what an abstraction ceiling feels like, but also made it clear that GP could have been talking about the effort of academic busywork.
Let's also not pretend that "you could have learned epsilon delta proofs, you just didn't try hard enough or your teachers weren't competent" or "you just didn't have enough time" and etc. is also not rude and presumptuous. Denying the existence of such limits is equally offensive.
You’re in a kind of compulsive ideology here. This same vein of thought and why it is harmful is described by David Bessis in the book Mathematica: A Secret World of Intuition and Curiosity.
“To stop thinking in terms of “gifts” and “talents,” one has to find an alternate explanation. My way of looking at things, which has served me well throughout my career, was to imagine that creative mathematicians were hackers who had found ways to unlock “hidden modes” of our cognition. Most of the time, they’d done so unwittingly, and were entirely incapable of explaining how.”
It is a phenomenon like child-like mental yoga of attention.
Uh huh. A bunch of quasi-mystical bullshit to justify utterly inept garden variety intelligence denialism (it also just shifts terminology: if we accept your metaphysics, it would still be strange to propose that everyone has exactly equal "hacking" ability).
The hubris and willful ignorance required to imagine that everyone is just equally and infinitely unbounded in their cognitive ability is simply mind-boggling in 2026.
But everyone is as a kid when they learn language and everything else. Some people retain that level of watching, listening, and babbling (hacking) when they don't know what to do. So the task is just to get people back into that mindset. Innate intelligence isn't necessarily only symbolic manipulation (analytical), it also is experiential/creative and practical per Sternberg.
> But everyone is as a kid when they learn language and everything else
Also clearly false by almost all current research.
> So the task is just to get people back into that mindset.
Again, you have no evidence, and this is clearly wrong in cases of mental retardation or brain damage. Modern genetic studies also seem to suggest intelligence is related to a lucky absence of errors / genetic problems that are otherwise inconsequential (or even advantageous) in other domains, so really, your "everyone starts perfectly equal in intellectual ability" is just empirically disconnected and ignorant fantasy.
I haven't talked to a child educational psychologist lately (I imagine that would be quite renewing to spend some time in a kindergarten even if only virtually, don't you?), but I don't think biological claims about elite abstraction is the only way when it comes to explaining more and more with less and less for mathematical education. For example, non-symbolic distinction and indication operations are far more fundamental than symbolic manipulation, and using that is a bottoms-up foundation more in keeping with constructivist understanding exhibited by math exemplars.
Cognitive disparity is because of compounding investment of attention and metacognition in development, preferably in a self-referential non-symbolic universal way because intuition is partly based on sensual metaphors and embodied cognition. Everyone has issues distinguishing ungrounded concepts if they don't have a map of them from their attention previously..
Mental rigidity (aka "fragile perfects") is a fairly common phenomenon for math anxiety, whereas Grothendieck advised uninhibited playfulness to deal with uncertainty. The perceived difficulty of mathematics is a social phenomenon rather than organic comprehension limits on abstraction ceiling.
It is the social aspect of math that is the superintelligent part of it, which transcends the genetic determinist perspective. Civilization advances because education transforms the breakthroughs of genius (which all children have ultimate capability for) into the baseline intuition of the rising children by sharpening their attention. Math is supposed to be a democratization of human understanding, that's why the Greeks were so keen on deduction and why proofs are for systematic communication. If a stupid-ass computer can do math, so can any human being.
> Cognitive disparity is because of compounding investment of attention and metacognition in development [...]
Sure, but exclusively? There are no other factors that don't depend on effort / investment / social context?
I can't take you seriously when you take such an absolutist stance on these things when science has long since accepted nothing complex about humans is 100% nature or 100% nurture (really, shared vs. non-shared environment vs. genetics: but, surely you know this).
There certainly are numerous factors claimed. I believe cognition happens from distinguishing reality and people can always learn better how to do this.
Whereas “abstraction ceiling” is hard science backed by ample literature, not a loose metaphor directly contradicted by several prominent mathematicians who didn’t quit when the going got tough.
I think the fact the Feynman took an IQ test and scored 127 is damning of the entire concept. I think what turns mathematicians off is the thought that psychologists who couldn’t tell you the difference between a scheme and a metric space think they can actually measure who has the capacity to be a mathematician and who doesn’t.
Like, why fucking bother doing anything? Why run the 100 meters at the Olympics, let’s just do some genetic testing and measurements to pick the fastest man in the world. Why teach kids music, let’s just measure hand size and do some sight singing exercises and teach the talented kids piano. This whole nonsense reeks of Gattaca-style quasi-eugenics where people get sorted into profession by people who don’t actually have expertise in any of them. You’re not in the guild, you don’t get to appoint to the guild, and you certainly don’t get to gatekeep who can apply for the guild.
Edit: Dropping slurs when someone compares your views to eugenics is an interesting strategy. Shouldn’t you be at a meetup discussing Curtis Yarvin’s work or something?
"Abstraction ceiling" is just a way to talk about intelligence at at the tails that reveals one of the difficulties / limitations you can encounter when it comes to compressing / abstracting complex mathematical objects. Your objections to the term are facile and clearly stem from an obviously unvocalized intelligence denialism that is simply indefensible today. Also, intelligence != IQ, obviously this is too simplistic.
Everything about your arguments and other posts is similar reductions to retarded extremes (our only options are "eugenics 2.0" or deranged intelligence denialism - there is no room for anything in between, e.g. the idea that base intelligence matters and sets a hard average ceiling on potential, but that effort and other factors might push one slightly above/below this ceiling relative to others with a similar intelligence, and etc). Or alternately you hallucinate things I never said or even remotely implied (e.g. we should gatekeep based on dumb psychology metrics or hand sizes).
Just be honest: you know intelligence is real and matters, but you want to dance around this fact because you find it ideologically inconvenient, or you can't admit you yourself have limits (and lack the courage to realize the obvious social broader consequences of this personal admission).
Dropping slurs after being compared to a eugenicist is an interesting choice. Maybe you would be more comfortable in the comments of Curtis Yarvin’s blog. You’d certainly get less pushback there.
Falling into deranged ideological projection is also an interesting choice - one I chose to mostly ignore. You seem to really obsess a lot about this Yarvin fellow: I tried reading his stuff once and found it intolerable.
I imagine you think calling some of my language choice a "slur" here is some kind of gotcha, when the term I used is specifically one widely disputed as actually being offensive, given it is mostly used now to refer to normal people acting in intellectually deficient ways, and not generally to those with actual learning disabilities that deserve our sympathy. There are studies on this, which you surely are aware of.
If I had referred to your more deranged positions as "smooth-brained halfwit extremes", you likely wouldn't haven't tried to impotently pull this "slur" card, even though the semantics are basically identical. Which basically goes to show that you value irrelevant surfaces over substantial realities, and frankly is perfectly consistent with the midwit intelligence denialism on display in your posts in this exchange.
> This sounds a lot like you may have in fact succumbed to your abstraction ceiling
It sounds more like you're turning a vibes based theory into a tautology.
Hofstadter struggling with math for the first time in graduate school isn't a unique story, nor is his self introspection about this event a good basis for an apparently unfalsifiable theory about human cognition.
We have mountains of evidence that humans differ dramatically in cognitive potential, and more again that often effort / practice can only explain a small amount of the variance in performance in a wide variety of fields. We have basically zero evidence at all that anyone can just learn anything if they try hard enough under the right teacher, and plenty of evidence to the contrary.
Abstraction ceilings are about rates and difficulty of learning, so even if we assumed the (absurd) claim that no one has any fundamental cognitive limits, until we are immortal, being slow enough still creates an effective ceiling.
Intelligence denialism is the incoherent and indefensible position here.
That stuff always gives me such a eugenics 2.0 vibe — no, no, the hierarchy is based on innate cognitive ability now. Gives me the creeps that they're actually in academia pushing that stuff.
I think it is worse than that. Science is a process for resolving disagreements, ambiguity, and uncertainty, and also for discovering abstractions (patterns) among phenomena. It is social and can not be reduced to a binary / digital file, as it is dynamic and ongoing, and, fundamentally, exploratory.
Software is a static program and basically none of these things.
Software development is kind of like science, in some ways, in that you discover abstractions and patterns, and this requires resolving disagreements and ambiguity between you and your users, but in the end, the user demands are usually fairly concrete and specific (though no one may know how to express those demands precisely, initially), and the process is not really exploratory in the way science is.
I actually disagree that software is a static program. If a program always have the same behaviour, it's a pretty bad program. You need inputs and, as demonstrated by the recent LLM inputs, parameters you can tune to correctly address a specific context.
I would also contest the point about science being social. To me, science is the interplay between social constructions and physical reality. Which is why tests and experiments are so fundamental. Your point about software development as kind of like science is spot on: why not be exploratory/empirical in software development? Couldn't we imagine an LLM, say, that would go out and try to solve a question we frame? This seems strikingly close to autoresearch [1]
> If a program always have the same behaviour, it's a pretty bad program
This is false in far more cases than the cases where it is true, and even in the cases where you are right, you still usually want similar behaviour.
> You need inputs and, as demonstrated by the recent LLM inputs, parameters you can tune to correctly address a specific context
You seem very confused about what "static" means here.
> I would also contest the point about science being social. To me, science is the interplay between social constructions and physical reality.
Well, almost all philosophers of science would disagree with you, and IMO this sentence immediately contradicts itself.
Frankly, you should really work on learning to write more coherently and carefully. You maybe have some good ideas, but they are being communicated extremely poorly and inconsistently.
> Science is a process for resolving disagreements, ambiguity, and uncertainty ... It is social
More dramatically stated as: Science progresses one funeral at a time.
> An important scientific innovation rarely makes its way by gradually winning over and converting its opponents. What does happen is that its opponents gradually die out, and that the growing generation is familiarized with the ideas from the beginning.
Science should be more like this, in current times, yes.
But until much of academia is burned to the ground, or until science can be properly separated from modern academia, this will never be so. The current academic incentives are all wrong: low-quality research is rewarded and results in publications, whereas high-quality research (that takes time, and usually reveals that most exciting publications depend on p-hacking or other highly data-dependent analyses and selective presentations) is not published or actively blocked during peer review.
So instead you get BS arguments about how data can't be released for various privacy concerns (when in reality the vast majority of most datasets are trivial to scrub of identifying factors, and even in more complex datasets where you need to consider k-anonymity, it is still trivial to release data that allows replication of core analyses), and academic science is increasingly irrelevant unless it is tied to tech and industry, where producing junk actually has real negative economic and personal consequences.
I don't know what world this article / post lives in, but it isn't the messy world of actual reality.
It's changing, though, and articles like this are important. TFA is arguing for change in the future, not presenting this as a fait accompli.
In the 27 years I've been in academia, I've seen a lot of progress in data openness (NCBI GEO was a game-changer) and FOSS analysis software (it's now widely expected that a high impact pub will make all data and code available for review, and then publicly available upon manuscript publication; most major journals will not allow submission without this). It is becoming common for big journals to specifically ask reviewers to review the analysis code. It is starting to become more and more common for papers to release all the code used to generate all the figures (including supplementary figures)
There is still a long way to go, I agree. But it's always better to light candles than curse darkness, etc.
> academic science is increasingly irrelevant unless it is tied to tech and industry
While I have some sympathy with a lot of your bitterness, this statement is insulting silliness that a quick look at the list of Nobel Prizes in physiology and medicine would prove wrong. Almost all major breakthroughs in the applied sphere stem from decades of basic research that happened just because it interested someone.
> it's now widely expected that a high impact pub will make all data and code available for review, and then publicly available upon manuscript publication
I am also in academia and regardless, factually this is not true at all for data, not even remotely (less than like 10% of journals even have data availability policies which are recommendations, and in practice only a small percentage of papers actually make anything available), unless by "publicly available" you mean "available to some academics or academic labs after an often tedious and slow approval process requiring an academic email and various signed agreements". Maybe what you are saying is true in some very specific domains (e.g. machine learning research), but in general what you are saying here is IMO wildly out of touch with present realities in the vast majority of fields, but especially those involving human subjects.
> While I have some sympathy with a lot of your bitterness, this statement is insulting silliness that a quick look at the list of Nobel Prizes in physiology and medicine would prove wrong. Almost all major breakthroughs in the applied sphere stem from decades of basic research that happened just because it interested someone.
Nobel Prizes are so rare they don't speak at all to the generalizations I am making here. Also, much medical academic research is arguably successful because it is in fact ultimately industry-funded or tied to industry. It is of course though highly dependent on the academic subfield, for sure, and I was painting with a broad brush.
If I had to narrow things, STEM academic research isn't so bad, so long as we exclude social science from STEM. Much social science research needs to be defunded ASAP. And I'm not claiming industry research doesn't also have warped incentives. But, on balance, I'd wager outside of pure math/physics and certain more algorithmic/pure domains in comp sci, the smartest people today are going to choose (and be found in) industry, not academia.
Well said! And thank you for recognizing the effort.
The important part here is, as you say, to light candles and insist on rigor. Coincidentally, history tells us that that also gets us further. So by pure memetic selection, this strategy should win
"Burning academia to the ground" is a terrible idea. We need to fix funding and incentives. If funding and research all happen in industry, where are the incentives to do basic research?
Your analysis completely ignores the physical and biological sciences, engineering, and the humanities. It mostly applies to a small subset of academic fields in the social sciences and medicine. You're also ignoring the changes that have happened since the replication crisis. Preregistration, publishing all code and data, reporting null findings, replicating results, etc. are becoming the norm.
Of course "burning it to the ground" is rhetoric and not meant literally. It is meant to convey though that "nice" and "gentle" solutions might not really be enough here.
Yes, for the most part the fixes have to be in terms of funding and incentives. Funding needs to be more careful, and more careful funding can be a carrot rather than a stick here.
Re: incentives, IMO we clearly need a stick: there need to be harsh negative consequences for engaging in degenerate research programs and methods that have clearly been shown to result in pathological or cargo-cult science. Null-hypothesis significance testing is one clear practice that needs to go, but building entire fields on phony / meaningless uncalibrated metrics (think: a lot of self-report instruments that are never properly calibrated to objective outcomes or real-world behaviours and/or consequences, with results being reported only as standardized effect sizes) are another more pernicious practice permeating far too many fields. Ideological bias also needs to have funding consequences. Replication issues are still only surface problems in many fields, where the research would still all be worthless even if it replicated 100% perfectly.
> Your analysis completely ignores the physical and biological sciences and the humanities
I admitted later to painting with a broad brush, and yes, it is always hard to generalize and cover everything fairly. But IMO humanities has serious ideological and methodological rigor problems as well, and is overdue for disciplining. I would tend to have stronger positive feelings toward the biological sciences generally, yes. Yes, the social sciences are the major source of the problem (in part because they are so bad they tarnish the reputation of all academia).
> These fields have gotten a lot better over the past decade in the wake of the replication crisis. Preregistration, publishing all code and data, reporting null findings, replicating results, etc. are becoming the norm.
IMO "a lot better" is subjective, and I don't see those things as being the norm yet (beyond as lip-service), and the rate is far too slow. I agree we'll get there eventually, but I am worried about the loss of public trust and thus the production of real knowledge if we don't try a bit harder at this. Plus, globally, countries like China do seem to be more willing to actively crack down on research misconduct, at least in the past years, and it might not be unrelated to them increasingly pulling ahead technologically in many areas.
Hear hear! There are so many obvious improvements to how almost everything is done. For instance, in medicine review articles as a class of articles largely represent a giant waste of time. RCTs flatten all their gathered data during publishing, summarizing complex trial data, which is gathered but never published, into a few numbers. Then review articles take a bunch of flattened data, discard the articles that don't fit the exact question they are reviewing, and then publish a doubly flattened conclusion. If any of the included articles turn out to have flaws, if treatments change in retrospect, if you are looking for the answer to a slightly different question or you are looking at a different subgroup, then the review is useless and has to be repeated.
All of these tens of thousands of man-hours could be replaced by a few GitHub repos, if only RCTs would just publish their damn data. Then you could just run and rerun the statistics on whatever subgroup you're looking for, instead of combing through decades of review articles answering slightly different questions, looking for the answer between the lines. With LLMs making mining of large scale datasets almost trivial (with the process most likely becoming trustworthy within a few years), the current status quo is looking more and more antiquated.
If you want to be even more radical, hospitals could just publish their data continuously. Of course, it is easy to point to the risks of doing so, but what's often ignored is the benefits. It is hard to overstate just how many medical mysteries a hospital encounters on a daily basis, how much unknown we are navigating in practice. The current norm is that 99.99% of these cases are never published, and are only ever thought about by a small group of people who happened to be at work. Particularly, when someone dies of something no one figured out, it is never published anywhere, because even if you tried it is not interesting reading material for a journal to publish. And no one ever tries because they're scared of being called out for a mistake. A hospital is essentially a continuously running and extremely interesting experiment, where 99.99999% of all results are thrown in the garbage, and the only published data is subject to extreme selection bias.
All of this could be different, and the risks involved are actually quite small in practice. It is easy to automatically anonymize data quite well, but extremely difficult to absolutely guarantee that it is anonymous. And since current ethical norms are extremely averse to any degree of risk, and usually entirely ignore potential benefits, we all suffer for it. It is not entirely unlikely that someone reading this post will one day die because of something that could have been prevented, had things been different.
This seems to be strongly US-centric. In other (welfare) countries, publicly funded registries are anonymized and made available to research. For every single case. Of course, there are tons of data we don't see, but that shouldn't be an argument for not trying. The 99.99% unpublished cases is because our models/explanations/knowledge can't efficiently condense the medical mystery into a diagnosis code.
Everything can be prevented given sufficient knowledge. That's not the point. The point is how to prevent as much as possible.
Yup, strongly agree with all of this, especially the RCT stuff.
This has all been profoundly obvious for at least well over a decade or even two now. A consequence has been that too many serious people are driven away from academia and research, to the detriment of science generally.
I've no idea what to do about all this, because people have voiced obvious and easy solutions for decades, but they are all routinely ignored.
I think political lobbying for legal changes might be the only realistic pathway, in that current legislation (eg. GDPR in the EU) is extremely punitive even for minor violations.
I personally am trying to float using local LLMs to create anonymized case files and auto-suggest publishing cases in my hospital, which knowing how things work will probably never amount to anything.
Or if you want to float truly insane ideas I guess you can shop around with blackhat groups and see if anyone has stolen some juicy records/data during all the ransomware attacks and databreaches over the years, and do some rogue scientific publishing. Obviously that's crazy, but I have to admit that the notion of pirate scientists plundering and publishing data is hilarious to me.
Make analysis code available. Make anonymized data available for download without people having to jump through hoops to get it. If you have highly sensitive data, release only the variables or other statistics needed to reproduce core analyses. Don't only do garbage null-hypothesis significance testing or statistical analyses on the full data, also do ML approaches were you have to actually show your analyses replicate on held-out subsets, and report this. Make reviews open (anonymizing as needed) so we can see when biased or incompetent reviewers are blocking good publications. Allow public review (or at least broader academic open review, in some form), since it is no longer defensible to delegate review and decisions to one or two random people that just happen to be emailed and have the time / are on some editorial / review board. Also allow public post-publication review. Publish null findings / results, if only in minimal forms so we don't waste time and money trying to reproduce garbage. Make articles available and don't charge insane article processing fees or open access fees of thousands of USD (especially since hosting fees are not that crazy, and also because journals don't do any of the formatting work half the time anyway, and make academics or RAs or students do all the typesetting and formatting, even though now this could all be automated with template files, mostly).
Most of these things are easy to do for the majority of papers, especially in the past 20 years with the internet and modern tech and software. Plenty of frameworks exist already that have done most and/or at least some of these things, but, collectively, academia is decades behind overall.
I agree that a lot of the practices in academia are misaligned with the original goal. But can't you say that for other systems/institutions as well?
Point being, what about keeping the scientific method as the north star - as a good heuristic to avoid BS arguments and awarding low-quality research. And, crucially, to stay sane. My post is pretty naive, but I stand by the ideal of pushing knowledge as reproducible models.
Of course, every area has similar issues. The main unique problem with (contemporary) academia is the one I mentioned:
> tech and industry, where producing junk actually has real negative economic and personal consequences
In academia, you can just endlessly produce low-quality garbage, and basically make a career out of this. In industry, things more often eventually at least have to work and survive contact with reality. Academia mostly lacks this basic check.
The scientific method should be the north star, sure. Much of what is happening in academia is cargo-cult / degenerate / pathological science though.
Rather, "nine times larger" means multiplied by nine, and "nine times smaller" means divided by nine. This is basic and not particularly awkward, certainly not more so than e.g. positive/negative correlation, or many much more awkward and more common linguistic constructions, IMO.
If you have to edit out words (i.e. context) to argue a phrase doesn't make sense... I am not sure what mental model you have for natural language, exactly, but it certainly isn't a very robust one.
IMO the divide here between pure vs. applied math feels a lot like the divide between those who enjoyed coding for the understanding it led to, i.e. the writing itself was the joy, vs. those that primarily coded for the results. I enjoy the creative part of coding, the thought of software jobs just devolving into writing specifications and doing code review very much kills it for me.
reply