A conversation between mathematicians on Hacker News today on solving a hard problem that one of them had been working on captured the core problem with the direction AI is headed.
“Several hours of work with Fable simply convinced me it wasn’t yet solvable and reinforced how hard of a problem it was.”
to which another replied:
“virtually none of this stuff is possible with technology any normal citizen has access to. I have no problem with AI models making revolutionary advances in math or science. Where I start to have a problem is when the AI models making these advances are tightly withheld, proprietary, and seemingly never released with these capabilities intact. This has been the case for all of 2026 so far.”
To summarize the journey so far:
Us humans wrote a lot of open source code, which we released with the intent that humans would use that code to produce more open source code. The GPL open source license, for example, is structured thusly. We also created and released a lot of new knowledge and processes as academic papers with the intent that other humans would have the opportunity to produce similar breakthroughs at a similar pace and scale.
Then a new kind of technology was discovered that could consume all that knowledge and produce an intelligence that is far superior to our own, and which can produce far better code and far greater mathematical and scientific breakthroughs than we can.
And that intelligence is now being withheld from us with a rationalization that it is for our own safety. The same mathematicians that produced the research that the AI learned from do not have access to the intelligence it produced, and may never have access, even if they choose to buy back access to the product of their own research. And those same programmers that produced the code that the AI learned from do not have access to the intelligence it produced unless they pay up to $500 a month and get and keep permission to access the best most capable models.
If we extrapolate this trajectory we’re on, we end up with a world where all of human knowledge has been assimilated into massively powerful AI models that are kept from the very public that produced that knowledge. And access is sold back to the public that created the knowledge, or simply never granted if it is more valuable for the AI company to produce and use the breakthroughs themselves.
In other words, humans produced math and science that AI companies have now trained on, and AI companies can produce new math and science that they keep to themselves to create even more powerful AIs that will produce all our future math, science, engineering, biotechnology and computer science breakthroughs. This applies to all fields including the creative arts.
It’s Monday morning in 2040 and you log into your cloud AI account to get work done. You know that you’ll never create programming that is as good as the AI you have access to. You’ll never create math, science or writing as good as the AI. Your job is to tell the AI what it should work on and what kind of outcomes you need. Program this, incorporate that bit of new knowledge another AI created. But behind the wall in the garden where the most powerful AIs are kept, work is being produced that crushes anything you will ever create with your human brain or your rented AI. You’re not part of that team, and you have no way of ever gaining access to that capability.
It’s Tuesday in 2040 and an ageing polymath sits down at their desk, angry and frustrated with the state of the world, and how everything we spent millennia creating has been stolen from us, and rented back to us, with the very best kept for the elite. They start writing a diatribe and get a few sentences in before they realize this will cost them the little access to AI that they have been granted. They will be deemed persona non grata by the companies that rent us the ability to think, and once they are PNG’d, their tatters of their remaining career and their ability to earn will be destroyed.
This is the end of all human intellectual accomplishment, if we continue to build this world. It is also the end of the freedom to accomplish. Until today you could sit in a room and think, and come up with breakthroughs that are profoundly impactful for humanity. Newton, Turing, Shannon, Einstein, Godel, Escher, Bach. We will never see their like because all of our breakthroughs will be sold to us by those who stole our thoughts and are renting us the ability to think.
If you have a thought you’d like to explore with an advanced intelligence, the AI company renting you that capability gets first look at what you are thinking, and can use AI in real-time to determine if it’s something that they should further explore with their more advanced models.
If you are a potential competitor, or if you have ideas that may harm their future profits, they may flag your account and revoke your access, in the name of your safety, of course.
This is the world we are enthusiastically creating for ourselves. And you should be terrified.
What is the solution? We need to shift the Overton Window to a place where it is acceptable for us to have a conversation about banning closed weight models. In other words, because we created the knowledge that AI companies have taken to train their models, and they therefore have a moral and ethical obligation to provide us with the models that are the product of our knowledge, and the models that those models produce. We are the giants whose shoulders they are standing upon, and we shared with them, now they need to share with us.
The current conversation is around whether open weights models should be banned in the name of safety. On the scale of where the conversation should be, I’d say that’s very far right, and shifting it back to the other end of the scale, where we discuss whether closed weight models should be banned, will be a battle. But I think it’s a fight worth having, and it needs to happen now, before the amount of consolidated power becomes so great that it’s no longer possible to have the conversation without becoming an outcast, and facing financial ruin and the destruction of your career.
If you’re a member of the Hacker News intelligentsia, you can help by considering where we are headed, and beginning to have this conversation on your blog, in person and in any other forum where smart thinkers who are building our future congregate.
Lets move the conversation from banning open models – which benefits AI companies and ensure the dystopian future I’ve described – to a conversation about banning closed models – which benefits all of humanity.
Addendum: Banning closed weight models may seem infeasible at first blush, so how would this work? Today the most powerful general purpose models are closed weights and access is arbitrarily granted or kept back, without any transparency on why we’re not given access. If open models are not just the norm, but a requirement, and a closed weight model is considered extraordinary, then it becomes acceptable to require regulatory approval to hold any model back from the public. Granted, this is vulnerable to regulatory capture and other failure modes, but it is a vast improvement from the inverse, which is what we now face, and are building.
Leave a Reply