Shovels, Chatbots, and the AI Bubble Nobody Else Is In

I talked to a pharma executive last week. Big company, UK based, he runs things from what I think is the Middle East control room in Jeddah. Senior guy, more than a decade in the industry. I asked him the obvious question, how is he using AI day to day.

He does not, not really. He uses Copilot to proofread emails and juggle the occasional idea, because Copilot is the only thing his work laptop will let him touch. He told me he thought about running a second laptop off the company firewall so he could actually explore what is out there, but keeping two work laptops was too much of a hassle. So he stayed on one laptop, inside the firewall, and stayed a chatbot user.

That is not a story about one lazy executive. That is a story about an entire class of sectors, heavy IP, heavy security, heavy regulation, where the friction of adopting anything beyond a sanctioned chatbot is high enough that the frontier simply does not reach them yet. Pharma, medicine, anywhere production touches actual human bodies, the API surface that AI could plug into is real, but so is the reason nobody is rushing to open it up.

The bubble is smaller than it feels

Here is the part that actually stuck with me. This man has never heard of Noam Brown. He does not know that a team of roughly ten thousand AI agents at OpenAI apparently cracked a Navier Stokes Millennium Prize problem in under four days using recursive self improvement, a ninety year old open problem in fluid dynamics. If he has heard of it, it reached him filtered through three layers of misinformed retelling and a headline, not the actual claim.

Meanwhile on X, on the timeline I live on, this is the entire conversation. GPT this, Claude that, Grok, Astra, whichever model dropped this week, whoever is winning the benchmark this month. It feels like the whole world is watching. It is not. More than ninety nine percent of people are living their entire working lives completely outside this bubble. They are not ignoring AI out of stubbornness, they simply have no reason yet to know it exists at the pace we talk about it.

That gap is not a footnote. It is the whole reason I am writing this.

I do not fully trust either side

I want to be honest about something uncomfortable. Some corners of the open source AI world feel cult like to me. Not all of it, but enough of it that I notice. I have never believed in anything hard enough to dedicate an account or a life to it, so maybe I am just missing whatever pulls people in, but I see people treat a lab or a movement the way other people treat a religion, and it makes me uneasy.

At the same time, I think the hate directed at Anthropic and OpenAI is often exaggerated, and I do not agree with most of it. Both things are true for me at once. I do not think large firms deserve love or blind enthusiasm, they are profit making machines and I should take what value I can from them without expecting them to act ethically on their own. That does not mean they should be free of scrutiny. It means the scrutiny should be accurate, not performative.

Nobody selling you a direction is neutral

Here is the part I actually find beautiful, in a cynical way. Every direction in this fight has a financial backer. None of them are as clean as they present themselves.

• Closed source labs are incentivized to hype their own capability, because the numbers before an IPO and the numbers after an IPO both need to justify the valuation. Their fear mongering about competitors, and sometimes about their own models, serves that story.
• Open source is heavily backed by the people selling the hardware that open weights run on. Nvidia, and the broader ecosystem of chip and infrastructure sellers, benefit every time more people are running local models instead of renting tokens from a closed API. They are selling shovels in a gold rush, and open source is the shovel buyer's favorite argument.

None of this means people on either side are dishonest as individuals. It means the incentive is always in the room, even when it is not moving the whole conversation, it is moving something.

The people getting hit from both sides

The funniest and saddest part of this whole mess is what happens to independent safety and security researchers. Closed source labs treat their proposed regulations as a threat to competitiveness, the argument that slowing down means handing the AI race to China. Open source treats the exact same regulatory push as a backdoor for regulatory capture, a way for large labs to pull the ladder up behind them.

So you end up with researchers who are, as far as I can tell, honestly trying to get ahead of a real problem, getting hated on from two directions that otherwise agree on almost nothing. Closed source labs do not actually support the parts of regulation that would constrain them, only the parts that box in everyone else. That inconsistency is exactly what makes the open source suspicion land so well, even when it is aimed a little too broadly at people who did not deserve it.

The question I kept circling without asking

I noticed something while writing this. I kept coming back to the same idea without ever saying it directly. If the market cannot be trusted to self regulate, and the labs cannot be trusted to regulate themselves honestly, and no government, especially not the ones actually racing each other, can be expected to slow down unilaterally, then who is actually supposed to hold this responsibly.

I do not have a clean answer. But I think the closest thing we have right now is independent voices, researchers and organizations that are not sitting inside a lab's PR department or an open source hardware seller's marketing budget. The Future of Life Institute is the example I keep coming back to, over a million signatures on their proposals, researchers making a case for proactive limits before anything catastrophic forces the conversation. I am not certain they are fully free of their own entanglements, financial or political, I genuinely do not know enough to promise that. But their stance, put reasonable stop signs in place before the explosion, not after, is one I find myself agreeing with even while closed source labs dismiss it as alarmist and part of the open source crowd dismisses it as a trojan horse for regulation they already distrust.

Dario Amodei has been fairly clear on this too, in his own posts, that regulation should target large firms and not choke out small open source developers. If he means that honestly, and I think he does, it is a more reasonable position than either side gives him credit for.

Where this leaves me

A pharma executive who has never heard of recursive self improvement is not an isolated anecdote. He is proof of exactly why this matters. The people running regulated, IP heavy, real world industries have no visibility into what is actually happening at the frontier, and the people who do have visibility are too busy fighting each other over incentives to give him or anyone like him an honest picture.

I do not think the answer is picking a side between open and closed. I think the answer is demanding more independent voices who owe nothing to either camp, and taking their warnings seriously before the gap between the bubble and the rest of the world gets too wide to close.