Using an LLM "off-the-grid"

From the moment that OpenAI launched, I was very curious. I had met one of the founders, Greg Brockman, a few years before and he had given me a lot of time and attention. He had reached out to me, not the other way around, and we had a delightful conversation about the early history of Stripe, where he was founding CTO.

When ChatGPT launched, I was naturally curious, and very excited about it. I, along with the rest of the world, got a little bit caught up in the hype about the capabilities of these new AI systems, and certainly enjoyed typing things into the box and it responding in a human-like fashion.

A few months later, I stopped using it. In general, it was inferior to just finding information myself. For coding, Cursor was annoying, and it slowed me down more than it helped. GenAI was clearly an extremely impressive technical demo, but I didn't feel that it was actually making me any more productive.

The next moment where the constant, incessant, unavoidable buzz buzz buzz of AI hype hit me was Claude Code (again, me and everyone else, particularly in Silicon Valley). Here was an "agent" (hate the term) who you could give a code problem to and without directing it to the files themselves, it would figure out what to do.

At launch, Claude Code was again very impressive. It was able to figure out what to do and where to do it, and I put a good bit of time into asking it to do things in my day-to-day work. The problem was that while it was good at finding the right files, and was able to solve some problems, it wasn't particularly good at solving them properly. I found I was doing as much cleanup as I was saving time with the computer writing the code. I pretty much gave up on Claude Code at some point last summer.

Then came the autumn, and a big project arrived on my desk that had a few particularly gnarly challenges. The project required features that I knew that I personally didn't really have the engineering aptitude for, but I believed would make Tito better. With a deadline fast approaching, I described the high-level problem to Claude Code and the recently launched Opus 4.3 (I think), and while the code it produced wasn't the prettiest, we delivered the project bug-free and ready to rock, on time. Something had shifted.

Since the arrival of Opus 4.5 a few months later, I haven't hand-written a single line of code. I haven't talked publicly about it because I've held some shame about it. The bait-and-switch non-profit to heavy-profit motives of OpenAI spooked me. The weird god-complexes of the leaders of some of these companies. The profession to be "open", and "ethical", but in reality being very closed. The selling of technology that works some percentage of the time and then the other non-zero percentage of the time can delete your production database. It's all a bit shady. Some of my best friends have decided that AI is simply bad, and not worth even looking at.

I don't have those beliefs.

Nor, however, do I subscribe to the hype. There are people who believe that an AI takeover of the world is inevitable. To me it seems they would idly sit by as we become vassals of companies run by people you probably wouldn't trust to mind your dog let alone your kids. There are people (including me) who are happily feeding data, connecting a bunch of private services to these two US firms, just for convenience.

And I get it: speaking to a computer like a human and it responding is one of the most joyous ways to interact with a machine that I've ever experienced. My whole life, I've had to conform to how machines communicate, as programmed by someone who is not me. Now the machine conforms to how I communicate, and adapts to my preferences. It's science fiction in real life.

The past few weeks, I have gone very, very deep down the rabbit hole. I've spent quite a lot of time (and tokens, 🤮) pulling the slot machine lever and refining the output. (Although not as much as some idiots). The two themes of my lever pulling have been: "if people insist on using AI, I want to see if I can uncover ways in which it can be used effectively, efficiently, and for the betterment of communities, not for individual gain". The other theme is: "no engineer would be stupid enough to do this". I believe I have uncovered a lot of interesting ideas worth sharing. I'll post about those separately.

For today though, I'm writing this in a text editor. I'm about to feed it in to an LLM, and to the best of my knowledge, it will be published on my very new and very plain website at paulca.com. [Update: it worked! I'm now going to send these edits to a slower but more capable model after some "tuning" and my expectation is that I will paste the changes and the next thing I'll need to do is refresh my browser]

The kicker is that doing that will not pass through any US-owned-or-controlled LLM provider. While Anthropic, and particularly Fable, remains best-in-class at software production and I have discovered a few tricks to make very nice software very efficiently, other open source models exist that can run very fast on hardware (ergo: GPUs [and more slowly, on CPUs]) that until a few years ago would have been considered a commodity.

So that's all I have to say for now. A first blog post about AI, hand-written, but posted using a robot running in a home-office in South Dublin, powered by an open source model, where none of the data leaves my control or my jurisdiction without my consent, ironically landing on GitHub, a company whose services I once loved dearly, made by people who became some of my closest friends, that I hope to soon leave behind and never look back.

Let's. Fucking. Go.