Using an LLM "off-the-grid"

From the moment that OpenAI launched, I was very curious. I had met Greg Brockman a few years before and he had given me a lot of time and attention. He had reached out to me, not the other way around, and we had a delightful conversation about the early history of Stripe.

When ChatGPT launched, I was naturally curious, and very excited about it. I, along with the rest of the world, got a little bit caught up in the hype about the capabilities of these new AI systems, and certainly enjoyed typing things into the box and it responding in a human-like fashion.

A few months later, I stopped using it because in general, it was inferior to just finding information myself. Cursor was annoying, and it slowed me down more than it helped. GenAI was clearly a very good technical demo, but I didn't feel that it was actually making me any more productive.

The next moment where the constant, incessant, unavoidable hum of AI hype hit me was Claude Code (and everyone else, particularly in Silicon Valley it seems). Here was an "agent" (hate the term) who you could give a code problem to and without directing it to the files themselves, it would figure out what to do.

At launch, Claude Code was again very impressive. It was able to figure out what to do and where to do it, and I put a good bit of time into asking it to do my day-to-day work. The problem was that while it was good at finding the right files, and making a good effort at solving problems, it wasn't particularly good at solving problems properly, so I found I was doing as much cleanup as I was saving time with the computer writing the code. I pretty much gave up on Claude Code at some point last summer.

Then came the autumn, and a project arrived on my desk that had a few particularly gnarly challenges that I knew that I didn't really have the engineering aptitude for, even though I believed they would make Tito better. With a deadline fast approaching, I described the high-level problem to Claude Code and the recently launched Opus 4.3 (I think), and while the code it produced wasn't the prettiest, we delivered the project bug-free and ready to rock, on time. Something had shifted.

Since the arrival of Opus 4.5 a few months later, I haven't written a single line of code. I haven't talked publicly about it because I've held some shame about it. The bait-and-switch non-profit to heavy-profit motives of OpenAI spooked me. The weird god-complexes of the leaders of some of these companies. The profession to be "open", and "ethical", but in reality being very closed, and selling technology that works some percentage of the time and then the other non-zero percentage of the time can delete your production database. It's all a bit shady. Some of my best friends have decided that AI is simply bad, and not worth even looking at.

I don't have those beliefs.

Nor, however, do I subscribe to the hype. There are people who believe that an AI takeover of the world is inevitable and would idly sit by as we become vassals run by people you probably wouldn't trust to mind your dog let alone your kids. There are people (including me) who are happily feeding data, connecting everything to these two US firms, just for convenience.

And I get it: speaking to a computer like a human and it responding is one of the most joyous ways to interact with a machine that I've ever experienced. My whole life, I've had to conform to how machines communicate, as programmed by someone who is not me. Now the machine conforms to how I communicate, and adapts to my preferences. It's science fiction in real life.

The past few weeks, I have gone very, very deep down the rabbit hole. I've spent quite a lot pulling the slot machine lever and refining the output. (Not as much as some idiots). The two themes of my lever pulling have been: "if people insist on using AI, I want to see if I can uncover ways in which it can be used effectively, efficiently, and for the betterment of communities, not for individual gain". The other theme is: "no engineer would be stupid enough to do this". I believe I have uncovered a lot of interesting ideas worth sharing. More on that later.

For today though, I'm writing this in a text editor. I'm about to feed it in to an LLM, and to the best of my knowledge, it will be published on my very new and very plain website at paulca.com.

The kicker is that doing that will not pass through any US-owned-or-controlled LLM provider. While Anthropic, and particularly Fable, remains best-in-class at software production and I have discovered a few tricks to make very nice software very efficiently, other open source models exist that can run very fast on hardware (ergo: GPUs) that until a few years ago would have ben considered commodity.

So that's all I have to say for now. A first blog post about AI, hand-written, but posted using a robot running in a home-office in South Dublin, powered by an open source model, where none of the data leaves my control or my jurisdiction without my consent, ironically landing on github.com, a company whose services I once loved dearly, made by people who became some of my closet friends, that I hope to soon leave behind and never look back.

Let's. Fucking. Go.