Care: The missing ingredient

There are loads of engineering ideologies that you can sign up to if you want to make awesome stuff that's actually good.

Here are some that I've tried:

• Formal Methods²
• Languages with chonky expressive type systems like Haskell
• Languages un-encumbered with significant type systems – like bash, perl, etc
• Languages with middling "strong dynamic types" like python
• Object-oriented programming like java
• Functional programming like haskell, or lisp
• Agile methods like XP/Lean, TDD, and Pair Programming³
• Document driven and/or literate programming

Many of these ideologies run counter to each other – you usually can't have both permissive perl-style types and expressive haskell-style ones for example – but I assert that they can all work. By this I mean that I have seen them all work at least once.

However, I've also seen them all fail. I've seen every one of these methods result in horrible, broken, impossible-to-maintain code that no-one likes working with or using.

It's tempting to think that this is a case of horses for courses – that some tools are better suited to some problems and others to other problems⁴ – but I recently noticed that most of my favourite software doesn't use any of these methods.

• OpenBSD
• Slackware
• Emacs

I have long loved Emacs for its docs. The way every keystroke, feature, and function is documented; and the way emacs-lisp embeds documentation into the code itself. The way the info manuals are packaged up as actual digital books with indexes, hierarchical chapters, tables of contents, and search facilities. So good.^{5, 6}

Slackware was my first linux distro in 1995, and I'm typing this on a slackware thinkpad right now. While other linux distros always seem to be actively looking for new ways to get on my nerves, slackware has just kept working properly for me for 30 years.

But it was learning about OpenBSD⁸ that started me thinking along the lines of this blog post.

The OpenBSD code, and the OpenBSD system are both beautiful. They've been going since 1996, and they boast "Only two remote holes in the default install, in a heck of a long time!" That got my attention, but it was the act of actually installing and running a server that really impressed me. Everything is clear, and simple, and well-documented. Keeping my server running is lovely and boring. I don't have millions of services and dependencies I don't understand, and the software I am running doesn't often need updating to stay secure. When I looked at the code⁹ I saw well-factored, well-commented, easy-to-read code, with a smattering of well-considered regression tests.

OpenBSD doesn't use Formal Methods or Agile ones. They definitely don't TDD¹⁰. They use C – a programming language famous for footguns and CVEs with very little in the way of either type protection or artistic dynamically-typed freedom¹¹. There are no Objects, no First Class Functions, and while the code is often beautifully commented, there's no javadoc-like structured code-documentation. While the project as a whole has famously good manpages, it's definitely not the result of Knuth-style document-driven literate programming.

And yet, the result is really good. Outrageously so. And it has been for "a heck of a long time".

The missing ingredient is Care.

The OpenBSD folks know what they care about, and don't compromise on it. Every line of code is scrutinised, and nothing is shipped until it's ready. For them; auditable, well-documented, secure software (according to their threat models) is all that matters. They don't put themselves under pressure to release headline-grabbing new features on a regular cadence¹². They don't try to support all the latest graphics cards.

Agile (Lean/XP) people might call this "Product Vision", but I don't think most OpenBSD devs would know or care what the Agile folks meant by that. They don't have any PMs. They just know what they care about; and they're free to act on that care and ignore everything else.

I now believe that you need two things to write software that's actually good:

• Care
• Experience

I think that all the stuff about processes and tools and techniques are about distilling Experience into neat packages that are accessible for other folks. That's all good stuff. But for many of us, quite a lot of the time, it's missing the point.

If you Care, you'll take the time to do what you need to do to make things Good.

If you Care, you can make use of fancy maths, or debugging tools, or TDD methodologies to make things Good.

If you don't Care about making things Good¹³, then no tools or techniques will save you.

But most amazingly, if you do Care, and you don't have access to any fancy tools or techniques, you'll probably end up making things Good anyway. It might take longer than if you'd had a teacher who could show you some of their favourite neat tricks, but you'll get there in the end. Because Care eventually leads to Experience.

Ok, so it's simple. We just have to ensure that software is written by people who Care. Right?

I think the problem is that our employers don't actually want us to Care about the things I'm talking about.

Veritasium recently produced a video on the XZ Utils attack that almost compromised all our OpenSSH deployments in 2024. One of the great things about that video is the way they showed the enormous lengths that Microsoft Linux developer, Andres Freund went to in order to investigate a 500ms delay in OpenSSH port connections. It kinda looks like he just stopped doing his actual job for a few days, and doggedly chased the issue that had nerd-sniped him. And then it turned out that the issue was a massively important industry-wide attack, and he'd thwarted it.

Ok, so this is a beautiful example of one of the wonderful things that can happen when we're guided by Care. Andres cared about a performance puzzle, did the right thing, and saved the day. Afterwards, his CEO gave him a special award. Yay.

But do you think this is the only time Andres has gone AWOL to chase a little detail that only seemed important to him? How many days do you think he spent in 2024 "gold plating" other things that his managers really don't care about?

If the MS CEO was able to wave a magic wand and make Andres spend all his work-time shipping features that Microsoft customers were actually asking for, do you think he'd hesitate to sacrifice the tiny possibility of saving the linux-server world?

I think it's awesome that folks like Andres can exist in corporate eco-systems like Microsoft, and still take the time to make software good. But I don't think shareholder-bound companies are usually a nurturing environment for those folks. I think for every day when someone like Andres takes the time to chase quality, there are 100 days where equally virtuous engineers "do what needs to be done" to survive this quarter's earnings report.

I think most companies have to pretend to want to make the best most robust most reliable software in the world. Because if they don't preach that party-line, then no engineer-who-actually-cares will want to work with them, and no customer will want to buy from them. But if they can get away with it, they'd rather spend less on engineering the quality, and more on whatever it takes to capture more paying customers.

I think most of the time, most companies don't want us to make stuff that actually good. They want just barely good enough.

Back in the late 90s, when I was a naive idealistic student¹⁴, I really believed in a kind of nerd-utopia. The Internet would let us all talk directly to each other without power-hungry governments and corporations distorting our messages, so peace and understanding would flourish¹⁵. And the software that mediated this beautiful new world would be written exclusively by Nerds Who Care. I believed that computer programming was a skill that demands real passion – intrinsic motivation – to master.

I thought that if you only want money and power-over-people, you'll never have enough motivation to learn to code. So the people who build our online world would be idealistic hippies like me.

Obviously, I was wrong. Here are two of the ways I was wrong:

1. You can become a pretty damned good programmer even if you're "only" motivated by money¹⁶.
2. As discussed above, most software doesn't have to be "good" in order to capture the market and make money.¹⁷

But even though the Internet isn't a magical nerd-hippie utopia, that doesn't mean there aren't loads of idealistic nerds out there writing beautiful stuff. Let's celebrate it, use it, teach each other about it, and maybe occasionally contribute to it.

If you want to know more about how I'm trying to use Beautiful Software to make my Internet-experience better, have a gander at grumpystack.uk