Rebuilding a system that only worked because people remembered how

When I took this on, the system worked, but only because a handful of people remembered how to keep it working.

It was a few dozen Windows machines running about a dozen different antivirus engines, scanning a constant stream of malware samples inside an isolated network. On paper it was a pipeline. In practice it was a collection of hand-built virtual machines, manual recovery steps, and knowledge that lived in people’s heads and a few scattered notes. When something broke, and things broke, recovery meant remembering what someone had done by hand, months earlier.

If you don’t run malware scanners, the shape is still yours: a process that only works because a few people remember it, that breaks when they leave, and that nobody can rebuild from a written description. Swap the antivirus fleet for your quoting process, your intake, or your fulfillment. Same problem, different domain.

The real problem wasn’t any single failure. It was that the system was illegible. You couldn’t look at it and know what it was supposed to be. You couldn’t rebuild it from a description. You couldn’t tell a genuine fault from a misleading symptom, because there was no baseline to compare against.

So the first work wasn’t to fix anything. It was to make the system describable.

• Turn the fleet into inventory. The set of machines, their network placement, their engine assignments, and their image versions became a tracked, reviewable record instead of state someone kept in their head. Standing up a machine became something you could review before it happened.
• Make every machine build the same way. Instead of mutating a virtual machine by hand until it worked, base images and per-engine images were built from scripts, so the same inputs produced the same machine every time.
• Give each engine a contract. Every antivirus product misbehaves in its own way: install quirks, self-protection, its own way of taking updates. Rather than rediscover that each time, each engine got a small, documented contract: how to install it, configure it, validate it, and keep it fed with fresh signatures, the daily list of known threats, without letting it delete the very samples it was meant to scan.
• Write down how it fails. Every real incident became a catalogued failure mode: what broke, how far the damage spread, how you’d catch it, and how you’d recover. Several of those became automated checks, so the system caught the problem before a person did.
• Find the actual constraint. When throughput was the complaint, I treated the build-and-deploy workflow as a production line and looked for the one step everything queued behind. Every engine was waiting on a single shared build line, and that same slow line was being used to test small changes. The fix that paid off first wasn’t to make the line faster. It was to stop rebuilding the whole machine just to try a one-line change, which cut the edit-and-check loop from many minutes to seconds. Widening the shared line itself was only partly solved; the durable win was separating the fast test path from the slow publish path, so a small change no longer had to wait in the full queue.

One stretch shows why this mattered. For about three weeks, machines across a whole product line kept dropping offline and rebooting themselves every few hours. Every sign pointed at failing hardware or a corrupted image, so that is what people kept chasing, and every fix was aimed at the wrong thing. The real cause was a leftover piece of an old software upgrade that had never been fully removed, quietly forcing the reboots. It was made worse by a cleanup job that would restart a sick machine but never mark it well again, so a single hiccup could snowball into days of the whole line looking dead. Once the system was legible, the fix wasn’t a heroic recovery. It was one check, baked into the build, that fails the moment that leftover piece shows up again, so a problem that once cost weeks of chasing cannot come back the same way.

None of this required inventing new technology. It required making an opaque system legible, and then improving the parts that the clarity revealed. The valuable output wasn’t a tool. It was a system other people could reason about, rebuild, and recover, instead of one that lived or died on who happened to be in the room.

That last part is the one that matters most, and it never shows up on a status report. This system had been handed down like an apprenticeship, one keeper at a time. The people who built it and kept it didn’t fail their way out, they got promoted, each in turn, into engineering leadership roles elsewhere in the company, and none of them has touched it in years. By the time it reached me it rested on one senior person who had carried it for over a decade, and the plan was the traditional one, he would teach me its quirks and I would become the next person it depended on. The fix was not to make me the next keeper. That would have just started the cycle over for another generation. It was to get the knowledge out of any single head and into the system itself, so the load on the people who had been carrying it gets lighter over time, and anyone can pick up the rest.

The benefit is hard to put a number on and easy to feel. And the work that used to eat months now moves in days: standing up a new scanning engine, once careful hand-work nobody wanted to touch twice, is now a reviewable change you can audit and trust. The migration isn’t finished; what I’ve described is the part already standing.