Please, stop making snowflakes

Something that's suprisingly common in my line of work is the snowflake server. It's a running installation of a server, usually set up by a wizard (a person who is no longer around and whose skill set was never replaced), which runs some irreplaceable process for a small business. This server is usually treated with a mixture of reverence and fear, and is entrusted to an operations person who doesn't know much about what's on it, but keeps it on life-support for years with the usual operations tactic of rigorous binary backup.

This server is, for me, a deep black hole of wasted time and opportunity. Because it exists and works in the here and now, nobody wants to invest in replacing it. Small businesses often run these - a consultant or transient rockstar hire sets it up, and there's no budget to have the work redone. At the same time, no one dares change it because it's in production and does work that pays bills. People higher up assume that because the server works now all is well. And because no one knows how it was set up, it can't be replicated, or developed upon.

Any time anyone sets up a server and hands it over for production without also supplying setup/recipe scripts, or at very least documentation about the actual software (include fixed version numbers and permalinks to procuring it), you know you have a snowflake, and if you're seasoned, you should start twitching.

In a perfect world we'd use some provisioning system like Pupppet, Chef or what have you to set all servers up, but alas that world is not here yet - many production machines are going to be built by hand. The very least you can do is use documention-based installation. Create a dry-run install by setting up your machine from scratch, documenting each step. When you're done, commit that document to source control. Then destroy your dry run server, and restart the entire process, but installing "dumb" by following your own documentation. Even better, have a buddy "proof" your docs by doing the actual setup for you, with bonus points for improving the guide.

There's nothing wrong with backups and images to keep your work alive, but these should be to save time, not as "source code". The tragedy is almost everyone knows how frustating a snowflake is, yet we struggle to break out of this short-sighted pattern of work.