Puzzle Solvers and Gardeners#

When I interviewed at my current employer, some years ago, I solved puzzles for about three hours. The other three hours were spent chatting about who I am, how I see myself, and what I’ve done. But the focus was on the puzzle solving, with all but one interviewer asking me to solve a puzzle. The solutions needed to be expressed in code. And as I interviewed other prospects, I made them solve puzzles as well. I thought this might give me some insight into their ability on the job. I have a mere fourty-five minutes to assess a candidate’s ability.

We have optimized for puzzle solvers. We have trained job offerors and job seekers to focus on puzzles. That is fantastic, if, like me, you enjoy puzzles. Crosswords, geometric puzzles, non-Sudoku numeric puzzles, logic puzzles, etc. are fun to me. Sordid is one I don’t enjoy so I wrote a simple solver so I may never need to solve one again, ever. Those are more tedium than they are clever. I made an imperfect machine to solve the puzzles isn’t perfect. I had to choose between finite run time and finding all possible solutions. The maths indicate the algorithm can only have one property or the other. I chose the former, but a human might attempt the latter.

The puzzle solver is great when a project is new. How do we do something for the first time? When AWS, Microsoft, and Amazon were first building their clouds, they needed puzzle solvers. The first versions of the products are puzzle rich. Over time, features become stable and fewer new features are readily apparent. And the problem of maintaining these solutions becomes significant. It is both a tedious chore and a constant interruption for puzzle solving.

The puzzle mind defends itself by converting this into a new puzzle. The solution is to build a machine to maintain the machine. Much like my Sudoku solver, the machine isn’t perfect. At some point, maintaining the machine to maintain the first machine becomes the next problem. The puzzle mind discovers, or even invents, new puzzles to solves and therefore new machines to build. Each machine doing its job imperfectly because it is a mere automaton. AI may expand the scope of these automatons, and maybe ease their construction, but it is not a substitute for a mind.

The risk is the puzzle solver creates an incomprehensible contraption. One wheel or cog breaks and the entire series of solutions grind to a halt. The breakdown can have catastrophic consequences (like a large regional outage or data loss). Greater automation and efficiency can accelerate the breakdown. As each automaton is often incomplete, or imperfect, the results become increasingly suspect. For example, false negatives on “critical” on automated alerts causing alarm fatigue. As the system becomes more a collection of machines it can appear brittle, inflexible, or downright hostile.

The gardener eschews creating a new machine in favor of direct engagement. The gardener doesn’t look at their yard and think the regular work is a problem to be automated away. Tools are the means of doing the work efficiently rather than a way to avoid gardening. Hands in the soil is a requirement to understand how to properly care for the garden. Process documents, trouble shooting guides, best practices, and so on help the Gardner structure routine work. A gardener routinely looks over the landscape for any troubling signs of needed upkeep. They are better at dealing with unique problems or unusual circumstances. They have the luxury of applying a mind to the the work, that can explore in a way an algorithm cannot.

As a rule, gardeners are not as efficient as puzzle solvers. As the system doubles in size, a commensurate number of gardeners would need to be hired. But perhaps few, if any, new puzzle solvers are needed. And gardeners have their own flaw, which is failing to fix underlying problems or exploit automation. As they build a more perfect system of maintenance, over a sprawling landscape, they can do needless, routine work. A company of gardeners is as troublesome as a company of puzzle solvers. But as people in the loop, they help keep the system flexible, resilient, and servicing the needs of end users.

On the first day of development, a project is largely all puzzle solvers. As the system matures, along side puzzle solvers, there needs to be an increasing number of gardeners. Traditionally, this was the divide between “development” and “operations” or “support.” There are fewer easy puzzles to solve and many more problems that fall outside the of the capabilities of the machines. Monitoring, control, updates, and so on are ongoing tasks. Like raking the yard, mowing the grass, or trimming hedges. It creates neatness, order, and adds value to the property.

The mixing of the two into “devops” has only anecdotally succeeded. At best, it seems to spark puzzle solvers to discover the rough edges in their solutions. With “on-call” as a system of collective punishment to force improvement of the machines or machine tending machines. Or perhaps to save money by hiring gardeners who can also solve some problems. Which leads to more crudely built machines, by individuals possibly focused on the wrong part of the problem. For example, instead of fixing an underlying complex algorithm, learning to ignore the problem and masking it in “necessary” toil. At best you find “mechanics” who are actually puzzle solvers over the domain of the machine as opposed to machine design. But in the end, people seem to place themselves in roles that are either primarily gardening or primarily puzzle solving.

I think this divide, and the fact we’ve favored puzzle solvers, helps to explain the current IT landscape. If I were to start a new project, I should start with requirements. Gathering together all the needs and wants is a gardening task. It involves talking to human users and wading through sometimes contradictory answers to the same questions.

Those are rarely interesting puzzles because human needs and desires are rarely solved through logic. The more interesting puzzle is working out the new infrastructure. How can I go from committing a change to some codes and those codes running on an infinitely scalable platform. One that automatically heals itself at the first sign of trouble. So that the system is continuously available? And I am constantly aware of my costs, incidents, or intrusions in real time.

The short answer is on day one, I don’t need any of that. Most sites could start off as an “all in one” (monolith) application, on a VM, with a plain-vanilla database. The fact they have more micro-services that customers, or that they’re overspending for power they don’t need, is lost on them. Their architects are puzzle solvers. They were hired by other puzzle solvers or by people who believe puzzle solvers are the best for the job. For a while they will look at the beauty of their solution. And then they will find new puzzles to solve in those context, or new solutions to solved puzzles. But to them it’s the solution that’s beautiful, not what they create for other people.

A system composed of a Google spreadsheet and a static web site is the product of a gardener. This is the efficient effort the needed to create something they can manage. A few charts, reviewed manually, alert them to any changes they need to make. It can grow quite a while on this simple configuration. This is fine, until it’s time to make a change. Then there is a risky, expensive re-write of the system to a more resilient, scalable platform. Little or no new functionality, but a lot of time and expense. Clearly a problem, but not necessarily a “day one” problem.

Of the two scenarios, we seem to see the former more than the latter. If I were to listen in on a 10 random software developer interviews right now, I would guess 8 or 9 are solving a puzzle. This results in the classic Silicon Valley parody of a founder solving a non-existent problem. Or solving for problems that normal people, in the real world, do not find are actual problems. Gardeners may make “bad” choices from a technology standpoint, but they deal with reality.

And that’s why we need to stop focusing on our puzzle solving skills and spend time gardening.