devops.md

OK. Slightly - slightly - less half-baked thoughts about devops.

At the time of the Agile Manifesto, Agile's big competitor was the Rational Unified Process (RUP). It was a process that was tailorable to any situation. But it required tailoring to any given situation.

The intro to XP Explained was a breath of fresh air because it sharply limited its scope: small teams, requirements in flux. Then it said, prescriptively: such teams should do these N things, learn to do them really well, and leave all the "well, we're special because..." for later.

Devops-in-practice, it seems to this newbie, has zoomed right past the limited scope and "just do this" of XP and even Scrum-of-that-time to the infinite-but-required tailorability of RUP. For example, we're on AWS. Like skazillions of other smallish companies, the potential cost of lockin to AWS is of no concern to us. And yet I have a lot of trouble finding tooling that doesn't want to inflict on me the option of customizing it to N other cloud vendors. That imposes learning and deciding costs on me that I do not want. And that I believe many other people do not want.

AWS itself is so sprawling that it's just as bad for those who stick to its tools. You must learn enough to make choices that benefit you little (but leave the possibility that choosing wrong could have big costs). You really ought to learn enough to be competent to handle N different kinds of operation scenarios - some of them very complicated - in order to safely handle your single simple scenario.

We handle relatively few (measured in the millions per year, not per day), relatively heavyweight transactions. It's a constant struggle (one for which I am ill-prepared) to keep things simple. I think there's an unexploited market niche. An "X is to today's devops as XP was to RUP".

---

One solution (which we are taking) is to hire a consultant who's deeply experienced in N scenarios - including one like yours - to give you the answers. Perhaps that's the only solution, like this could have been a universe in which the only solution was to hire the RUP consultant to tailor the process to you. I hope that's not the case.

Hmm. It doesn't feel right to say, "X is Heroku", but I'm not sure I can rebut that claim.

I think you're conflating loads of things here!

AWS is 'devops' in the same way that junit is 'agile', which is to say that it's a tool used by people, and some of those people have brought into a culture (devops, agile). Which is also to say that not all have.

It seems to me that there are three key problems you raise here:

1. You don't like the current state of the art of tools which provide abstractions over AWS
2. AWS is really complicated
3. (Reading between the lines) that managing operations isn't simple

None of these are about devops. Devops is simply saying we should break down the barriers between dev and ops. It doesn't say that devs have to manage their own operations (which seems to be where you are) although this is a direction that some people go in when embracing this idea. Some, but not all.

If you're using tools which are abstractions on AWS, but whose abstractions add too much complexity (supporting other vendors), then don't use them. Just use the AWS toolchain. Yes, AWS has a lot of stuff, as the problems it's trying to solve are many and varied. If you don't need all those things, then either limit yourself to a subset (very sensible - it's like deciding not to use 70% of the language features when programming in Scala!) or pick a simpler platform.

As to operations, managing IT operations is a much bigger space than that of custom software development. Last time I looked at the industry stats for example, spend on operations represented 70% of the total spend on IT, with software development being in the 30%. There are lots of things that operations people have to worry about that devs don't know about - this is stuff I had to learn too coming from a more dev-y background. I know enough to be a conscious incompetent, not that much more.

As someone who has had a foot in both camps for the last 15 years or so (dev and ops), I have become increasingly unconvinced that most developers can keep up with the state of the art for their chosen software platform and also get good enough at operations to run them well, all at the same time, unless they are using a platform of some sort. Both spaces are moving too quickly. If you really want to have the same team run operations and delivery, then you have two choices - pick a higher-level abstraction platform for your software delivery like Heroku, or a hosted Kubernetes (where you effectively outsource most of the ops work), or hire a specialist (or two). I find it surprising how many developers who go on about craftsmanship (this isn't aimed at you, but a trend I have seen!) who are then willing to disregard the importance of people with deep knowledge and expertise in sectors outside their own!

@snewman: All you say is fair enough. Probably more than fair. I have a few hesitations:

1. I've increasingly come to appreciate how we're kidding ourselves when we separate practices, principles, or even values from tools. Tools (in the general sense) both enable and shape all of those things. That goes all the way from Notation as a Tool of Thought to a nice paper I read about how the evolving form of the bicycle was heavily influenced by the structure and beliefs of the society that adopted it. (Its form did not just "follow function" within the scope of the material technology of the day.)

   Therefore: the existing state of tools cannot be separated from the goal of breaking down the barriers between dev and ops. It will materially affect them.

2. "Just limit yourself to a subset" is not the advice I'm looking for. To do that effectively with something like AWS, much less all the other options, you must be an expert to know what you can leave out (etc.) In my original analogy, there needs to be a Kent Beck to say "Limit yourself to this subset."

3. Introduction by way of a story. I was at a party shortly before the turn of the century. A veterinarian spent some time telling me how simple the Y2K problem would be to solve. I smiled politely and thought, "You are an asshole. You have no idea of the real-world complexity of the problem!" And then I realized: "I do this kind of thing all the time. It's practically part of the job description of programmers. Maybe I'm an asshole too."

   However: I am one of those who believe teams of people who are somewhat t-shaped or are "specializing generalists" are more powerful than seems simple. By analogy. When I came up as a programmer, it was received wisdom that programmers can't test their own code because they will be inherently blind to their own errors. This turns out to be wrong. Programmers can test OK or even well. They'll never be as good at testing as a specialized tester - for lots of reasons - but it is perfectly possible for a new company to do well for a long time without specialized testers.

   Why do you suggest the same is not possible with ops? A tester might say, "OK, testing outsider, so testing isn't a job so complex specialists are required, but your job is? How conveeeeeenient."

   Remember: I'm talking about a context that does not, and could not, spend 70% on operations. Facts about companies that do are not relevant. The question is not: "is it the case that there should be specialized ops people?", but "in what context can you get away without an ops person, and how?" I don't believe the answer is "in no context".