Orchestration is the Ultimate Order of (Deployment) Operations

Remember when you were in school, learning math, and you learned about the importance of the order of operations? You do? Okay, good. Pop quiz:

1 + 1 * 8 = ?

The answer is 9, not 16. Why? Because multiplication has precedence. If you want to get to 16 with those numbers, we’ll need to add some parentheses:

(1+1) * 8 = 16

Because parentheses have precedence over everything else. 

Okay, enough math for today. The point of this little exercise was simply to emphasize that order matters. And not just in math, but in just about anything that involves a series of steps. Like operations.

John Willis (aka botchagalupe) recently penned an article on Immutable Delivery which was, in itself, a fascinating read for those truly in love with DevOps. In it, he revisited this notion, that order matters and cited a paper on the topic  which was also a fascinating read. This is not the first time that John has written about the importance of order in operations, especially as it relates to immutable concepts. The basic premise is that the same set of commands when carried out in a different order can have disastrous results.

One of the examples in the paper that illustrates this is:

“First we will cover a trivial but devastating example that is easily avoided. This once happened to a colleague while doing manual operations on a machine. He wanted to clean out the contents of a directory which ordinarily had the development group's source code NFS mounted over top of it. Here is what he wanted to do:

umount /apps/src
cd /apps/src
rm -rf .
mount /apps/src

Here's what he actually did:

umount /apps/src
...umount fails, directory in use; while resolving
this, his pager goes off, he handles the interrupt,
then...
cd /apps/src
rm -rf .

Obviously this is bad Juju. And the next time you see me in person ask about my days at a GIS company where someone actually did something similar and nearly wiped out every digital map the company owned. Seriously.

So at this point we probably all agree that the order of operations is important.

But it’s not just important as a means to avoid catastrophic failures. After all most failures to honor the order of operations aren’t going to have this level of disastrous results. The other reason we should pay attention to the order of operations is because it has a profound impact on the velocity of the app deployment process. And given that there’s increasing pressure on IT to deliver apps (particularly mobile apps according to Gartner, Inc. who predicts that “by the end of 2017, market demand for mobile app development services will grow at least five times faster than internal IT organizations' capacity to deliver them”) and an increasing focus on decreasing the time it takes to get to market, anything we can do to improve the speed of the deployment process is going to be constructive and useful.

To do that, we have to look at the order of operations and start applying some DevOps math to optimize it.

Optimizing the Order of Operations

In most organizations there’s a distinct order of operations at the process layer that dictates which task (automated or otherwise) must be completed before another can move forward. This is often due to dependencies that require a very specific order of execution. Some dependencies are technical, e.g. you have to to deploy the app platform before the app, but others are business dependencies. Additions to DNS, for example, are not necessarily technically dependent on other tasks, but may be considered to have business dependencies that require them to be completed before or after certain other tasks. And still other tasks have no dependencies at all. Their order within the deployment process orchestration is mandated by convention, not corporate policy.

The trick for DevOps is to find the optimal order of operations at the process layer. That means determining not just the order of operations as it is now, but what its optimal state might be. That means combing through each layer – from automation up to orchestration – and ferreting out where there may be unnecessary wait times between tasks or process steps, which tasks can be moved and where, and whether or not there exists duplication that can be eliminated.

It may mean (gasp) math.

Orchestration is the ultimate order of (deployment) operations. Getting it wrong won’t necessarily destroy data or corrupt configuration files, but it will definitely impede your ability to improve the processes that deploy apps. That’s why it’s important to move up the deployment stack and evaluate deployment processes with an eye toward optimization.

Because the order of operations really does matter.

For a bit deeper view on Six Sigma, DevOps and how it impacts the deployment process, here’s a presentation from a session at DevOps Summit.