Ryan Armstrong's Blog

Dynamic stats in Go's expvar

Mon, 11 Jun 2018 00:00:00 +0000

Go’s expvar package enables a standardized way to publish runtime statistics for your application. Out of the box, by importing expvar, you will have a HTTP handler registered at “/debug/vars” that will print a whole mess of GC, memory and commandline stats as a JSON document.

Expvar includes many primitives for safely setting or incrementing various static metric types. This works great under normal circumstances, but there are occasions when you might need to dynamically compute the desired metric value on demand.

A key example is “uptime”. How long has my service been up? Using the traditional static approach, you would need to update this value periodically (wasteful) and you risk retrieving an outdated value on request.

The following code is the simplest way to export an “uptime” variable that is computed on demand, using expvar.Func and a closure function.

import (
	"expvar"
	"net/http"
	"time"
)

func init() {
	start := time.Now()
	expvar.Publish("uptime", expvar.Func(func() interface{} {
		return time.Since(start).Seconds()
	}))
}

func main() {
	http.ListenAndServe(":8080", http.DefaultServeMux)
}

If you had more complex needs - say, additional fields to consider - you could instead implement the expvar.Var interface

import (
	"expvar"
	"net/http"
	"strconv"
	"time"
)

type uptimeVar struct {
	start time.Time
	// other important fields...
}

func (v *uptimeVar) String() string {
	// other things to compute...
	d := time.Since(v.start)
	return strconv.FormatFloat(d, 'g', -1, 64)
}

func init() {
	v := &uptimeVar{
		start: time.Now()
	}
	expvar.Publish("uptime", v)
}

func main() {
	http.ListenAndServe(":8080", http.DefaultServeMux)
}

A possible tradeoff in using dynamic metrics is that you risk introducing latency to your monitoring tools or health checks if your dynamic metrics are too compute-heavy or if they depend on I/O.

Optimized abs() for int64 in Go

Sat, 13 Jan 2018 00:00:00 +0000

The Go programming language has no built in abs function for computing the absolute value of an integer. That is, the non-negative representation of a negative or positive number.

I recently needed an implementation of the abs function to solve the Day 20 challenge in Advent of Code 2017. If you want to learn something new and get a kick out of testing yourself, I strongly recommend you check it out!

Go does include an abs function in the math package: math.Abs. Unfortunately, this doesn’t fit my use case, as it only accepts float64 as input and output. I need int64. It is possible to use math.Abs via type conversion, but this introduces some overhead for the conversion to float64 and back to int64, as well as introducing truncation for larger numbers.

There is a great discussion on optimizing math.Abs for floating point integers, but unfortunately these optimizations don’t directly apply to integers, due to their underlying encoding.

There must be other options. So begins my adventure into optimizing a simple function in Go!

You can find the source code and tests for this article on GitHub.

Type conversion vs. branching

The most obvious solution to me for this problem was a very basic function which returns n if n is zero or greater, and returns negative n if n is less than zero. The double negative is, of course, always positive.

Since it relies on a branching control structure to calculate an absolute value, let’s call this function abs.WithBranch.

package abs

func WithBranch(n int64) int64 {
	if n < 0 {
		return -n
	}
	return n
}

It works! Curiously, this is currently (Go v1.9.x) how math.Abs is implemented for float64 numbers.

Great! Branching works. But, does it improve on a call to math.Abs using type conversions? Let’s implement that too.

package abs

func WithStdLib(n int64) int64 {
	return int64(math.Abs(float64(n)))
}

Here we are converting n to a float64, passing it to math.Abs and then converting the result from float64 to int64. Surely, this must introduce some performance overhead?

A couple of simple benchmark tests yield clear results:

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/abs
BenchmarkWithBranch-8           2000000000               0.30 ns/op
BenchmarkWithStdLib-8           2000000000               0.79 ns/op
PASS
ok      github.com/cavaliercoder/abs    2.320s

At 0.30 ns/op (nanoseconds per operation), WithBranch is more than twice as fast! But our benchmark doesn’t represent the real world…

Benchmarking

We know that branching code breaks the sequential flow of a program, meaning that pipelining processors must predict what will happen next to maintain similar performance to a sequential program.

In the above benchmark, we used a constant value as the input to each call to abs. Because of this, the processor is able to predict which branch to take every time, because it’s the same every time.

We can trip the processor’s ability to predict the flow of execution, and better represent real world use-cases by introducing a Pseudo Random-Number Generator. If the input is random, the output will be random and the processor’s predictions should be wrong more often.

Given there are only two possible futures, I assume a pipelining processor can compute both outcomes at once, without a performance penalty. Let’s test by experiment.

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/go-abs
BenchmarkRand-8                 1000000000               2.59 ns/op
BenchmarkWithBranch-8           200000000                6.36 ns/op
BenchmarkWithStdLib-8           500000000                3.13 ns/op
PASS
ok      github.com/cavaliercoder/go-abs 6.692s

I’ve introduced a new benchmark, BenchmarkRand, to measure the overhead of the RNG. We can substract the resulting overhead from the other benchmarks for an approximation of their actual runtime.

It turns out, abs.WithBranch doesn’t perform very well with random input. In fact, abs.WithStdLib is nearly seven times faster.

Truncation

Before we throw out the branching approach, it should be noted that it has the advantage that it won’t truncate any large numbers by converting from a signed int64 to an IEEE-754 float64, losing bits to represent the decimal point.

By example, abs.WithBranch(-9223372036854775807) correctly returns 9223372036854775807, while abs.WithStdLib(-9223372036854775807) experiences an overflow during the type conversion and incorrectly returns -9223372036854775808. It even returns the same incorrect, negative result for a high positive input, abs.WithStdLib(9223372036854775807).

The type conversion approach is faster with random input, but the branching approach is more correct. Can we do better?

A non-branching solution

Chapter 2 of Hacker’s Delight by Henry S. Warren introduces us to a branch-free way of computing the absolute value of a signed integer using a little Two’s Complement arithmetic.

To compute the absolute value of x, first, we compute the value y which is equal to x ⟫ 63. That is, x right shifted by 63 bits on a 64 bit architecture. Finally, compute (x ⨁ y) - y. That is, x exclusive-or y, take y. This yields the absolute value of x.

Because we live hardcore, let’s implement this in assembly!

// abs.go
package abs

func WithASM(n int64) int64

// abs_amd64.s
TEXT ·WithASM(SB),$0
  MOVQ    n+0(FP), AX     // copy input to AX
  MOVQ    AX, CX          // y ← x
  SARQ    $63, CX         // y ← y ⟫ 63
  XORQ    CX, AX          // x ← x ⨁ y
  SUBQ    CX, AX          // x ← x - y
  MOVQ    AX, ret+8(FP)   // copy result to return value
  RET

We start above with a Go function declaration for WithASM that has no immediate implementation. In a separate ASM code file, we declare the body in Go’s Assembler language. Note that this implementation is only valid on amd64 architecture systems, so we advise the compiler by giving the file name the _amd64.s suffix.

And the benchmarks results:

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/go-abs
BenchmarkRand-8                 1000000000               2.52 ns/op
BenchmarkWithBranch-8           200000000                5.91 ns/op
BenchmarkWithStdLib-8           500000000                2.88 ns/op
BenchmarkWithASM-8              1000000000               2.78 ns/op
PASS
ok      github.com/cavaliercoder/go-abs 9.454s

All right! We observe a minor improvement of 0.10 ns/op. That’s actually a ~38% speed up. Still, there could be more! It might help to understand how Go treats ASM code.

Compiler optimizations

We need visibility of the optimizations the compiler is applying to the Go functions. The compiler accepts the -m flag to “print optimization decisions”. This can also be enabled in go build or go test with the -gcflags=-m argument.

Here’s what we see:

$ go tool compile -m abs.go
# github.com/cavaliercoder/abs
./abs.go:11:6: can inline WithBranch
./abs.go:21:6: can inline WithStdLib
./abs.go:22:23: inlining call to math.Abs

For simple functions like ours, the Go compiler supports function inlining. Function inlining means that calls to our function are replaced inline with the actual body of our function.

By example,

package main

import (
  "fmt"
  "github.com/cavaliercoder/abs"
)

func main() {
  n := abs.WithBranch(-1)
  fmt.Println(n)
}

… might actually be compiled more similar to:

package main

import "fmt"

func main() {
  n := -1
  if n < 0 {
    n = -n
  }
  fmt.Println(n)
}

According to the compiler output above, WithBranch and WithStdLib are able to be inlined, but WithASM is not. Even the underlying call to math.Abs is inlined into WithStdLib.

Since our WithASM function cannot be inlined, each caller incurs the overhead of a function call. This means writing a stack frame, copying in arguments, branching the program pointer, etc.

What if we even the playing field and disable inlining on our other Go functions? We can do this with a simple pragma comment before the function declaration: //go:noinline.

For example:

package abs

//go:noinline
func WithBranch(n int64) int64 {
	if n < 0 {
		return -n
	}
	return n
}

Running the compiler again, we see many fewer optimizations:

$ go tool compile -m abs.go
abs.go:22:23: inlining call to math.Abs

And here are the benchmark results:

$go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/go-abs
BenchmarkRand-8                 1000000000               2.40 ns/op
BenchmarkWithBranch-8           200000000                7.06 ns/op
BenchmarkWithStdLib-8           500000000                3.46 ns/op
BenchmarkWithASM-8              1000000000               2.86 ns/op
PASS
ok      github.com/cavaliercoder/go-abs 10.032s

The performance of WithASM remains unchanged, while WithStdLib is three times slower thanks to the overhead of the non-inline function call.

The lesson learned for me here, is that the performance gained by implementing ASM by hand needs to outweigh the benefits of compiler type safety, garbage collection, function inlining, etc. In most cases, this won’t be the true, though there are exceptions; like taking advantage of SIMD instructions for cryptography or media encoding.

Running the benchmarks a few times it becomes clear that there is a reasonable performance gain to be had via ASM. But, can we somehow retain the benefits of inlining?

One inline function, please

The Go compiler cannot inline functions that are implemented in assembler, but implementing our Two’s Complement arithmetic in Go is easy:

package abs

func WithTwosComplement(n int64) int64 {
	y := n >> 63          // y ← x ⟫ 63
	return (n ^ y) - y    // (x ⨁ y) - y
}

The compiler says our new function can be inlined:

$ go tool compile -m abs.go
...
abs.go:26:6: can inline WithTwosComplement

How does it perform? It turns out, when we re-enable inlining, it performs even better! A whole 0.18 ns better, or roughly 60%.

$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/cavaliercoder/go-abs
BenchmarkRand-8                         1000000000               2.39 ns/op
BenchmarkWithBranch-8                   300000000                5.71 ns/op
BenchmarkWithStdLib-8                   1000000000               3.01 ns/op
BenchmarkWithASM-8                      1000000000               2.88 ns/op
BenchmarkWithTwosComplement-8           1000000000               2.70 ns/op
PASS
ok      github.com/cavaliercoder/go-abs 14.417s

Now that the overhead of a function call is gone, the Two’s Complement implementation in Go out performs the ASM implementation. Curiosity might lead us next to wonder how similar the compiler’s output is to our hand-crafted ASM?

There’s an app for that.

Passing -S to the Go compiler causes it to “print assembly listing”.

$ go tool compile -S abs.go
...
"".WithTwosComplement STEXT nosplit size=24 args=0x10 locals=0x0
        0x0000 00000 (abs.go:26)        TEXT    "".WithTwosComplement(SB), NOSPLIT, $0-16
        0x0000 00000 (abs.go:26)        FUNCDATA        $0, gclocals·f207267fbf96a0178e8758c6e3e0ce28(SB)
        0x0000 00000 (abs.go:26)        FUNCDATA        $1, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB)
        0x0000 00000 (abs.go:26)        MOVQ    "".n+8(SP), AX
        0x0005 00005 (abs.go:26)        MOVQ    AX, CX
        0x0008 00008 (abs.go:27)        SARQ    $63, AX
        0x000c 00012 (abs.go:28)        XORQ    AX, CX
        0x000f 00015 (abs.go:28)        SUBQ    AX, CX
        0x0012 00018 (abs.go:28)        MOVQ    CX, "".~r1+16(SP)
        0x0017 00023 (abs.go:28)        RET
...

The compiler’s implementation is exactly the god-damned same. Only this time, it has the advantage of being correctly configured and being cross-platform portable! Running the compiler again with GOARCH=386 produces a more complicated program that handles the 64-bit arithmetic on 32-bit machines.

I need to be more trusting and slightly less hardcore.

One last note about memory allocations; all of our implementations exhibit the same ideal behavior. When I run go test -bench=. -benchmem, I observe that each of the functions result in zero allocation operations and zero bytes allocated within the function body.

Homework

A few exercises for the reader:

What if we implemented the assembly for abs as a compiler intrinsic, rather than a runtime function call?
What if we used the PABSQ instruction included in SSSE3? This is not currently supported in Go’s ASM.

Conclusion

The Two’s Complement implementation in Go offers portability, function inlining, predictable performance, zero allocations and no integer truncation due to type conversions. The benchmarks showed a significant speed advantage over the other approaches.

The end result:

func abs(n int64) int64 {
	y := n >> 63
	return (n ^ y) - y
}

Finding the latest CentOS AMI

Thu, 02 Nov 2017 00:00:00 +0000

Looking for the latest CentOS AMI image in your region? I’m frequently rewriting the following code snippets, so thought I would capture them here for posterity.

Each snippet makes the following assumptions:

You want an HVM, EBS backed, x86_64 image of the latest version of CentOS 7
You have configured the desired region and credentials for the AWS SDK
The Owner ID for the official CentOS organisation remains 679593333241

Bash / AWS CLI

aws ec2 describe-images \
  --owners 679593333241 \
  --filters \
      Name=name,Values='CentOS Linux 7 x86_64 HVM EBS*' \
      Name=architecture,Values=x86_64 \
      Name=root-device-type,Values=ebs \
  --query 'sort_by(Images, &Name)[-1].ImageId' \
  --output text

Python / Boto3

import json
import boto3

EC2 = boto3.client('ec2')
response = EC2.describe_images(
    Owners=['679593333241'], # CentOS
    Filters=[
      {'Name': 'name', 'Values': ['CentOS Linux 7 x86_64 HVM EBS *']},
      {'Name': 'architecture', 'Values': ['x86_64']},
      {'Name': 'root-device-type', 'Values': ['ebs']},
    ],
)

amis = sorted(response['Images'],
              key=lambda x: x['CreationDate'],
              reverse=True)
print amis[0]['ImageId']

Terraform

data "aws_ami" "centos" {
  owners      = ["679593333241"]
  most_recent = true

  filter {
    name   = "name"
    values = ["CentOS Linux 7 x86_64 HVM EBS *"]
  }

  filter {
    name   = "architecture"
    values = ["x86_64"]
  }

  filter {
    name   = "root-device-type"
    values = ["ebs"]
  }
}

Inline vs. discrete rules for AWS Security Groups in Terraform

Sat, 15 Jul 2017 00:00:00 +0000

There are two ways to configure AWS Security Groups in Terraform. You may define rules inline with a aws_security_group resource or you may define additional discrete aws_security_group_rule resources.

My first instinct was to define a “base” Security Group using inline rules and then extend on it using external rules. Bad idea. More on that later.

For the two valid options though, there are important implications and I found these were not clear at the time of writing (circa Terraform v0.9.11). After a little research and experimentation I have a much clearer understanding and hope to save you all the bother.

This article focuses on managing AWS Security Groups in Terraform but you will find that all of the principals explored here apply equally to Network ACLs and Route Tables - both of which allow inline or external rule management.

Two approaches

Here’s how an inline Security Group definition looks:

resource "aws_security_group" "allow_all" {
  name        = "allow_all"
  description = "Allow all inbound traffic"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Above there are two rules, an ingress and egress rule defined inside or inline with the aws_security_group resource block.

Here’s how the same idea can be expressed using external rules via the aws_security_group_rule resource:

resource "aws_security_group" "allow_all" {
  name        = "allow_all"
  description = "Allow all inbound traffic"
}

resource "aws_security_group_rule" "ingress" {
  type        = "ingress"
  from_port   = 0
  to_port     = 0
  protocol    = -1
  cidr_blocks = ["0.0.0.0/0"]

  security_group_id = "${aws_security_group.allow_all.id}"
}

resource "aws_security_group_rule" "egress" {
  type        = "egress"
  from_port   = 0
  to_port     = 0
  protocol    = -1
  cidr_blocks = ["0.0.0.0/0"]

  security_group_id = "${aws_security_group.allow_all.id}"
}

The Security Group and each of its rules are defined as discrete resources, intimately linked together in loving union by the security_group_id attribute.

A reasonable person might posit that the outcome of both configurations would be the same, but they are different in subtle ways - ways that might hurt a bit if not clearly understood.

Option 1: External rules

I had hoped that external rules would function similar to Puppet’s concat module - gathering partial resources defined anywhere in the graph and then enforcing the sum state. The reality however, which does make sense, is that the desired state is managed non-destructively.

You can test this, by manually adding a rule to the Security Group created by the Terraform code above (the snippet with only external rules). If you run terraform apply, it will ignore this manually created rule.

What this means, is that you can add and enforce rules on a Security Group that was created elsewhere. Just be cautious of conflicts with existing rules, precedence and collisions in your rule numbers.

Pros

Rules can be added non-destructively to unmanaged Security Groups. This might be useful for Security Groups associated with shared services, such as a Bastion host. Terraform configurations from all over the kingdom can create rules on a common Security Group to enable the access they require.

Ideally, one would create and associate distinct, well-tagged Security Groups for each use-case but we are limited to only five Security Groups per network interface.
Security Groups can be “modularized” and extended upon. You might write a Terraform module that creates a Security Group with standardized naming, tagging, lifecycle and maybe even some base access rules. Additional rules could then be added to the Security Group returned by the module using aws_security_group_rule resources.

Cons

Security Group rules that are added by accident or by nefarious means will not be nuked by Terraform. It won’t even report that they exist.
Infrastructure code should succinctly describe and quasi-document your actual infrastructure. Defining rules for a single Security Group in disparate files in your code-base makes it difficult to see at a glance what the state of a Security group should be.
It’s difficult to know or manage for how externally created rules will effect the rules you are configuring in Terraform. This is particularly important for Network ACLs, where a rule defined elsewhere could have higher precedence than yours and might explicitly deny a port you intend to open.

Option 2: Inline rules

When rules are defined inline, a Security Group is managed destructively. That is, any rule not defined inline, including rules defined elsewhere in Terraform and rules added manually or via other tools, will be unapologetically destroyed whenever Terraform next runs.

Pros

Security Groups are for security. They need to be clear, understood and well configured. Using inline rules means your resource definition is complete, definitive and deterministically provisioned. You can have confidence that there are no rules being described elsewhere in the code-base or added outside of Terraform. If there are, they get destroyed.

Cons

There’s no way to share code between inline-style Security Groups. This might mean lots of repetition for Security Groups that are mostly the same; having a common set of rules with minor exceptions between them. Make use of the five available Security Group slots per network interface to allow re-use of shared base Security Groups - though this is difficult if you’re using least-privilege, point-to-point access rules.

You can’t use the count meta-parameter to described rules in a loop.

It works with external rules:

variable "http_ports" {
  default = ["80", "443", "8080", "8443"]
}

resource "aws_security_group_rule" "ingress_http" {
  count = "${length(var.http_ports)}"

  type        = "ingress"
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
  from_port   = "${element(var.http_ports, count.index)}"
  to_port     = "${element(var.http_ports, count.index)}"

  security_group_id = "${aws_security_group.allow_all.id}"
}

… but not with inline rules:

variable "http_ports" {
  default = ["80", "443", "8080", "8443"]
}

resource "aws_security_group" "allow_all" {
  name        = "allow_all"
  description = "Allow all inbound traffic"

  ingress {
    count = "${length(var.http_ports)}"

    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
    from_port   = "${element(var.http_ports, count.index)}"
    to_port     = "${element(var.http_ports, count.index)}"
  }
}

# 1 error(s) occurred:
#
# * aws_security_group.allow_all: ingress.0: invalid or unknown key: count
#

You’re going to need to code up an inline rule for each port.

Why not both?

My naive first approach was to blend both approaches. I hoped to create a configurable aws_security_group “module” that contained some mandatory rules, like allowing ingress SSH, monitoring, etc. This module could then be extended with additional rules using aws_security_group_rules resources. A kinda pseudo-OOP-abstract-class approach.

Fortunately, the Terraform documentation contained a well-lit warning sign:

At this time you cannot use a Security Group with in-line rules in conjunction with any Security Group Rule resources. Doing so will cause a conflict of rule settings and will overwrite rules.

What happens when you combine both methods? Here’s a cool high-school science lab experiment for you!

The following Terraform code defines both inline rules, and an external ingress_http rule.

resource "aws_security_group" "allow_all" {
  name        = "allow_all"
  description = "Allow all inbound traffic"

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_security_group_rule" "ingress_http" {
  type        = "ingress"
  from_port   = 80
  to_port     = 80
  protocol    = 6
  cidr_blocks = ["0.0.0.0/0"]

  security_group_id = "${aws_security_group.allow_all.id}"
}

Instructions

Apply this code with terraform apply - it should create the ingress_http rule
Apply the same code again with terraform apply - it should remove the newly created ingress_http rule
Go to step 1

Terraform will create and then destroy the external rule on each alternating invocation - like that scene with Dormammu in Marvel’s Dr Strange.

Bug? No. It actually kinda makes sense. Jake Champlin from HashiCorp explains it on a related GitHub issue.

Summary

I’ve come to prefer using inline rules where possible. It means our Security Groups match the code and phantom ‘allow all’ rules can’t be introduced that would break our security model and the integrity of our tests.

It does mean we have to duplicate some rules in a few Security Groups, and keep these definitions in sync, but I share the opinion that a little copying is better than a little dependency.

For more information about AWS Security Groups in Terraform, please see:

A WebOps Postmortem

Tue, 27 Jun 2017 00:00:00 +0000

Early in the day, May 18, 2017, alerts started appearing from our Content Distribution Network that a number of user requests were being served with 504 Gateway Timeout responses.

I lead the Production Engineering team at Seven West Media in Western Australia and the alerts indicated an issue with thewest.com.au.

Our internal process did produce a postmortem report that differs from this one, but only in formality, not in terms of the facts. This is my summarized and more technically focused treatment of the incident.

I hope to highlight here some of the principles that we employ that were advantageous to us, as well as simply exploring an interesting problem that reminded us to stay humble.

While reports came in from our colleagues and customers that images were failing to display on the site, a killer team of Developers and SysAdmins invoked our Incident Response process. We borrow heavily from ITIL Incident Management and Google’s Site Reliability Engineering.

The process calls for us to stay calm, objective and to seek out quality information for better decision making.

Network I/O deviations

Initially we discovered unusual network I/O patterns on our Content API servers.

The Content API is a Node.js/Express application fronted by a local instance of nginx. It is responsible for serving article content, listings, curations, etc. to both our internal web servers (for server-side rendering) and to our client-side Javascript which uses the Single-Page Application model (see: Isomorphic Javascript: The Future of Web Apps). It also serves static assets, such as images, which it pipes directly from Amazon S3.

The above figure from Zabbix shows that network I/O became more “ordered” on one of the Content API instances around 7 AM. A brief exploration of Complexity Theory teaches us that things go wrong when systems transition to exhibiting ordered behaviors.

We employed Brendan Gregg’s USE method to build a clearer picture of network performance. We observed:

throughput between each component was otherwise typical, with plenty of headroom
no dropped packets, interface errors or other known TCP/IP issues that would indicate saturation or congestion
no error messages in any of the counters or logs we knew to check
no network issues were being reported by AWS
Health checks on the Content API servers were passing

There was one important metric that we neglected to check for saturation but more on that later!

HTTP 499 errors

One fastidious engineer noticed something else amiss: a large number of 499 Client Closed Request responses being logged by nginx on the same Content API servers.

The above screen grab from Sumo Logic of the same instance shows the uptick in 499 status messages at the same time that the network I/O anomaly appeared. The uptick is obvious in this image, but it was visually masked on our dashboards by the thousands of successful requests occurring at the same time.

Reviewing the nginx logs on each server revealed an interesting pattern that helped us to understand the scope of the issue: all 499 responses were constrained to static assets only. Requests for dynamic content were unaffected.

This finding was easy to confirm locally on any of the production servers. A quick $ curl -iv http://localhost/... for any static asset would hang until the client cancelled the request. The same curl request for any other resource behaved as expected.

This test also validated that no error was logged via nginx or the Node.js application. The application logs did show an incoming request but no outbound response or error.

So here’s what we knew:

Either our CDN or Load Balancing layer was cancelling some requests and returning status 504 as no timely response was received from upstream
Nginx was logging a status 499 when the downstream components cancelled a request
Requests to the Content API for static assets were falling into a black hole which meant that image content on the site was failing to display

At this point we decided we had sufficient information about the scope of the issue to consider a mitigation plan.

Stem the bleeding

Our incident response process emphasizes small, non-destructive changes over wholesale repairs where possible. I annoyingly repeat the mantra to my team that we must:

Avoid corrective surgery - just stem the bleeding

Corrective surgery is best performed once the situation has stabilized, we have better information and everyone is cool, calm and collected.

We formed a hypothesis that freshly launched instances of the Content API might behave correctly and we found that we were right! The new servers were not producing any errors once marked in.

We did have the option to instead simply restart the unhealthy instances, but our process also encourages the preservation of evidence. Restarting the instances would have reset their state - and as we learned later - would have reset the key metric we neglected to observe. We might not have found the root cause until the issue surfaced again.

Besides that, servers should be treated like cattle, not pets. A freshly launched instance represents a known good state.

With the healthy instances in rotation, everything stabilized. The 499 log messages disappeared, our curl test started behaving correctly and users were confirming a positive outcome.

Root cause analysis

With clearer heads and happy customers, we had the opportunity to take a detailed look at the unhealthy servers and compare them to healthy servers.

A call to $ netstat -t4 shows all TCPv4 sockets on a Linux system. The following figure shows the results of this call on both an unhealthy (left) and healthy (right) Content API instance.

Notice that the unhealthy instance has a large number of sockets in a CLOSE_WAIT state and also that most sockets have a large quantity of packets in the Receive Queue (second column). All of these sockets list AWS S3 as the foreign end point.

By comparison the healthy server has only one CLOSE_WAIT socket and the Receive Queues are mostly empty, except for the lonely CLOSE_WAIT socket. Could this be the beginning of a very slow connection leak?

Remember that key metric that we neglected to observe during our network performance observations? We needed to measure the number of CLOSE_WAIT sockets on a server. Queue length and the other sockets states might also have been useful under differing circumstances.

According to Gordon McKinney’s TCP/IP State Transition Diagram, CLOSE_WAIT indicates that the socket is:

… waiting for a connection termination request from the local user

In our case the “local user” was the Node.js application. Adding -p to the netstat call printed the PID that owned the leaked sockets. The PID belonged to our application.

Somewhere in our application, sockets were being created for S3 requests, but never cleaned up. These sockets were exhausting some unknown upper limit, causing new requests to hang.

Now we know where to look and our picture of the control path has a little more detail. Here’s how it looked once the leaked sockets reached critical mass:

We set about tracing the code path for S3 requests in our codebase and found the following.

AWS SDK for Node.js

All of the connections to AWS S3 were being established via the AWS SDK for Node.js.

The SDK uses socket pooling for connections to S3. The default maximum size of the pool is 50. If sockets were leaking and there is an upper boundary to the number of active sockets, it stands to reason that new requests to S3 (in our case, for static assets) would fail in some way once 50 sockets had leaked. The number 50 anecdotally matched the number of CLOSE_WAIT sockets we had seen with netstat.

We also found that our version of the SDK had been inadvertently pinned in our codebase to v2.6.14, which was released shortly before we relaunched thewest.com.au.

Given the age of our SDK version, it was conceivable that the socket leak was now a known issue. We trawled through GitHub issues, PRs and the SDK Changelog and found a suspect note in the 2.50.0 release notes:

bugfix: Request: Updates node.js request handling to obey socket read timeouts after response headers have been received. Previously timeouts were being ignored once headers were received, sometimes causing connections to hang.

Inspecting the code also revealed a significant body of other work had also been released concerning sockets.

We formed a hypothesis that updating the AWS SDK would cause all sockets to be cleaned up correctly or emit errors that could be safely handled. There were strong coincidences that suggested this was our issue. Patching the SDK was low risk, easily tested and definitely overdue, but…

Measurements

It was important that we started measuring these socket metrics before implementing a fix. Before-and-after measurements are critical to validate any fix, and they make for great graphics in blog posts (below).

I wrote a Zabbix module named zabbix-module-sockets to capture the following metrics:

the number of sockets in each state
the sum of the receive queue of all sockets
the sum of the send queue of all sockets

With a little simulated load we were able to replicate the connection leak in pre-production environments and measure the results in Zabbix.

The fix

With a reasonable hypothesis, measurements in place and way to validate our plan in pre-production, we deployed the updated AWS SDK to our pre-production environments.

It worked! The following figure from Zabbix shows the dramatic change in behavior around June 2nd, when the fix was subsequently deployed to production.

The gradual upward trend of the Receive Queue length (green) stops after the production deployment and the queue length stabilizes. The earlier sawtooth pattern you may observe is caused by application restarts that were triggered to remediate the socket pool being exhausted.

The following figure shows sockets states on the same server. You will notice the thin, red slice of CLOSE_WAIT sockets discontinues around the time of the fix deployment.

Hey, check out the value in the max column for CLOSE_WAIT sockets…

Case closed.

Lesson learned

In our context, we have to “move fast and break things”. Most things require a trade-off between time-to-market and accrued technical debt. We aim to make the best possible moves with the limited information and resources we have available at any given time.

That said, this event highlighted some priorities that could at least help us to prevent similar issues in the future.

Alert on 499 responses

For HTTP servers that support HTTP 499 like nginx, a small number of 499 Client Closed Request responses should be expected at the edge, where clients do have the prerogative to disconnect prematurely. But in higher frequency, or at other points in your signal chain, these responses might indicate that timeouts are incorrectly configured. So…

Get your timeouts right

Each component should be configured with a timeout that is longer than all its upstream dependencies. By example, if you have a 3 second timeout on database queries, then the application server should not cancel requests any sooner than this. Otherwise the source of a problem is obscured and all you are presented with is a timeout error from a component other that the one that misbehaved - as we experienced.

This solution is complicated by services that need multiple round-trips or have asynchronous behaviors, but…

Everything should be bounded

There is an expected operational behavior of every component in our system. Any operation that completes too quickly or too slowly should create an error that is meaningfully propagated, logged and alerted on.

In this case, our Content API servers should have been configured with a tightly constrained timeout on all requests to S3. Error handlers in our code would then have sent a meaningful 5xx error back downstream.

This principle should also apply to the incoming request to the API. We should cancel any request that runs too long, for any reason. This has operational advantages, but is also good security practice to mitigate denial of service.

Health checks must validate all dependencies

Each of our micro-services has a health check route that quickly validates each of its known dependencies (e.g. it can connect to the database, write to a log file, etc.). In this case we had neglected to include S3 in the dependency checks. As a result, each service was advertising itself as healthy.

If we had included S3 in the health checks, the checks would have failed, these instances would have been marked out of the load balancer, alerts would have escalated and new healthy instances would have been rolled in automatically.

Summary

Under the stress of trying to restore services, we never felt any hunch to check the battle-hardened AWS SDK for leaked sockets to the notoriously reliable AWS S3 service. But this was our root cause.

As unlikely as the issue was, having a response plan in place and sticking to it gave us the best possible outcomes:

We followed the facts and found the root cause
Services were restored quickly, without introducing unnecessary risk or destroying evidence
We uncovered other weaknesses in our systems that could be addressed before they led to disaster
We learned a whole lot about the software and systems we work with

Thanks for reading! I’d love to hear your thoughts in the comments section below.

Appendix: Leaks from other buckets

With our new socket monitoring in place in Zabbix, we noticed a much smaller number of sockets to S3 being leaked on almost all of our application servers.

We discovered that the AWS CodeDeploy Agent was the culprit. We use CodeDeploy to deploy our Node.js applications and S3 is used as the storage backend.

Fortunately, these sockets leak much slower (we’ve only seen up to 3 on any one instance). For now, routine restarts will suffice until AWS are able to address the issue.

Restart services in docker-compose

Sat, 05 Nov 2016 00:00:00 +0000

The following shell function allows you to quickly restart one or more services in a running docker-compose service composition.

# restart services in docker-compose
docker-compose-restart(){
	docker-compose stop $@
	docker-compose rm -f -v $@
	docker-compose create --force-recreate $@
	docker-compose start $@
}

Once loaded into your shell, you can call it with:

$ docker-compose-restart [SERVICE...]

I often have this requirement when using docker-compose for development. If a Docker image is updated, it’s inconvenient to restart all of the containers managed by compose.

This function will:

stop the services
remove the associated containers and any anonymous volumes attached to them
recreate the containers from updated images
start the services again

Any services not specified are left unchanged.

Installation

Copy the function into your shell profile script (For BASH, this is ~/.profile).
Restart your shell by logging out and in again

Example

The following docker-compose.yml file defines two services: web and db:

---
version: '2'
services:
  web:
    image: nginx
    links:
    - mongo
  db:
    image: mongo

Suppose these services are running via docker-compose up, but you have made changes to the nginx image which is used to run the web service. These changes are not apparent in the running containers.

Restart the web service as follows for your changes to be available:

$ docker-compose-restart web

Zabbix Conference 2016

Sun, 18 Sep 2016 00:00:00 +0000

I’m finally recovering from jet lag after landing home from Zabbix Conference 2016 in Riga, Latvia. My good mate and colleague, James Cook and I had the pleasure of attending the conference this year to discover what’s new with Zabbix, how others are making use of it, and most importantly, to meet and greet with the superb team at Zabbix SIA HQ.

After my talk at Perth DevOps Meetup, Zabbix were kind enough to ask me to give a similar presentation at the conference. You can download a PDF copy here.

My talk covered a little about Kinetic IT and our client, the WA Department of Education. I then described some of the automation tools we have written to turn Zabbix into a fairly autonomous monitoring system. Finally I gave a demonstration of a selection of the automation tools that I have released or rewritten for open source.

Please also, check out the talks for all the other speakers at the Zabbix Conference 2016 homepage.

If you have any questions or comments, start the discussion below or get in touch with me on Twitter.

Monitoring 6000+ hosts in Zabbix

Thu, 26 May 2016 00:00:00 +0000

Last night I had the privilege of speaking to the Perth DevOps Meetup group about our journey delivering a large scale monitoring project with Zabbix. Thanks to everyone who came long and for the great questions afterwards.

Click the slide deck to download a PDF copy.

If you have any questions or comments, start the discussion below or get in touch with me on Twitter.

Downloading large files in Go

Tue, 19 Jan 2016 00:00:00 +0000

The net/http package bundled with Go provides a really solid HTTP implementation which excels particularly as a base for HTTP based API clients and servers.

If you’re writing software which needs to download large files from the Internet, such as ISO images or software packages, you may need to implement some client side logic to overcome some more use-case specific challenges such as:

multiple concurrent downloads
naming of downloaded files
UI feedback with progress indicators
clean cancellation of running downloads
resuming of interrupted downloads
validating downloaded files using checksums

This article will step you through using a Go package called grab which abstracts net/http to provide these features. We’ll build a simple ‘wget’-like binary to make use of all such awesomeness.

grab provides convenience methods grab.Get, grab.GetAsync and grab.GetBatch for simple operations. When you need more control over the HTTP session, you can use a grab.Client and configure it to your needs.

Examples for these functions and using a custom client are included below.

To get started, install the grab package with:

$ go get github.com/cavaliercoder/grab

Download a file

The simplest way to download a file is using grab.Get. It accepts two parameters; a destination file path and the source URL. grab.Get uses grab.DefaultClient as a HTTP client which has default settings. It will follow redirect responses from remote servers and use a corporate proxy if configured on the host system. Essentially, grab.Get is a wrapper for grab.DefaultClient.Do.

You may specify an existing or non-existing file path as the destination or you may specify an existing directory.

If a directory is given as the destination, grab will determine the filename using Content-Disposition headers if they are returned by the remote server or extract a filename from the source URL. If either of these features fails, an error is returned which can be identified using grab.IsNoFilename.

If the destination filename exists, grab assumes it is a complete or partially complete download and will resume downloading from the end of the file if supported by the remote server. Otherwise the file will be overwritten.

grab.Get and all other download functions return a grab.Response which includes context about the downloaded file; including the path where the file was saved.

grab.Get is a blocking, synchronous operation, which means that the function does not return a response until the download is complete or encounters an error. This is not terribly useful for lengthy downloads so I’ll solve this problem a little further down.

First, the following example will create a simple binary which will download a source file from a URL specified on the command line and save it to the current working directory (".").

package main

import (
	"fmt"
	"github.com/cavaliercoder/grab"
	"os"
)

func main() {
	// get URL to download from command args
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "usage: %s url\n", os.Args[0])
		os.Exit(1)
	}

	url := os.Args[1]

	// download file
	fmt.Printf("Downloading %s...\n", url)
	resp, err := grab.Get(".", url)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", url, err)
		os.Exit(1)
	}

	fmt.Printf("Successfully downloaded to %s\n", resp.Filename)
}

Build it with:

$ go build -o grab-example

and run it with:

$ ./grab-example http://some-url

Add progress updates

The simple example above will download a file but it is not practical for lengthy downloads which should provide some feedback to the user with the progress of the download.

The following example uses grab.GetAsync which immediately returns a channel which will receive a *grab.Response and close as soon as the download has been negotiated with the remote server, before the file transfer has started.

grab.GetAsync is a wrapper for grab.DefaultClient.DoAsync.

Once the response is received, it can be polled periodically to monitor the progress of the file transfer until it is finished. This example simply prints a progress update every 200ms.

All of the grab.Response methods are thread-safe and atomic.

package main

import (
	"fmt"
	"github.com/cavaliercoder/grab"
	"os"
	"time"
)

func main() {
	// get URL to download from command args
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "usage: %s url\n", os.Args[0])
		os.Exit(1)
	}

	url := os.Args[1]

	// start file download
	fmt.Printf("Downloading %s...\n", url)
	respch, err := grab.GetAsync(".", url)
	if err != nil {
		fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", url, err)
		os.Exit(1)
	}

	// block until HTTP/1.1 GET response is received
	fmt.Printf("Initializing download...\n")
	resp := <-respch

	// print progress until transfer is complete
	for !resp.IsComplete() {
		fmt.Printf("\033[1AProgress %d / %d bytes (%d%%)\033[K\n", resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
		time.Sleep(200 * time.Millisecond)
	}

	// clear progress line
	fmt.Printf("\033[1A\033[K")

	// check for errors
	if resp.Error != nil {
		fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", url, resp.Error)
		os.Exit(1)
	}

	fmt.Printf("Successfully downloaded to ./%s\n", resp.Filename)
}

Batch downloads

The next example allows multiple URLs to be given on the command line and downloaded simultaneously using grab.GetBatch() which is a wrapper for grab.DefaultClient.DoBatch().

Files will be transferred three at a time, as 3 is given for the worker count parameter in the call to grab.Getbatch(). To download all files immediately (one worker per request), simply give 0 as the worker count parameter. Each download will be saved to the current working directory as "." is given as the destination parameter.

With a batch operation we don’t have immediate access to any grab.Response. These will be sent via the channel returned by grab.GetBatch() each time a worker starts a requested URL. We don’t know when all of these responses will arrive, but we want to monitor downloads which are already in process so we create a for loop and select between two channels; one to receive responses and the other (the ticker) to periodically print the status of the responses which have already been received.

package main

import (
	"fmt"
	"github.com/cavaliercoder/grab"
	"os"
	"time"
)

func main() {
	// get URL to download from command args
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "usage: %s url [url]...\n", os.Args[0])
		os.Exit(1)
	}

	urls := os.Args[1:]

	// start file downloads, 3 at a time
	fmt.Printf("Downloading %d files...\n", len(urls))
	respch, err := grab.GetBatch(3, ".", urls...)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%v\n", err)
		os.Exit(1)
	}

	// start a ticker to update progress every 200ms
	t := time.NewTicker(200 * time.Millisecond)

	// monitor downloads
	completed := 0
	inProgress := 0
	responses := make([]*grab.Response, 0)
	for completed < len(urls) {
		select {
		case resp := <-respch:
			// a new response has been received and has started downloading
			// (nil is received once, when the channel is closed by grab)
			if resp != nil {
				responses = append(responses, resp)
			}

		case <-t.C:
			// clear lines
			if inProgress > 0 {
				fmt.Printf("\033[%dA\033[K", inProgress)
			}

			// update completed downloads
			for i, resp := range responses {
				if resp != nil && resp.IsComplete() {
					// print final result
					if resp.Error != nil {
						fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", resp.Request.URL(), resp.Error)
					} else {
						fmt.Printf("Finished %s %d / %d bytes (%d%%)\n", resp.Filename, resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
					}

					// mark completed
					responses[i] = nil
					completed++
				}
			}

			// update downloads in progress
			inProgress = 0
			for _, resp := range responses {
				if resp != nil {
					inProgress++
					fmt.Printf("Downloading %s %d / %d bytes (%d%%)\033[K\n", resp.Filename, resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
				}
			}
		}
	}

	t.Stop()

	fmt.Printf("%d files successfully downloaded.\n", len(urls))
}

Customizing requests

None of the previous convenience methods offer any control over the HTTP request or transport. Much like the net/http package, grab enable such controls as well as additional features such as checksum validation via grab.Client, grab.Request and grab.Response.

In the same way that the grab.Get* methods work behind the scenes, using a client requires that you define and configure a grab.Client, one or more grab.Requests and pass them to one of the grab.Client.Do* methods which then return a grab.Response for each request.

Let’s take a look at some of the configuration options available when creating a download request in the following example:

// create a download request
req, err := grab.NewRequest("http://some-url/my-file")
if err != nil {
	panic(err)
}

// set destination file path
req.Filename = "./my-file"

// set expected file size if known
req.Size = 1024

// set expected file checksum if known
b, _ := hex.DecodeString("b982505fc48ea2221d163730c1856770dc6579af9eb73c997541c4ac6ecf20a9")
req.SetChecksum("sha256", b)

// delete the downloaded file if it fails checksum validation
req.RemoveOnError = true

// request a notification when the download is completed (successfully or
// otherwise)
ch := make(chan *grab.Response)
req.NotifyOnClose = ch

You may also configure the HTTP request itself, including request headers, cookies, authentication, etc. using the http.Request nested in the grab.Request.HTTPRequest field, as demonstrated in the following code:

// set custom HTTP method
req.HTTPRequest.Method = "POST"

// set request headers
req.HTTPRequest.Header.Set("X-SOME-HEADER", "Custom value")

// set a cookie
req.HTTPRequest.AddCookie(http.Cookie{})

// set basic HTTP authentication headers
req.HTTPRequest.SetBasicAuth("username", "password")

Using a custom Client

grab provides a default client, grab.DefaultClient which is used by each of the grab.Get* methods. If you wish to customize HTTP transport rules such as connection timeouts, proxy configuration, redirect policies, etc. you may create a custom client with grab.NewClient.

The following code includes examples of customizing a client:

// create a custom client
client := grab.NewClient()

// set the user agent string for all HTTP requests
client.UserAgent = "MyApp"

// set a custom connection timeout
client.HTTPClient.Timeout = 3 * time.Second

// replace the HTTP client with one that bypasses any system proxy settings
client.HTTPClient = &http.Client{
	Transport: &http.Transport{
		Proxy: nil,
	},
}

Once you have configured a client and some requests, you pass the requests to whichever of the grab.Do* methods best match your use case. These methods are synonymous with the grab.Get* methods and behave as follows:

grab.Client.Do - blocks and returns a response once the download is completed or an error occurs
grab.Client.DoAsync - immediately returns a channel which will receive a single *grab.Response and close as soon as the download has been negotiated with the remote server; before the transfer has started
grab.Client.DoBatch - accepts multiple requests and executes them simultaneously. It accepts a workers parameter which determines how many downloads will be in process at any given time, while the remaining requests are queued until a worker is available. It returns a channel which will receive a grab.*Response for each request and close once they are all sent. The responses are sent through the channel as soon as the download has been negotiated with the remote server; before the transfer has started

The following and final example uses a custom client to download a batch of files with periodic progress updates:

package main

import (
	"fmt"
	"github.com/cavaliercoder/grab"
	"os"
	"time"
)

func main() {
	// get URL to download from command args
	if len(os.Args) < 2 {
		fmt.Fprintf(os.Stderr, "usage: %s url [url]...\n", os.Args[0])
		os.Exit(1)
	}

	// create a custom client
	client := grab.NewClient()
	client.UserAgent = "Grab example"

	// create requests from command arguments
	reqs := make([]*grab.Request, 0)
	for _, url := range os.Args[1:] {
		req, err := grab.NewRequest(url)
		if err != nil {
			fmt.Fprintf(os.Stderr, "%v\n", err)
			os.Exit(1)
		}

		reqs = append(reqs, req)
	}

	// start file downloads, 3 at a time
	fmt.Printf("Downloading %d files...\n", len(reqs))
	respch := client.DoBatch(3, reqs...)

	// start a ticker to update progress every 200ms
	t := time.NewTicker(200 * time.Millisecond)

	// monitor downloads
	completed := 0
	inProgress := 0
	responses := make([]*grab.Response, 0)
	for completed < len(reqs) {
		select {
		case resp := <-respch:
			// a new response has been received and has started downloading
			// (nil is received once, when the channel is closed by grab)
			if resp != nil {
				responses = append(responses, resp)
			}

		case <-t.C:
			// clear lines
			if inProgress > 0 {
				fmt.Printf("\033[%dA\033[K", inProgress)
			}

			// update completed downloads
			for i, resp := range responses {
				if resp != nil && resp.IsComplete() {
					// print final result
					if resp.Error != nil {
						fmt.Fprintf(os.Stderr, "Error downloading %s: %v\n", resp.Request.URL(), resp.Error)
					} else {
						fmt.Printf("Finished %s %d / %d bytes (%d%%)\n", resp.Filename, resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
					}

					// mark completed
					responses[i] = nil
					completed++
				}
			}

			// update downloads in progress
			inProgress = 0
			for _, resp := range responses {
				if resp != nil {
					inProgress++
					fmt.Printf("Downloading %s %d / %d bytes (%d%%)\033[K\n", resp.Filename, resp.BytesTransferred(), resp.Size, int(100*resp.Progress()))
				}
			}
		}
	}

	t.Stop()

	fmt.Printf("%d files successfully downloaded.\n", len(reqs))
}

The Goal: A challenge to IT Operations

Sat, 12 Dec 2015 00:00:00 +0000

This week I took an adventure in 1980’s manufacturing plant optimization in Eliyahu M. Goldratt’s “The Goal: A process of ongoing improvement”. The book is featured in discussions around DevOps, particularly concerning its heavy reference in Kim, Behr and Stafford’s The Phoenix Project and all things Lean.

It was an unusual read for me but has certainly changed the way I approach problems in IT operations and even in getting my daughter to school on time.

In this post, I’ll aim to condense some of the principals taught in the book, their application in DevOps and finally offer a challenge to IT organizations where application deployment is not the primary line of business

Summary

The Goal is a fictional business parable that follows Alex Rogo, a manufacturing plant manager at UniCo manufacturing. His plant (and the business as a whole) are deep in the red, despite employing some of the best management wizardry of cost cutting exercises, order expedition, localized optimizations and even fancy new robots.

Alex has a chance meeting with a physics professor, Jonah, from his university days, who challenges some of Alex’s (and my own) most basic assumptions about productivity and the reason the business exists at all. Jonah leads Alex, via provocative Socratic inquiry, to approach his manufacturing and personal issues scientifically.

The result for Alex was saving the business and some general kickassery.

I was surprised how entertaining the book was. For me the subject matter of the plant was fairly foreign and uninteresting but the story telling is very human and relatable.

Manufacturing jargon is kept to a minimum, with only enough detail given to solidify some of the abstract ideas being taught.

The book didn’t feel dated at all. The only reminder that we are not in the current millennium is the mention of dot-matrix printers and smoking in the office.

Being a work of fiction, I did doubt whether the scale of some of the successes encountered in the book could be replicated in reality. While the turnaround of the fictional UniCo is extreme, the common-sense nature of the science described lends credibility. Fortunately for us, there is now also a long history of well published successes and failures with Eliyahu’s process and similar ideas from DevOps, the Toyota Production System or Lean manufacturing.

Socratic inquiry

All of the lessons taught in the book could be condensed into a two or three page cheat sheet; even with substantial context. Unfortunately, some of the lessons are so contrary to our modus operandi that in this form they would read as nonsense; much like the notion of the world being round once did.

For example, Jonah suggests that most efficiencies gained in local areas of production actually serve to cripple the entire system, contrary to my intuition.

The beauty of the relationship between Jonah and Alex, is that Jonah never provides answers to anything. He only provides questions and validation of Alex’s assertions. In doing so, much like the Greek philosopher Socrates, Jonah teaches Alex how to think. Essentially, Jonah could give Alex a fish and feed him for a day, but instead teaches Alex how to fish, so Alex can own his own destiny.

This form of dialog caused me to continually consider my own situation. Clearly the nuances of manufacturing do not directly apply to IT operations, so some of the answers Alex discovers have no relevance to me (I don’t use heat treat ovens in the datacenter). Instead the challenges Jonah offers to my assumptions and way of thinking lead me to answers of my own.

The goal

The first question Jonah poses to Alex is “What is the goal”? In my position in IT Operations, I might have answered “uptime” or “minimizing costs” or “information security” or “good customer service”. Jonah highlights that these sorts of answers are actually just a means to an end, not the end itself.

For UniCo manufacturing, the end, or the goal, is to make money. I work for an outsourcing company, who have assigned me to a government body concerned with education. For me the goal is a little unclear, so I’ll give it some more thought and perhaps conversation with the powers-at-be. Is the goal to make money for my employer? Or to improve education for my client? Or to make money for my employer by improving education for my client? Or is the goal to assist the current government in winning the next election (not educational outcomes) so we can keep the client and continue to make money?

The point is, without clarity on the goal, you might unwittingly work to undermine it. For example, UniCo purchased new welding robots that improved local efficiency, but actually impacted negatively on their goal; to make money.

Once the goal is clear, Alex devises a succinct list of critical measures of progress towards the goal which are used to classify everything in the plant:

Throughput: the rate at which the system generates money though completed sales
Inventory: the money currently invested in assets it intends to sell (unfinished products, work-in-process, etc.)
Operational expenses: money spent to turn inventory into throughput (wages, power, etc.)

These measures are specific to UniCo’s goal of making money through manufacturing but could be abstracted to other applications.

For example in education, throughput might be the number of student who graduate and proceed into successful work placements, inventory might be the number of students currently in the system and operational expenses remain; the cost of employing staff and keeping the lawns.

Dependent events, statistical fluctuations

Jonah highlights that a manufacturing plant represents a sequence (or many sequences) of dependent events. Meaning, some events cannot happen until preceding events have completed. E.g. assembly cannot begin until all required parts have arrived. Each step in the process is also subject to statistical fluctuations; where the duration of each execution of a discrete process will fluctuate. Welding or painting a part, will always vary in time.

This is fairly self-evident, and like me, Alex assumes that the average execution time of each production line station will determine the throughput of the system (i.e. time from customer order to dispatch and receipt). In practice however, the plant is failing to keep up and the issue is compounding over time.

While hiking a group of scouts in single file through a nature trail, Alex becomes frustrated that the group of boys cannot seem to stay together and are running behind time to complete the hike. Alex identifies that the hike is much like his plant:

there is a goal: everyone arriving on time, all together
one boy cannot move forward until the next boy has moved forward
each step of each boy fluctuates in distance and duration
the distance covered by the last hiker is the throughput of the system
the energy expended is the operational expense of the system
the distance between the first and last hiker, is the inventory or work-in-process.

Much like the factory, the work-in-process is in continual growth, with the distance between front and back of the line constantly expanding, increasing risk and the energy required to compensate for the distance.

Alex’s expectation was that the average pace of the boy scouts would determine the arrival time. Instead he discovers, that it’s actually the maximum accumulated, negative deviation from the average that determines the throughput (arrival time) and inventory (distance between front and back) of the group.

Each step that each boy takes will deviate slightly from the average. Every time a deviation is in the negative (slightly slower), that deviation accumulates as the boys behind cannot pass and slow down. Additionally the distance between the hiker in front increases. Compensating for this phenomena, requires significant effort and is often outside the capacity of the boys.

Alex further models the problem by passing matches along a sequence of bowls, based on the fluctuating rolls of a dice. The throughput of the last bowl is significantly lower than anticipated.

This misconception about the capacity of the hikers being based on average pace is shared at the UniCo plant and in my own estimation.

The illusion of local efficiency

What Alex learns (and Jonah later validates) is that the whole process is actually hinged on the bottleneck (or constraint), the slowest boy scout hiker. Alex forces all of the boys to maintain pace with the slowest boy, young Herbie, by placing Herbie at the front of the pack. As a result, the boys behind start operating below capacity which in turn leads to the entire pack walking closely together (a reduction in inventory or work-in-process). The boys are no longer required to jog to catch up the gaps between them and so operational expenditure is reduced.

Now that Herbie is setting the pace for the group, his capacity gets greater scrutiny and the group realize Herbie is carrying several kilos of obscure camping equipment in his ruck sack. The equipment (or load) is distributed amongst the boys with higher capacity and suddenly the entire group is moving quicker (throughput)!

Prior to understanding this metaphor, I might have hinged the success of the group on the fastest kids (highest capacity resources). Ultimately, any efficiency gained in the faster boys (strength, speed, etc.) is a complete and utter waste when considering the goal of arriving on time and together.

Likewise, the robots purchased by UniCo were a waste. Because of their impressive capacity, they only served to increase inventory (by piling up products in front of the next work station) and operational cost (staff working overtime to catch up). The robots did increase the local efficiency of their work centers, but undermined the goal of the company, to make money, by increasing inventory and operational expenditure without increasing throughput of the entire system.

The process of ongoing improvements

The author provides a process to solve these problems, which in the parable, is devised by Alex and his team of colleagues. This is the process Alex used to rectify the hiking boy scouts, and his manufacturing plant woes:

Identify the system’s primary bottleneck/s (Herbie, or the plant’s heat treat machine)
Decide how to exploit the bottleneck (move Herbie to the front or keep the heat treat machine running during lunch breaks)
Subordinate everything else to the above decision (all kids slow down to Herbie’s pace and all other workstations prioritize parts bound for the heat treat machine and remain idle if there are sufficient parts ready)
Elevate the system’s constraints (distribute the contents of Herbie’s rucksack or install additional heat treat machines)
If a bottleneck is broken (i.e. no longer the bottleneck) repeat from step 1, but don’t allow inertia to become a constraint (in this case, market demand became the new bottleneck for Alex’s plant, but inertia caused them to stockpile too many completed products)

Change

Once the manufacturing plant is once again profitable, Alex faces the challenge of responding to changes in the marketplace and planning ahead to prevent any further decline. With further prompting from his mentor Jonah, Alex determines there are three critical challenges for managers in applying his new knowledge:

What needs to change?
What does it need to change to?
How do you execute the change?

These steps are particularly critical when approaching changes to the most difficult of all business resources to manage: people. When trying to change the culture, values and thinking of people, the book highlights two important strategies:

Socratic inquiry: leading people to answers by simply invoking curiosity. Alex did this by sharing Jonah’s questions with his colleagues and working together to find the answers.
The scientific method: create a hypothesis, A/B test the hypothesis (with a control) and share the conclusion. Alex did this by comparing the results of his hypothesis to the results attained in other UniCo plants, or simply with “before and after” results.

Contribution to DevOps

Prior to reading this book and The Phoenix Project, I would have described DevOps as:

Enabling frequent production releases by making developers and sysadmins
play nice together and using some cool new automation tool chains.

I now have a completely different perspective. In fact, the cooperative culture and tools have taken a heavy dive in significance for me. They are simply a means to an end. They were a tailored response to a discrete problem. Truth is, in my own experience in adopting “DevOps”, the CI/CD (Continuous Integration/Continuous Delivery) tools simply became the overpriced, under performing, state-of- the-art, highly “efficient” robots in Alex’s manufacturing plant.

CI/CD tools and DevOps culture were born from a specific problem. That is, that successful implementations of Agile development methodologies have significantly increased the production capacity of development teams, typically without exceeding marketplace demand. Historically, operations teams held the title of “most efficient” as software releases were infrequent. Now the tables have turned and operations has become a bottleneck in businesses achieving their primary goal; make money first, fast and forever.

DevOps addresses this problem, using Goldratt’s principals in the following ways:

Production deployment is identified as the bottleneck in achieving the goal
The bottleneck is “exploited” in that more frequent releases keep operations functioning at full capacity. Operational activities that don’t directly progress the goal are deprioritized.
Work is buffered in front of the bottleneck using “feature toggles” that operations can enable via configuration when ready
Testing and quality assurance are moved before the bottleneck using Continuous Integration and automated testing tools. This elevates the bottleneck by preventing it from processing defective goods (i.e. buggy software)
Batch sizes are reduced, increasing the work of the testing, packaging and release work centers, but increasing flow through the bottleneck (smaller changes, lower risk, easier planning, etc.)
Automation is used to elevate the operations bottleneck. E.g. one touch deployments, infrastructure as code, containers, etc.
As bottlenecks are defeated (i.e. ops can keep up with dev), other bottlenecks (prioritized according to the goal) are identified and attacked such as operational outages, technical debt, infrastructure projects, compliance problems, etc.

Application in IT Operations

Deploying applications into production is actually a very small part of what my organization does. Our focus is more in service desk, process management and infrastructure services.

Essentially, CI/CD is of very little (if any) value to us in achieving our goals. For this reason, often my colleagues express that DevOps is also of very little use to us. Eliyahu’s parable illustrates that the Theory of Constraints, a core component of DevOps thinking, is actually more relevant than ever; it’s just that the application will look different to CI/CD.

Our goal, hypothetically, is to make money. We do this by increasing throughput (projects delivered, requests fulfilled, etc.) while minimizing operational expenses and inventory (incomplete works).

Our bottleneck is not the frequent deployment of new software features. What we need to determine, is where the primary bottleneck is in achieving our goal. Hypothetically, it could be:

Service outages
Change and release management
Skills shortages
Quality assurance
Inter-team politics
Project delivery
Etc.

Improving any one of these in isolation could actually serve to undermine our goal by increasing inventory or operational expenses, without increasing throughput.

The next step, the challenge to you and hopefully the subject of a future article, is the application of Goldratt’s process of ongoing improvement to more diverse IT Operations organizations and overcoming the challenges of organizational change.