Features DevOps Lead image: © J. Allspaw and P. Hammond 2009

Building better software on schedule with DevOps

Collaboration Station

DevOps makes IT departments more efficient and makes their employees happier – but what is it? We describe some basic ingredients of the DevOps recipe. By Mathias Huber

Buzzword alert! The term DevOps, which includes components of software development and operations, is being bandied about wherever you look: You'll find abundant references to DevOps in blogs, books, articles, training courses, and software.

This new style of cooperation between developers and admins was released on an unsuspecting audience by John Allspaw and Paul Hammond (among others) at California's Velocity Conference 2009. Their slides [1] describe how techies from the photo community Flickr achieve 10 or more deploys per day.

The approach described by Allspaw and Hammond is now part of everyday business in other Internet companies. "My impression is that you can no longer survive economically as a company with the old work models," said Schlomo Schapiro, System Architect with Immobilienscout24, in an interview with the author.

IT professionals describe themselves as DevOps specialists with increasing frequency, and some companies even have DevOps departments. The "2014 State of DevOps Report" [2] indicates that 31 percent of more than 9,000 respondents see their position as a DevOps Engineer. According to the survey, 16 percent work in a DevOps department.

The report, with 46 percent of respondents from the United States and 25 percent from Europe, was commissioned by Puppet Labs, IT Revolution Press, and ThoughtWorks. Keep in mind that these companies offer software, publications, and services related to DevOps, which means the way this work method is described is likely to be more positive.

My analysis of these self-assessments showed the following: Departments with above average performance use DevOps techniques. The deployment frequency, the time required for changes, and the mean time to recover are used as the metrics for determining the IT department's performance in the case study. This correlation might make you curious as to what DevOps followers do differently.

DevOps Principles

The DevOps principles are still basically those described in 2009 (see the "DevOps Rules" box). The key here is closer cooperation between developers and admins, usually in collaborative, product-specific teams. No longer are developers solely responsible for new software features and admins for reliable and efficient IT operations (Figure 1) – they collaborate on all of these goals.

Figure 1: The DevOps philosophy abolishes the traditional boundaries between development and operations. © J. Allspaw and P. Hammond 2009

A YouTube video from Rackspace offers a succinct introduction to DevOps [3]. The short film explains that DevOps focuses on the deployment of small features for production systems, as the agile camp has recommended for some time. The idea is that these small features can be easily tested and rolled out without major interventions in the infrastructure.

Against Reform Gridlock

This policy of small but frequent steps keeps internal users or customers from having to wait too long for new features, and it reduces the time to market. At the same time, the procedure ensures that the development and operating system environments do not evolve separately.

According to the 2014 DevOps study, successful departments install new code on production systems 30 times more often than their less successful competitors. To progress smoothly, more actions are needed. All code must be managed with a version control system, which allows the team to roll back changes in case of problems. Continuous delivery and automatic building of the software from the latest sources prevents waiting times.

The stakeholders also integrate tests into this process that also run automatically in the course of a build, after each code change, to ensure the quality. Continuous-integration software, such as Jenkins [4] or the Travis CI service [5], helps the followers of DevOps.

Also, more than just the source code of the applications is version controlled. The infrastructure is increasingly being implemented in the form of software. All code that admins write for operations must thus also pass through Git or Subversion. This improves the mean time to recovery for infrastructure problems, because admins can quickly roll out an older version.

To distribute the configuration, your best bet is an automation system, such as Chef [6] or Puppet [7]. Monitoring software also ensures that admins discover errors at an early stage and can take preventive action before major failures occur.

Metrics

DevOps practitioners also apply metrics that reflect the work processes: How long does it take for a new feature to reach operational status? How many tickets are open and how quickly is the service desk processing them? Does the IT department comply with service-level agreements with customers and with internal operating agreements? A company needs such performance indicators to determine whether it benefits from newly introduced methods.

However, just correlating DevOps methodology and performance, as in the 2014 study, is not likely to convince decision makers; practitioners must provide proof individually. Additionally, some departments publish the metrics – for example, on a display in the office or the cafeteria. Whether these figures serve to motivate or intimidate depends on the general atmosphere at work, I suppose.

A Question of Culture

In addition to the more tangible aspects of the new IT methodology, which have much to do with technology and tools, don't forget the human factors that influence the work in IT. This includes job satisfaction, for example, which DevOps can help improve, thanks to productivity increases. Before that can happen, however, a few requirements need to be fulfilled.

The non-technical conditions for DevOps [8] can be described as organizational culture. This includes not seeking scapegoats in case of errors and problems (Figure 2). If the developers put the blame on the Linux admins, who in turn point to network people, this approach is of little benefit to the users. Gene Kim describes this vividly in his instructive story The Phoenix Project (see the "DevOps Novel" box).

Figure 2: Blaming breakdowns on others is frowned upon in DevOps. Instead, the idea is to learn from mistakes. © alphaspirit, 123RF

DevOps Novel

The Phoenix Project [10] is a work of fiction, which at the same time teaches DevOps principles. It was written by entrepreneur Gene Kim along with several co-authors. As a student, Kim programmed the intrusion detection tool Tripwire and later founded the company of the same name. Since 2000, he has focused on IT organization, written books, and presented keynotes.

The novel tells the story of Bill Palmer, who surprisingly is promoted to head of company-wide IT operations, although he was only the head of a small department at the time. In his new role, Bill has many difficulties to cope with: On his first day, payroll accounting goes haywire, and the company makes negative headlines in the press. Additionally, the new software release – code-named "Phoenix" – is already two years behind schedule. As if all of this were not enough, the Sarbanes-Oxley auditor is waiting in the wings with a long list of complaints.

Time is short and Bill does what he can. He first discovers what is wrong with the system: The right hand has no idea what the left hand is doing; Most system knowledge is concentrated in a single employee, and no one adheres to processes or writes documentation.

The problems described, the characters, and the dialog are so true-to-life that anybody who was worked in IT will empathize with the sufferings of the protagonist. The company needs some sort of change management tool so that the changes do not get in other people's way. Wes, an employee from Bill's department, protests: "It takes twenty minutes to fill out the whole damn fields, even if it's just a simple five-minute task!"

If you follow Bill's progress, you can learn much about the problems of an organization – and how to solve them. Especially when the mysterious Dr. Reid appears, who becomes a kind of mentor for Bill.

Gene Kim, Kevin Behr, George Spafford:

The Phoenix Project

IT Revolution Press, 2013

350 pages

US$ 17 paperback

US$ 9.99 Kindle

UK£ 13.42 paperback

UK£ 6.37 Kindle

Errors Are Natural

In their lecture in 2009, Allspaw and Hammond called for a healthy attitude toward failure: Mistakes would always happen, they said, but you can fix them and learn from them – for example, by keeping the remedy in a knowledge base or setting up a contingency plan. Netflix takes this principle to extremes. It uses a tool called Chaos Monkey [9], which switches off randomly selected services to test the resilience of the cloud and train the work processes for troubleshooting.

The values of DevOps culture also include respect: Prejudices against developers as being lazy and reckless or stereotypes of admins as bureaucratic naysayers have no place here. Everyone works toward a common goal and shares opinions and knowledge. It is unacceptable, for example, to withhold information from others, cover up your own mistakes, or strengthen your own position. Such power games cost time and energy and impair the company's success. Instead, DevOps relies on trust between all parties, including, for example, admins allowing programmers to make certain changes to systems.

Exemplary

Of course, employee attitudes cannot be planned at a board meeting. This is an emergence process that typically requires slow changes. Consultant Mandi Walls [8] recommends, among other things, that senior employees and respected professionals act as role models. Additionally, a well-defined pilot project is best suited, in her opinion, for testing and introducing the new approach.

DevOps for New Projects

Udo Seidel, Manager of Linux Strategy and Server Automation at Amadeus, an IT service provider for the travel industry, sees this in a similar way: "DevOps is a paradigm shift that historically evolved data centers like Amadeus cannot easily undertake. This requires rethinking and must be reflected in the structure of the organization and our whole process engineering. We start with the sections that include new lines of business. This means you can start something more like a greenfield site and have less historical ballast."

Introducing DevOps might not be quite as easy as some enthusiastic blog posts and conference papers suggest. However, the goals that the system can help you achieve are so attractive that it is worthwhile for many companies to at least try a pilot project. The references mentioned in this article are designed to help you.