Interview and Book Review: Chef Infrastructure Automation Cookbook
Matthias Marschall contributes powerful tactical information for all users of Opscode Chef, from the beginner up to expert, in his recently published book "Chef Infrastructure Automation Cookbook". Beginners can read straight through gaining progressive knowledge by working through the "How to do it..." sections using open source tools and free services. The novice can use the book as a way to build up motivation to try the advanced capabilities of Chef and using them with new technologies. The expert will find it is useful as a starting point for investigating the usage of Chef to automate the non functional requirements expected of production applications in data centers.
Structure for the Beginner
The structure of the book takes the user through a series of sections that build on each other. This is explicitly called out in later chapters in the "Getting ready" sections that describe activities previously accomplished earlier in the book. More subtly though, Matthias is leading the reader along a path to build high quality "Infrastructure as code" in an enterprise environment. In chapter 1 the book describes the necessary actions to acclimate to developing Chef code in an enterprise environment. Then in chapter 2 the book puts an emphasis on quality by describing the available techniques for testing Chef code and investigating individual Chef runs. These two chapters come before diving into developing skills involved in writing good Chef code. Chapters 3 and 4 help the user to write better Chef code. Good Chef code properly divides responsibilities in the code so that the right Chef constructs are used to build data driven automated configuration. On top of proper divisions the book covers techniques for securing data, cross platform usage, and idempotent in execution. Next, begins the three chapter progression through using Chef to automate infrastructure using a progression of systems in the examples. Chapter 5 goes through the utilization of packaging systems, files, and building of source code. With software distribution understood, Chapter 6 illuminates the usage Chef to automate the installation of application and their supporting systems. Chapter 7 concludes with the automation necessary to automate the build out of systems in support of non functional requirements including: performance, availability, durability, monitoring, etc.
Motivation for the Novice
The books abundant usage of Vagrant and Berkshelf can be motivating factors for the novice Chef cookbook coder. While coders don't necessarily need Vagrant and Berkshelf to write Chef code, the usage of them can speed up the productive use of Chef and increase the usage of community cookbooks. Additionally, while it is not necessary to write or understand Ruby as a language to effectively leverage Chef, the book introduces readers to using Ruby in Chef to advance the leverage of the cookbooks. Matthias makes explicit usage of numerous community cookbooks including: apache2, HAProxy, heartbeat, iptables, MySQL, nginx, NTP, snmp, varnish, wordpress, and others. The example usage of community cookbooks helps to overcome the initial resistance some novices may feel as they grow their skill set in Chef.
The book also provides reference information for preventing the unnecessary utilization of round about mechanisms to achieve a capability provided by Chef or associated technology. Examples include:
- the "-o" command line parameter can be used with "chef-client" to override a run list currently associated with a node in Chef Server. Using it prevents excessive modifications in order to run a single recipe on a host.
- Rspec in combination with chefspec and fauxhai provide a test driven development framework for Chef cookbook development.
- Test-kitchen used with kitchen-vagrant, Vagrant, and minitest allow for integration testing of Chef cookbooks.
Multi-Node Configuration for the Expert
Chef provides intuitive constructs and flow for the configuration of nodes individually. Even the configuration of sets of similar nodes can be understood fairly quickly. However, it can be challenging to understand how to use Chef to configure nodes that depend on the current configuration and state of other nodes, not just the configuration coded for in Chef. The book includes the following sections relevant to this capability:
- Using search to find nodes
- Using search to find data bag items
- Using HAProxy to load-balance multiple web servers
- Building high-availability services using Heartbeat
- Deploying a Nagios monitoring server
Chef does not provide a mechanism for chef-client runs to be coordinated amongst themselves. Instead it provides a central storage in Chef Server with state information that clients can use to coordinate their individual configurations. The HAProxy recipe in the book uses the "app_server_role" role to determine which nodes requests will be proxied to which is an example of using the configuration stored in Chef server to configure it. Additionally HAproxy has its own capabilities for taking nodes out of the load balance configuration based on their availability. It is a combination of Chef and the technology itself to produce multi-node configurations. Another example of this is the recipe for setting up high-availability, it also depends on an external technology to perform the internode setup using broadcast, in this case it is Heartbeat. As a further example, consider that Nagios is a well known monitoring solution, however proper configuration through the community cookbooks requires usage of a variety of properly configured Chef components Including: user data bag items, services data bag items, role definition, and Chef server search. Matthias introduces the usage of Nagios by walking the user through a basic initial check, thereby helping the user overcome the complexity impedance.
InfoQ had the opportunity to speak to author Matthias Marschall on his book.
InfoQ: What is your view on using Chef for system configuration and application deployment? Or should application deployment be done through another tool?
Matthias: While the various flavors of the application cookbook have come a long way, my preferred approach to application deployment is using Capistrano or a continuous deployment tool, which pushes (rather than Chef pulling) the latest releases.
InfoQ: The second chapter has some advanced material, why did you place it ahead of chapter three and four?
Matthias: I had two reasons: number one is that it is very helpful for the following chapters if you're able to find out what goes wrong in your Chef runs. The second reason is that I'm very much in favor of a test first approach for your cookbook. Because of that I wanted to show the reader how to write tests for their cookbooks before going into more detail about the cookbooks themselves.
InfoQ: What is one (or a few) new features Chef needs to improve its capabilities?
Matthias: I would wish for a tighter integration with version control systems. All your infrastructure code needs to be under version control. With the current Chef Server approach, there are two sources of truth: the source repository and the Chef Server. There is a risk that both of them diverge over time. A feature to make sure your version control system and your Chef Server are always in sync would be great!
About the Book Author
Matthias Marschall is a Software Engineer "Made in Germany". His four children make sure that he feels comfortable in lively environments, and stays in control of chaotic situations. A lean and agile engineering lead, he's passionate about continuous delivery, infrastructure automation, and all things DevOps.