Today we are announcing the open source release of octocatalog-diff: GitHub’s Puppet development and testing tool.
GitHub uses Puppet to configure the infrastructure that powers GitHub.com, comprised of hundreds of roles deployed on thousands of nodes. Each change to Puppet code must be validated to ensure not only that it serves the intended purpose for the role at hand, but also to avoid causing unexpected side effects on other roles. GitHub employs automated Continuous Integration testing and manual deployment testing for Puppet code changes, but it can be time-consuming to complete the manual deployment testing across hundreds of roles.
Recently, GitHub has been using an internally-developed tool called
octocatalog-diff to help reduce the time required for these testing cycles. With this tool, developers are able to preview the effects of their change across all roles via a distributed “catalog difference” test that takes less than three minutes to run. Because of reduced testing cycles and increased confidence in their deployments, developers can iterate much faster on their Puppet code changes.
octocatalog-diff, let’s address the existing solutions and the reasoning for creating a new tool.
There are three main strategies for Puppet code testing in wide use, and GitHub uses all of them:
Deployment testing – actually running the Puppet agent (possibly with
--noop to preview actions without actually making changes) allows the developer to review log files or examine the system to see if the results are as intended.
Automated testing – this may include unit tests with
rspec-puppet, acceptance tests with
beaker, syntax checking
puppet parser validate or linting with
puppet-lint. These types of tests often run in a Continuous Integration environment to verify that the code meets a set of specified criteria.
Catalog testing –
octocatalog-diff and Puppet’s
catalog_preview module both allow comparison of catalogs produced by two different Puppet versions or between two environments.
GitHub needed a catalog testing approach that could run from a development or CI environment, because for security reasons only a small number of engineers have direct access to the Puppet master. Because
catalog_preview is designed to be fully integrated into the Puppet master, it would be inaccessible for a large portion of GitHub’s Puppet contributors, and as such it was not the right fit. Therefore, we embarked upon our own development of a tool that could run independently of a Puppet installation, and produced
This screen shot shows
octocatalog-diff in action, as run from a developer’s workstation:
In this example, the developer is comparing the Puppet catalog changes between the master branch and the Puppet code in the current working directory. Two resources are being created (an Exec resource to create the mount point, and a Filesystem resource to format
/dev/xvdf). Two resources are being removed (the old Exec resource to change permissions on the work directory, and the old Filesystem on
/dev/xvdb). And one resource is being changed (several parameters of the mount point are being updated).
The output was generated in under 15 seconds, obviating a traditional workflow of committing code, waiting for CI jobs to pass, deploying code to a node, and reviewing the results. The process that generated this output did not require access to, or put any load on, either the Puppet master or the node whose catalog was computed.
The next graphic shows output from
octocatalog-diff when run via a distributed CI job, to preview the effects of a code change on nodes across the fleet:
In this example, the developer wishes to see which systems will be affected by a particular change to the Puppet code. The output from
octocatalog-diff reveals that the changes affect certain GitHub API nodes. The developer can use this information to test deployment on just those six representative systems instead of hundreds or thousands of nodes. This cuts down on unnecessary testing and provides confidence that there will not be unexpected side effects, allowing the developer to complete the work more efficiently and with less risk.
octocatalog-diff has several useful features:
octocatalog-diff is able to compare catalogs obtained in the following ways:
octocatalog-diff is being used in “catalog only” mode as a Continuous Integration (CI) job in GitHub’s Puppet repository. Upon every push,
octocatalog-diff compiles the catalogs for over 50 critical roles, using real node names and facts, to ensure that changes to one role do not unexpectedly break Puppet catalogs for other roles. In addition, developers use
octocatalog-diff in “difference” mode to preview their changes across the fleet, which has enabled them to perform major refactoring with minimal risk.
Over the past year, GitHub has successfully upgraded from Puppet 3.4 to 4.5, migrated hard-coded parameters from thousands of manifests into the
hiera hierarchical data store, transitioned node classification from hostname regular expressions to application and roles, expanded roles to run in different environments and in containers, and upgraded roles to run under new operating systems. Using
octocatalog-diff to predict changes across the fleet, a relatively small number of developers accomplished these substantial initiatives quickly and without their Puppet changes causing outages.
While we find
octocatalog-diff to be reliable for our needs, there are undoubtedly configurations or customizations within others’ Puppet code bases that we have not anticipated. We welcome community participation and contributions, and look forward to enhancing the compatibility and functionality of the tool.
We acknowledge and thank the Site Reliability Engineering team at GitHub for their suggestions and code reviews, and the other engineers at GitHub who worked patiently with us to diagnose problems and test improvements during the pre-production stages.