Application transition is the most challenging and risky aspect of migrating an organization to the cloud. At Geocent, we created a process for application cloud transition that is based on the Development Security Operations (DevSecOps) paradigm that focuses on automation, infrastructure as a code, security, and configuration as a code.
As described in Figure 1, a typical process for transitioning an application to the cloud includes collecting configuration documentation, using that documentation to "build-out" virtual server instances in the cloud, testing those instances, resolving issues, saving those virtual instances using the cloud provider's tools, and finally deploying them for use.
While this process for cloud transition is adequate and will accomplish the task, it presents various issues, including vendor lock-in and configuration management challenges.
Very few tools exist for exporting or moving instances and templates to another cloud provider. As such, once they are created in a new cloud environment, if necessary to move to another provider, all of the initial work used to create the specific configuration will be lost. This lack of tools to easily switch cloud providers essentially forces vendor lock-in due to the additional costs that would be required to make a change.
Configuration management is always important in managing systems and the cloud provides us with many new tools to do so, including the ability to create numerous templates that can be built upon for standard configurations. While this a great improvement from the past, it is still configuration management at a virtual machine level, i.e. the configuration is stored as a whole and there is no mechanism to extract pieces and parts of the configuration separately. This results in less re-use of configurations, as well as, difficulties in managing standard configurations.
Geocent's cloud transition process takes a similar but different approach to the typical process. Instead of just building out the system, we automate the application deployment (see Figure 2). Using various deployment automation and cloud automation tools that have gained in popularity primarily due to the DevSecOps movement, we build automation scripts that completely automate the configuration of the system. Additionally, we use software development tools and processes to build our infrastructure via automation scripts (rather than documentation and change request) and to automate the instance creation and cloud configuration.
Our approach to the cloud transition process provides various benefits, such as freedom from vendor lock-in, automation of configuration management, and configuration modularization.
In the typical approach, instances would be built-out using the vendor's tools to create, store, and manage virtual instances and configurations. These instances do not port or transfer from cloud vendor to cloud vendor. As such, in order to change cloud vendors, the entire configuration would have to be re-built from scratch. Our approach uses open source, platform agnostic infrastructure automation tools to create and manage the server configurations. Configuration data is managed as code that can be executed in any cloud environment, allowing a business to make decisions based on the cost of a provider rather than contemplating the complexity required to move to another provider.
Generally, vendors provide tools for managing instances and configurations; however, these tools tend to be based on management of entire instances, i.e. a completely configured virtual machine. We gain all of the advantages of Infrastructure and Configuration as a Code by using automation tools to automate the configuration. Because our configuration and our entire environment is created as automation code, we can manage it using robust software development tools and processes, including version and configuration management tools, such as Git. These tools allow us to track configuration changes down to the most minute detail, manage those changes using branching and tagging, and merge changes across configurations (see Figure 3).
Other software development processes that we perform to improve our approach to the transition process include verification and validation practices. Each change, once approved, is coded into the configuration and tested in development/test environment prior to moving to production. This approach allows the technical individuals to review the actual change in a test environment and removes any ambiguity that may be in the change request.
Because our configurations are in code, we can organize and manage them as modules, greatly increasing reusability, reducing the amount of time it takes to build a configuration, and ensuring standardized configurations for standard components (see Figure 4). This is especially valuable for government environments, which require specific configurations and settings to ensure proper security. There is a rigorous and time consuming validation process involved in getting an Authority to Operate (ATO) and reusable modules help save time and money by providing a level of predictability when it comes to getting an ATO.
To enable our DevSecOps cloud transition process, we have put together a set of Open Source tools (see Figure 5) that collectively allow us to organize, manage, and deploy applications into any cloud environment.
Configuration modules are managed using Git, which allows us to easily manage virtually any number of configuration changes. Configurations can be managed by module, as well as, by system, and configuration can be merged. Configuration changes are developed as code and checked into the Git repository. The system then gets the changes from the repository and deploys them to appropriate environments.
Puppet is an open source configuration management tool used to automate many common system administration tasks. It can greatly simplify the repetitive tasks that often are required of large systems with hundreds or thousands of servers and ensure that configuration instructions are carried out in a systematic predictable way across the entire virtual infrastructure. Puppet is essential in scaling infrastructure by configuring new instances on demand to production specifications as load increases. It can be deployed in two ways:
Puppet is used to ensure organizational compliance by routinely executing and reversing any deviations from the approved infrastructure baseline.
Terraform is an open source tool for managing the virtual infrastrucutre lifecycle. It is cloud provider agnostic and can be used to manage resources in today's most popular cloud providers such as Microsoft's Azure, Amazon Web Services (AWS), etc. Terraform promotes a model whereby System and Infrastructure Administrators make infrastructure or configuration changes "in code", i.e. they check-in system or configuration changes to a code repository and allow plan based on the resources and dependencies required to complete the change and will execute the plan in the most efficient way possible, often parallelizing changes that are not dependent on each other.
Terraform and Puppet complement each other to complete higher-level tasks that fall outside of the scope of either tool individually. While Terraform is an ideal tool for provisioning infrastructure across many cloud providers in a seamless way, Puppet can complete application installs and make fine-grained configuration settings on a newly provisioned Terraform server prior to putting it into service.