A New Kind of DevOps Database Platform
Accelerating time to value with efficiency and safety, for digital transformation
By Marc Hornbeek
Engineering DevOps Consulting
Introduction
Many enterprises are journeying towards digital transformations, with DevOps continuous delivery of business applications, and cloud adoption central to their strategies. However, challenges with the integration of databases into the DevOps continuous delivery pipelines are preventing enterprises from achieving the full potential of these transformations.
This white paper explains:
- Why is a new kind of DevOps Database Platform needed?”
- “How would a new kind of DevOps Database Platform work?”
- “What an organization needs to do to transform to a new kind of DevOps Database Platform? “
Why is a New Kind of DevOps Database Platform Needed?
There are multiple reasons why a new kind of DevOps Database Platform is needed, which can be categorized as follows:
- Drivers for DevOps Database
- Compatibility Requirements
- Integration with Current CI/CD Pipelines
- Integration with Portfolios of Value Stream Management
- Infrastructure Cost efficiency
- Productivity for developers, testers, and operations
- Security and Governance
- DevOps Performance
Each of these reasons are explained in the following paragraphs.
Drivers for DevOps Database
Redgate’s “2021 State of Database DevOps” report indicates that data is integral to enterprise operations, and data delivery must be enabled by DevOps practices. The “2019 State of DevOps Report” by DORA, indicates that “high performing” enterprises are more likely to have implemented DevOps and Database DevOps practices. On average, Elite performers are considerably faster in delivering database changes. The main drivers for automating delivery of database changes includes improving the delivery process in terms of speed, reducing downtime, and improving team efficiency and collaboration.
Figure 1 indicates that increasing speed of databases changes is the top ranked driver for DevOps databases, followed by the need to free up developer’s time for value-added work, minimizing downtime, facilitating collaboration between development and operations, reducing risk of data loss during deployment, and faster resolution of issues and audit and regulatory requirements.
Figure 1: Rank of DevOps Database Drivers
(Credit: “2021 State of Database DevOps” by Redgate)
Compatibility Requirements
Enterprises often have a variety of Database systems, deployment environments and DevOps toolchains.
A DevOps Database Platform must be compatible with:
1. Any database (E.g., Relational, Non-Relational, Vendor specific such as MSSQL, Oracle, and others).
2. Deploy on popular operating environments (E.g., Windows, Linux, VMs, and Cloud).
3. DevOps processes that support containers, Kubernetes, OpenShift, and Continuous Integration systems such as CircleCI, Azure DevOps, and others.
4. Compatibility with data engineering tooling, such as data masking, synthetic test data generation, and others.
Integration with Current CI/CD Pipelines
Applications need distinct DevOps pipelines. As indicated by Figure 2 from “Engineering DevOps” by Marc Hornbeek, Database changes corresponding to application changes are coordinated across multiple pipelines for each stage in the CI pipeline.
Provisioning secure and realistic database development and test environments also presents added challenges. according to the “2021 State of Database DevOps” “, 69% of respondents share database development environments among developers. More than 50% of respondents also use full-size copies of production databases in development and testing environments, and only 30% use data virtualization technologies for on demand data. The second most popular way of provisioning data is using a subset of the production data (38% of respondents), emphasizing the importance of working with realistic data. Only 30% reported use of synthetic data created for each application change.
A new DevOps Database is needed to allow each pipeline to include databases without the complexity and overhead of coordinating each application pipeline with separate database pipelines.
Figure 2: DevOps Pipelines and Databases
(Credit: “Engineering DevOps”, Marc Hornbeek)
Integration with Portfolios Value Stream Management
With DevOps, each CI/CD application pipeline is part of end-to-end value stream that needs to be managed from planning stage all the way into deployment to production. In well-engineered environments, Value Streams are operated asynchronously, or can bottleneck another while waiting for the slowest to catch up. Value stream management practices are needed to oversee the co-ordination of application changes with database changes needed to complete the tasks for each application pipeline. This is illustrated in Figure 3. A DevOps Database is required to operate end-to-end from planning through operations for each value stream. To do that requires the DevOps Database have capabilities to be orchestrated and operations made visible in the Portfolio of value streams.
Figure 3: Value Stream Management over Portfolio of DevOps and Databases Pipelines
(Credit: “Engineering DevOps”, Marc Hornbeek)
Infrastructure Cost Efficiency
To make the DevOps pipelines and Value Streams efficient, a DevOps Database instance should be served efficiently as a virtual database for each stage of the DevOps pipeline and Value Stream as illustrated in Figure 4. The large number of databases would be costly using current approaches. By virtualizing the DevOps Database, the same infrastructure can be used to host many DevOps Databases, with dramatic savings of infrastructure costs.
Figure 4: DevOps Database Instances
Productivity for Developers, Testers and Operations
Virtualized DevOps databases reduce processing time and labor that is required for developers, testers and administrators, that would otherwise be applied to standup and co-ordinate backup and restore operations for databases needed. Virtualized DevOps databases enables developers and testers to be more productive by focusing on their value-added tasks such as development, testing, debugging and retesting application and database software, rather than toiling over administering the database.
Security and Governance
Distributed applications running over distributed infrastructures required for modern enterprises present an increasing number of attack surfaces, and increasingly complex security and governance. Virtualized databases improve data security and governance to ensure security for short-lived and increasingly short life-spans of DevOps databases.
Security requirements include database version management, privileged access controls, data masking, and encryption protection for sensitive data. Examples of governance requirements include tracking of databases used in development and production logging of database usage for compliance audits. DevOps database operations are improved with restful APIs for integration with DevSecOps and Governance tools and practices.
DevOps Performance
According to DOR ’s 2019 State of DevOps reports four key metrics that are most important for DevOps environments, and are the strongest indicators of DevOps performance. These metrics need to be supported and tracked for DevOps pipelines and value streams:
1) Lead time. This is the time measure from the time a developer commits a change until that change is deployed to production.
2) Release Frequency. How often do the pipelines and value streams release to production?
3) Mean-Time-To-Recover. How long does it take to recover from a failure?
4) Change Failure Rate. How often are changes rejected?
How Would a New Kind of Database DevOps Platform Work?
The preceding sections outline requirements for a new type of DevOps Database Platform. Figure 5 lists the
capabilities required and maps them to the drivers and requirements.
Figure 5: DevOps Database Platform Capabilities
Automated database provisioning: the ability to deploy a database using automated procedures.
Database cloning: writable database environments are delivered on demand, based on a built versioned image.
Distributed repository: database images can reside on Linux and Windows servers, as well as storage systems.
Simple declarative builds: a simple, predefined method to make the creation of versioned data images simple, while enabling user/group access controls, data masking, and other data engineering.
Scalable data cloning: database clones are served on demand, with many simultaneous clones served from a single image.
Comprehensive database support: the repo supports all common relational, NoSQL, machine learning, and file based data stores.
Versioned databases: each image becomes a versioned, immutable DevOps artifact.
Deploys equally to all cloud environments: DevOps Databases can be deployed and operated on any on premise infrastructure or cloud.
Cross platform: DevOps Databases can be deployed and operate on common operating system environments such as Windows and Linux.
Security features including Access Management, data masking, and encryption: database images easily incorporate access control, data masking, and synthetic test data, via SQL scripts, batch files, or command line tools.
Compatible with CI/CD platforms: DevOps Databases can be deployed and operate equally well in multiple CI/CD platform tools such as Azure DevOps, Jenkins and CloudBees.
Compatible with containers and container orchestrators: DevOps Databases can be deployed and operate equally well in multiple containers environments such as Docker, Kubernetes and OpenShift. Deliver writeable data at the speed of DevOps: Deliver writable databases in seconds. Easy user-interface for user self-service: is self-explanatory.
API for easy integration with DevOps tools: An API is available to enable integration of the DevOps Database control plane with other tools.
Audit logging: The DevOps Database includes a complete audit log of operations.
What is Needed to Transform to New Kind of Database DevOps Platform?
A DevOps Database platform in an Enterprise takes more than simply buying and installing software and telling everyone to use it. With multiple entrenched database platforms in place, there are practical and political considerations that must be overcome. Introducing change to a large group of applications in an enterprise can be chaotic, and will often be more costly and take too much time if the changes are not well coordinated in accordance with a carefully crafted strategy and implementation roadmap that leaders and team members are
aligned with.
A Seven-Step Transformation methodology, illustrated in Figure 6, is a logical approach to drive the transformation that is described in the book “Engineering DevOps” by Marc Hornbeek͘ The transformation requires changes not only to platform technology, but also people and process aspects of the Enterprise.
Figure 6: Seven-Step Transformation Blueprint
(Credit: “Engineering DevOps”, Marc Hornbeek)
The seven steps are as follows:
1) Leadership alignment - Business level managers that oversee applications need to agree on the vision to migrate to the DevOps Database Platform and establish business goals that will be most important to accomplish for the Enterprise.
2) Team alignment - Leaders of each of the Application Development teams and DevOps teams need to translate the vision from the business managers to goals that best fit the priorities of each application. Bear in mind that there may be legitimate differences between each application. Rather than “boil-the-Ocean” by trying to change the pipelines for all applications at once, it is wise to pick a small number of “model” applications to implement first and use those as references for the period of the transformation.
3) Assessment of current state - Take an inventory of current pipelines, capabilities, processes, and people practices for the applications, as well as assess the gaps between the current state and the goals set in the prior step. Figure 7 is an example of a checklist that can be used to determine the current priority of requirements relative to current database capabilities in use by the organization.
Figure 7: DevOps Database Platform Requirements
4) Strategic alignment around a solution roadmap - A cross functional Dev/QA/DB/Ops team, with expertise on the organizations’ databases, applications, and pipelines, analyze the results of the prior step and recommend a solution to transition for each pipeline to the new platform. Each organization needs to develop their own roadmap that matches the organizations’ priorities͘ Figure 8 is an example of a high-level roadmap for guiding implementation steps. The steps in the diagram follow “The Three Ways of DevOps͘” fter lignment of Leaders and Teams, focus on building the DevOps Database Platform into the “front-end” Continuous Integration (CI) capabilities of DevOps application CI/CD pipelines. Once a solid CI model application has been achieved satisfactory, then integrate the DevOps Database Platform into of the “back-end” Continuous Delivery pipeline to complete fully integrated Continuous Flow - The First Way of DevOps. Once an application has achieved Continuous Flow then it makes sense to add feedback mechanisms to monitor flow, in accordance with the Second Way of DevOps - Continuous Feedback.
Finally, as the application pipelines mature and the team gains confidence, it makes sense to experiment and introduce innovations to optimize the flow.
Figure 8: Implementation RoadMap Example
5) Realize an MVP of model applications - Implement the changes from the roadmap from step 4 and evaluate whether it meets the goals for the model applications. Iterate the model implementations until they meet the minimal requirements.
6) Operationalize the model applications - Determine and implement the remaining Service Level Objectives (SLOs) that are needed to satisfy Business, App-Dev users, QA, Ops and Admin users for the model applications in production.
7) Expansion to other applications - Apply steps 1 through 6 for additional applications until all the application pipelines are covered by the new DevOps Database Platform.
WHAT THIS MEANS
Digital Transformations are an imperative for modern enterprises and organizations to remain competitive. However, the transformations of many enterprises and organizations are seriously impeded by the challenges of integrating database and database into their DevOps application pipelines and value streams. A new DevOps Database Platform is needed that can allow the databases to be treated like other DevOps artifacts that are orchestrated and automated with the DevOps toolchains. This white paper explains the “Why?”, “How?” and “What?” for realizing a DevOps Database Platform. I also want to express my appreciation for time spent with the Windocks team, who has pioneered many
of the concepts outlined in this paper, and contributed valuable insight. This paper is reproduced by Windocks with my permission.
REFERENCES
1. “2021 State of Database DevOps”, Redgate
2. “2019 State of DevOps”, DOR
3.“Engineering DevOps”, Marc Hornbeek
4. Windocks.com
5. EngineeringDevOps.com
About the Author
Marc Hornbeek, a.k.a., DevOps-the-Gray esq. is a globally recognized expert for Software Testing, DevOps, DevSecOps, and SRE. He is CEO and Principal Consultant at Engineering DevOps Consulting , author of the book Engineering DevOps , Analyst for the Accelerated Strategy Group and Ambassador and Author for The DevOps Institute .