Trends

Data as a first-class Docker Artifact

In this blog we take a look at how Windocks approaches relational data management differently from other container engines, by including data as a container artifact.


Artifact: an object made by a human being, preserved as an item of cultural or historical interest.

This blog is focused on artifacts, items of human origin, which are commonly associated with ancient cultures. In our introduction we celebrate native Pacific coast North Americans, and in particular the Chilkat blanket. These blankets are unique works of art, and part of a culture that celebrated the “potlatch,” a ceremonial feast and dispensing of gifts that was marked by extreme generosity.

A sea of digital ink has been spent on persistent data for Docker containers. In an era of cloud native and application driven automation, it is odd that developers are required to deal with storage volumes, mount points, and complex directory structures. Volumes involving a sizable data set can involve hours to provision. The approach seems to fall short of what a “cloud native” development experience should be.

As an independent port of Docker’s open source project to Windows, Windocks responds to customers’ needs for full stack SQL Server support. In our third year of public operation customers have led us to decisions that differ in some respects from the mainstream Docker ecosystem. While Windocks supports the use of volumes, we’ve also introduced new methods for working with relational data. This article outlines how Windocks has evolved, with particular emphasis on how relational data is a first-class Docker image artifact. The topics covered include:

Database cloning: supports delivery of Terabyte class databases in seconds, with virtually no storage costs. Windocks supports both Windows Virtual Hard Drives (VHDs), and storage array volume snapshots, with support on any public cloud or on-premise infrastructure.

Data included in images as a first-class artifact: relational data gains equal status to code and applications, with immutable imaging that incorporates enterprise security policies, with meta-data for enterprise audit and reporting.

Automated and abstracted infrastructure: Windocks manages the complexity of storage volumes, mount points, and container state, and provides DBAs with easy to use tooling to create and manage data delivery without involvement of storage admins.

For SQL containers and instances too: not surprisingly, enterprise customers have a notion that technology should serve the entire enterprise. Windocks supports database clone delivery for both SQL containers and conventional instances, including SQL Server environments back to SQL Server 2008, along with Microsoft’s new SQL Server containers on Linux.

Windocks elevates relational data to be a first-class artifact in the container ecosystem, included in the immutable image along with code and applications. As an immutable artifact, data gains meta-data, with enterprise security and privacy policies for use throughout the organization. For the first time an authoritative data catalog can be published, to support development and test, as well as other uses.

Database cloning

Docker’s open source project has achieved industry-wide support for stateless application packaging and delivery, but use with relational databases is inhibited by database size and complex integration with storage systems. Windocks addresses the issue of fast and reliable data access with database cloning based on Windows Virtual Hard Drives (VHDs), or storage volume snapshots. These two forms of cloning deliver read/write supporting Terabyte data environments, in seconds, with virtually no storage consumption.

Windows based cloning is supported by Virtual Hard Drives (VHDs) with databases restored to the VHD using full or differential backups or database files. Once the VHD is built, additional VHD “clones” (differencing disks) are provisioned on demand using the parent VHD as a read-only source. VHDs are proven in support of Hyper-V virtual machines, and scale to support scores of simultaneous clones. An example Dockerfile illustrates a basic VHD based clone, created from three full backups.

artifact 2 714 x 168

Modern storage systems support similar capabilities with volume snapshots. A source volume can be cloned in seconds, and occupy minimal incremental storage, with the parent volume supporting read/write child volumes. Volumes are mounted to a Windocks host, where they can be mounted to SQL Server instances via SMB or NFS file shares.

Database clones as a Docker image artifact, for containers & instances

Database clones are useful, but must be combined with configured SQL Server instances to include user and group access, database or column level encryption, External Key Managers (EKM) support, and other configurations. Windocks accomplishes this by including the parent VHD or storage volume in the image along with the SQL Server configuration. Once built images deliver data environments to conventional SQL Server instances and containers. The Windocks Management Server below shows one image for delivery of data to SQL Server containers, and a second for delivery to conventional SQL Server instances on the LAN. Users provide the run time parameters needed to support data delivery.

artifact 1 750 x 240

Images are built with dockerfiles that define the target (SQL Server containers or instances), and the build-time parameters needed. In the example dockerfile below targets a local SQL Server 2017 container with a VHD clone based on three full SQL Server 2017 backups, and includes two scripts. Scripts can be applied during the image build (“build time”) for data masking, and others can be saved and applied during the container “run time.” The application of scripts is determined by relative position to the environment variable: ENV USE_DOCKERFILE_TO_CREATE _CONTAINER=1. Scripts above the environment variable are applied at build time, and scripts following the environment variable are applied to each container as it is provisioned.

artifact 3 766 x 180

Built for developers and DBAs, and existing infrastructure

In addition to supporting database clones in Docker images, Windocks abstracts and automates storage infrastructure to enable DBAs and developers to operate without involvement of storage admins. Windocks provides life-cycle management of storage volumes (or VHDs), mount points, containers or instance run-time environments, and cleans up the moving parts when an environment is “deleted” by the user. As an automated, declarative system, Windocks is dramatically easier to implement and maintain compared to custom scripts that are typically involved in storage operations.

Windocks supports existing applications, systems and infrastructure, simplifying the deployment and use in any organization. Windocks supports database cloning and SQL containers for all editions of SQL Server 2008 onward, providing a full decade of backward compatibility. Windocks also supports Active Directory and Windows authentication, use of existing storage arrays, and SQL Server applications. All VSS and SQL writer apps will interoperate with Windocks based containers.

An automated system managed by DBAs and developers that integrates with existing infrastructure and processes, and runs wherever Windows Servers are supported (on any public cloud or on-prem infrastructure), leads to rapid evaluation. Windocks evaluations and pilots are often provided to developers within a day of installation, and evaluations are often complete in one to two weeks. There is no need for complex multi-month pilots when evaluating Windocks.

Summary

Windocks has evolved to better address Windows enterprise needs for complete SQL Server environments for dev/test, reporting, and other needs. The solution is built on a faithful port of Docker’s open source project, but unlike other Docker implementations Windocks includes database cloning and includes the database clone in the Docker image. Windocks also supports existing enterprise infrastructure, including Windows auth and storage arrays, and is unique in supporting data environment delivery to MS SQL containers as well as conventional SQL Server images.

Get started in discovering the benefits of Windocks by downloading a free Community Edition..

Similar posts

Get notified on new test data management insights

Be the first to know about new insights on DevOps and automation in the test data management space.