Test Data Management
Test data management is the process of creating and delivering data appropriate for development and testing.
Data could be relational databases, no-SQL databases, row column data, images and more. Production data copies, generated data, schema only databases are all valid forms of test data.
Windocks offers containers, cloning, and automation to implement DevOps for test data management.
Test data requirements
Delivery on demand
Test data must be available on demand to development and test teams as well as to automated DevOps pipelines - via user interface, command line line or API.
Type of test data aligned to dev, test requirements
Based on the type of applications and testing, the type of test data required could be production data, generated data, or an empty database with correct schema. A best practice is to test with production data and combinations of generated data.
Obfuscation of sensitive data (PIIA)
To comply with data security and privacy regulations, senstive data from production such as email addresses, birth dates, social security numbers must be obfuscated. In the case of relationa databases, the obfuscated data must also satisfy database constraints such as foreign keys and formats.
Access controls
Each set of test data should be available only to those authenticated and authorized. A user or program should have login credentials or a token to access data. Further, certain data should be accessible only to certain groups of people.
Steps to get started with test data management
1. Install Windocks
Download the Windocks Community Edition or email support@windocks.com for a full featured evaluation edition.
Provision a Windows Server VM (Server 2016, 2019, or 2022), install SQL Server (for SQL database delivery) and then install Windocks as described here.
For Oracle database delivery, also install the Windocks service for Linux as described here.
2. To mask sensitive data, install the built-in masking software and create the mask
Email support@windocks.com for the masking add-on. Open the database in a SQL Server or Oracle and use the masking application UI to create the mask.
3. Provide the spec to the Windocks orchestrator to build the image
Specify:
- the path to one or more SQL backup or database files or Azure SQL BACPAC files, which database cloning to use Windocks database cloning or volume cloning from other companies)
- where to deliver the database clones (Windocks SQL Server Windows containers or SQL Server instances)
- how often to refresh from production, customizations such as database scripts to be applied
- which masking software to use (Windocks masking, other masking solutions or scripts)
- authorization controls
The spec is provided in the form of a dockerfile. Build the image via a web application, command line or REST API.
4. Deliver customized database clones on demand from the image
From the image, deliver database clones to containers or instances. The image you built defines where the databases are delivered to. You may define an image that delivers containers with clones as well as clones to instances. To deliver containers and clones, use the web application or use the docker command line or the REST API.
5. Use the writeable clones for testing
Database clones are delivered either to containers or instances based on the image you built. If you delivered to an instance, then connect to that instance for the database clone. Clones are writeable. Containers are delivered on the machine running Windocks on specified ports. Access these containers with regular database tools or command lines or connection strings.
SQL Server containers are accessed with SQL Management Studio or Azure Data Studio or SQLCMD. Connect with the instance name (such as INSTANCE10002 when the container port is 10002). Or you can use the form IPAddressOfWindocksServer,10002 to connect to the SQL container running on port 10002.
Oracle containers are accessed with SQLPlus, RMAN or UI tools for Oracle. You may put in entries in tnsnames.ora for the containers as you would for Oracle instances. With SQLPlus or RMAN, use the form SQLPLUS sys/password@IPAddressOfWindocksLinuxServer:HostPort/orclcdb as sysdba