AgileData.io DataOps Testing

Testing in the data domain is always seen as a secondary focus, something we know we should do but the time we have to do it gets squeezed as we near the end of the iteration and push to get to Done Done and release to production.

In addition the time we require to test our changes proportionally increases in line with the amount of code, data and analytical models we have have already delivered.

With a DataOps lens our goal is to automate our testing, to remove this manual effort.

However in the world of data and analytics, we need to adopt multiple patterns to automate our tests.

Testing Perspectives

Testing is what we term a very large elephant.

When two or more people talk about testing they will often be viewing testing through a different lens, and this difference in context will result in a conversation full of mis-communication and friction.

Some of the perspectives we find useful when understanding which context a person is talking about are:

Use cases we care about

Applying a perspective of what use cases could break something and therefore we should have a testing pattern in place to mitigate this risks, we have the following use cases:

Source system changes

and effects the structure of the data we collect and how we land it.

Developer changes a piece of code

and inadvertently effects a related piece of code or data.

Transformation rules rely on a fixed set of values

for example there is only four distinct values in a field and the source system adds a fifth.

Source system data delay or outage

where the source system does not send data to us for a day.

Source system data spike

where an unusual update to a large number of tables, records or events is made in the source system.

New data collection confidence

where we need to collect data from a new source systems and make sure we have collected it all.

For each of these we need a testing pattern that mitigates the risk of these happening.

Industry standard tests

Applying a perspective of standard test types and terminology we have in the Information Technology domain we have the following:

Accessibility Tests

test that an application is accessible for as many people as possible, verifying such things as the presence of alternate text tags for images for the visually impaired.

Component Tests

check the various services that the code is dependent on, such as a database connection, external API or dependent change rule, are available.

Penetration Test

tests the infrastructure, network, code and applications to find security vulnerabilities that an attacker could exploit.

Performance Test

tests the execution time of code or models on a specific set of data, the throughput of collecting the data and the performance of visualisations of that data when accessed by users.

Regression Test

tests to confirm that a recent code, model or infrastructure change has not adversely affected existing capabilities or outputs.

Security Test

tests that verify that security rules such as authentication and authorization are being respected.

Unit Tests

the smallest automated test possible. They test the behavior of just one function or method, and is written directly in the code. These tests will usually mock out dependencies such as databases or services, so that only the function or method is being tested.

Visualisation Test

test that the visualisation functions as expected when viewed and interacted with by the user .

Visualisation Content Test

test that the data displayed in the visualisation is correct.

Data Testing Techniques

Applying a perspective of how we test the data meets its expected context and is fit for purpose we have:

Alert

something is not quite right with the data and it notifies somebody who should care.

Reconcile

the data we is correct and we have not lost any data.

Validate

the rules we have applied to the data has not resulted in the data being invalid.

Reconciliation vs Validation

Data reconciliation and data validation are two different patterns we can use, both of which prove the data is fit for purpose.

Data reconciliation

Data reconciliation confirms that two sets of data are exactly the same. It is typically used to ensure we do not lose data when collecting it from the system of record and the landing it in the history zone. To reconcile the data we need to be sure we are not changing the data in this process, so that what is on the left should always equal what is on the right. For example, when we count the number of records on the left and get a result of 100 we should be able to count 100 recrods on the right.

Data validation

When we change the data, for example applying a business rule to cleanse the data, or filter the data, we can no longer reconcile the data, as what is on the left will no longer match what is on the right as we have by the very nature of the transformation process changed the data. In this scenarion we still need to ensure the data context is correct and the data is fit for purpose, so we apply testing patterns that validate the data is correct. For example, we may filter transactions to only populate data with transactions that are in a status of “approved”. When we count the number of transactions on the left we will get a result of 100, when we count the records on the right we will get a result of 80, so we cannot use a reconciliation pattern in this scenario.

Test Pattern Concepts

There are a number of pattern concepts that are helpful to understand.

Assertion Based Testing

(TBD)

Left vs Right

We create a test that queries one set of data (the left). We create a second test that queries a second set of data (the right).

If the result of the first query equals the second query (the left = the right) then the test passes.

For example we count the number of records in a source table (left) and get a response of 100. We count the number of records in the history table (right) and get a response of 100. As left = right (100 = 100) the test passes.

From a systems point of view we can then create an automated testing capability that receives a left test and a right test and visualises if it passes or fails.

This leaves the main complexity in the creation and automation of valid tests where it should be.

We can automate some of those tests by creating a number of reusable testing patterns.

Data Service Level (DSL)

Data does not have the same service level throughout the data platform.

For example the DSL for data collected from a database maybe daily, as we are only allowed to query that data at 2am each day. Or the DSL for a csv file may be load it into the history zone as soon as it is dropped into the filedrop area.

For changes rules, a Concept may be populated with a DSL that is based on the data arriving in the History zone. However it may also be dependent on multiple other change rules, example calculating profit is dependent on the expense data and the revenue data both being at the same time window. Expenses may come from a source system with a DSL of daily and Revenue might come from a source system with a DSL of hourly, in this example the profit change rule would need to have a DSL of daily to be accurate.

The DSL of the data must be taken into account when executing any tests.

Data Test Patterns

There are a number of test patterns that can be adopted when developing an automated test capability. These patterns can be used to ensure we deliver a pirce of the testing elephant one small bite at a time.

Reconcile Source Data Collection Patterns

Questions that drive pattern

  • Can we dynamically access the source system?

Pattern 1: Brute Force

Compare a full table in the source (left) with a full table in history (right).

Requisites:

  • Access to query the source table

  • or access to a snapshot copy of the source table in the data platform.

Pro’s:

  • Full reconciliation of a table

Con’s:

  • Expensive query on both the source system and the data platform.

Automation Patterns:

  • Parameter to set a time window for records to reconcile;

  • Snapshot source table and copy to temporary table in data platform, to reduce latency on the compare execution.

Pattern 2: Row Hash

Create a hash value from the concatenated field values for a row and store the hash in a new column in the target table. Execute this logic each time a new row is collected from the source system.

Dynamically query the source system and generate the row hash.

Compare the dynamically generated row has (left) with the stored History hash (right)

Pro’s:

  • Lighter query on data platform;

Con’s:

  • Expensive query on source system;

Automation Patterns:

  • Parameter to set a time window for records to reconcile;

  • Snapshot source table and copy to temporary table in data platform, to reduce latency on the compare execution.

Pattern 3: Column Sum

(TBD)

Pattern 4: CheckSum

(TBD)

Pattern 5: Row Count

(TBD)

Unit Tests Patterns

(TBD)

Technical Rule Validation Test Patterns

Technical Rule Validation Tests are test that we can define and run automatically regardless of the business rule that is being applied.

Unique Keys

Some tables, such as Concepts in the event zone should always contain unique records. This validation test can be automated rather than relying on the validation test being manually created each time.

Some tables, such as the Customer table in the trusted area should always contain unique records. This validation test can be automated rather than relying on the validation test being manually created each time.

Business Rule Validation test

(TBD) count of unique values for classification fileds wholesale vs retail

Other Patterns

When it breaks, fix it and then create an alert test

When you find an example where the data has behaved unexpectedly, then fix the issue and at the same time create a test that will alert you if the problem either arises again.

Automate the test results

A dashboard should be available that displays the test that have failed. You should also be able to drill down to see test that have passed.

Couple Test to Deploy

Once the tests have successfully passed, the CI/CD process should automatically execute.