Table Of Contents
Table Of Contents

2024-01-16 - Spanner workload - ensemble config checking

Release

Status: Available

Type: DataOps

Date: 2024-01-16

Problem

Our heaviest spanner workload is when we check/create/update ensemble config.

This is because we use a realtime viewe of all the BQ objects to detect changes that require a consume rebuild.

This query was being run everytime an event load completed, before we started a consume load.

Solution

Check for ensemble config and only update if it doesn’t exist or the table that has just loaded isn’t part of an existing ensemble, this should drastically reduce the execution of our heaviest config query

Leverage the Magic

We are always looking for ways to maintain a steady workload (without heavy spkies) against our spanner instance to maintain the responsiveness and performance of the App

Last Refreshed

Doc Refreshed: 2024-05-20