2024-01-16 - Spanner workload - ensemble config checking¶
Problem¶
Our heaviest spanner workload is when we check/create/update ensemble config.
This is because we use a realtime viewe of all the BQ objects to detect changes that require a consume rebuild.
This query was being run everytime an event load completed, before we started a consume load.
Solution¶
Check for ensemble config and only update if it doesn’t exist or the table that has just loaded isn’t part of an existing ensemble, this should drastically reduce the execution of our heaviest config query
Leverage the Magic¶
We are always looking for ways to maintain a steady workload (without heavy spkies) against our spanner instance to maintain the responsiveness and performance of the gui
Last Refreshed¶
Doc Refreshed: 2024-03-02