When the alert arrives
When a table misses its update, the first job for the on-call engineer is to find out why. Databricks opened the Genie ZeroOps beta on November 19. The tool can flag a late update or an unexpected drop in row count in a shared incident inbox. It examines run logs and traces where the data came from to identify a likely cause. That gives the engineer somewhere to start.[1]
A drop in row count can mean different things depending on who uses the table and for what. Reprocessing late data might get a report out on time. An engineer who knows that the rerun could count records twice in a downstream table has a reason to check that table first. Databricks says more than half of the over 80,000 issues it examined in its limited trial required knowledge specific to the organization to classify, prioritize and solve correctly. That makes it worth asking what the engineer needs to know before signing off on a proposed change.[1]
Who reviews the fix?
For a job backed by Git, the proposal can go up for review on a branch separate from production code. Databricks says a person must approve changes to production code or data. Reviewing the suggestion may require the on-call engineer to speak to a team that relies on the table. In that case, searching logs might take less time while examining the change and waiting for a colleague account for more of the shift.[1]
Databricks says teams can tell the agent which sources regularly fail and what remedies they prefer. If the instructions cover how to handle late data from a particular source, the next person on call may be able to settle a recurring question without tracking down a colleague. Someone still needs time to update those instructions when the pipeline changes. The team that sets the on-call rota can treat that upkeep and the authority to approve a production change as separate responsibilities.[1]
Measuring on-call work
The shared inbox gives two people investigating the same fault a common view. A team can judge the benefit by comparing similar incidents on two measures: time to the first useful finding and time until the table is usable again. It can also track who spent time on review and how often someone outside the rota had to be contacted. If a finding arrives sooner but recovery takes longer, the team can check how much time it spent finding the table owner or getting approval. Fewer overnight calls would be a gain for the people on call in its own right.[1]