Eigen RadarAI
Analysis

Databricks opens its data pipeline troubleshooting tool to public testing

Databricks says it has opened Genie ZeroOps to public testing after a limited trial. The tool can begin an investigation when a data job fails or when a table receives fewer rows than expected. It puts incidents and suggested fixes in a shared inbox for engineers. The company says a person still has to approve changes to production code or data.

Artificial Intelligence··Morning
High-angle overhead view of a glass-and-brass row-sorting frame where a brass magnifying arm traces a sparse, halted middle row back to a jammed source hopper beside a shared wooden inbox and test branch.

An alert can open an investigation

Databricks, a US data platform company, says it has opened Genie ZeroOps to beta testers through their workspace administrators. The software watches for conspicuous failures, such as a job that stops with an error, as well as delayed table updates and unexpectedly low row counts. It can start an investigation as soon as it detects a problem and use logs and information on where the data came from to trace the failure.[1]

Engineers can share an incident inbox

The company introduced Genie ZeroOps in June and gave hundreds of customers access before the public beta. The beta adds an inbox that teams can share to see incidents and suggested remedies. Colleagues can tell the software which sources often fail, how they name jobs and which failures they already expect, so that routine interruptions can be filtered out of notifications. An alert can carry a short description of the problem alongside the suggested change.[1]

Production changes still require human approval

For jobs whose code is tracked in Git, engineers can review proposed changes on a separate branch. The software can test a fix away from the live system. If it proposes changes to production code or data, Databricks says those changes still require a person’s approval. Teams can receive incident notifications through Slack, Teams or PagerDuty.[1]

References

  1. News sourceDatabricksDatabricks opens Genie ZeroOps pipeline troubleshooting to public beta↩1↩2↩3