data monitoring

With businesses struggling with data quality issues, data observability platforms offer a way to ensure reliability through targeted observability, connection to analytics dashboards and the ability to map dependencies across modern and legacy data sources.

The release of Dependency Driven Monitoring from Bigeye aims to address these concerns, through optimized monitoring tailored to the needs of enterprise users.

By automatically identifying and monitoring modern and legacy data sources, the platform can provide targeted monitoring on the columns that are actively utilized.

The technology is powered by Lineage Plus, a data lineage technology providing Dependency Driven Monitoring with 50 connectors covering modern and legacy enterprise data sources.

Potential benefits for data engineering teams include reduced cost of ownership, decreased alert noise via AI-driven anomaly detection, and faster issue resolution facilitated by lineage-powered root cause and impact analysis.

Bigeye ensures prompt notification to affected data source owners through communication platforms like Slack or Microsoft Teams upon detection of a data issue and facilitates incident management by automatically generating bi-directional tickets in ITSM tools such as JIRA and ServiceNow.

For data consumers, Bigeye integrates data health updates into their analytics dashboard, offering real-time insights into the reliability of their analytics.

Utilizing Bigeye’s lineage-powered root cause and impact analysis, data engineering teams can trace data issues to their source.

Supporting both cloud and on-premises infrastructure, it captures ETL job information and offers connectors a range of data sources, including Tableau, Microsoft Power BI and Snowflake.

Bigeye plans to expand its connector offerings to include Informatica PowerCenter, IBM Netezza, and other platforms throughout 2024, providing additional integration capabilities.

Kyle Kirwan, CEO and co-founder of Bigeye, said data observability is rapidly growing in demand, but often faces obstacles in the deployment of necessary monitoring.

Common challenges include over-monitoring, which drives up wasteful compute costs and distracting alert noise, under-monitoring (missing key fields that eventually flow into the analytics), and the heavy lift needed in deciding what to monitor.

“DDM addresses this by automatically identifying and deploying monitoring on only the fields that actually power the analytics,” he said.

He explained that by connecting into existing ITSM and chat tools, Bigeye can seamlessly integrate into a company’s incident management process.

“This reduces the need to jump back and forth between tools and allows for efficient communication between distributed teams,” he said.

Data source owners get a clear picture of exactly which issues are impacting their data source and where they may have originated, and data consumers get real-time insights into the status of any data outage.

“Data engineering teams can perform faster triage and resolution actions with their existing tools while leveraging helpful root cause analysis insights from Bigeye,” Kirwan added.

Scott Wheeler, cloud practice lead at Asperitas, said best practices for implementing a data observability strategy within an organization include establishing clear goals and objectives and automating where possible.

“Define what you aim to achieve with data observability, whether it’s improving data quality, ensuring data reliability, meeting compliance requirements, or enhancing operational efficiency,” he said. “Clear goals will guide the selection of tools and the design of processes.”

Automating the monitoring and alerting processes will help data professionals identify and respond to data issues quickly and can be used for routine tasks like data validation and anomaly detection, freeing resources for more complex analysis.

Data observability should also be integrated with broader data governance frameworks, which requires defining policies and procedures for data access, quality, security and compliance, ensuring that observability practices adhere to these guidelines.

Wheeler said he encourages collaboration between IT, data engineering, data science and business units, noting a cross-functional approach ensures that data observability is aligned with business objectives and leverages diverse expertise.

“Ensure your team has the necessary skills and knowledge to implement and manage the data observability strategy,” he said. “Regular training and access to resources are crucial for empowering them to use observability tools and techniques effectively.”

Techstrong TV

Click full-screen to enable volume control
Watch latest episodes and shows

Mobility Field Day