Cross-Platform Data Lineage with OpenLineage, A Foundational Layer for Data Reliability

As Data & AI systems become foundational to modern software, the pipelines that power them deserve the same engineering rigor as production services. Yet many software engineers still treat data pipelines as someone else’s responsibility. We’ll discuss why end-to-end lineage is essential for observability, debugging, and trust in Data & AI workflows, how it can allow software engineers to maintain the health of data pipelines and how this aligns with principles familiar to the SRE community. The talk will introduce the OpenLineage standard (a project under LF AI & Data), explain how it compares and complements OpenTelemetry, and present current integrations with Airflow, Spark, dbt, Flink, and more. Presented by a Technical Steering Committee (TSC) member of OpenLineage, this session is both a practical introduction and a call to action: to treat Data & AI pipelines as first-class citizens in reliable, scalable systems.