Subscribe
Reference Architecture

Databricks Data Intelligence Platform Reference Architecture

A reference architecture for the Databricks Data Intelligence Platform, organized into Source, Ingest, Storage, Transform, Serve and Analysis swim lanes.

  • Data Engineering
  • Lakehouse
  • Generative AI
  • MLOps
  • Cloud
  • Cross-industry

The challenge

Placing any Databricks capability — a new connector, a governance surface, a serving endpoint — in the flow of data from source system to business application, without a map of the platform to point at.

Layered diagram of the Databricks Data Intelligence Platform, from Source and Ingest through Storage and Transform to Serve and Analysis, with Unity Catalog governance and Lakeflow orchestration as cross-cutting rails
Databricks Data Intelligence Platform Reference Architecture — reference architecture

The Databricks Data Intelligence Platform organizes every capability into seven swim lanes — Source, Ingest, Storage, Transform, Query/Process, Serve, and Analysis — with Unity Catalog governance and Lakeflow orchestration running underneath all of them. Reading left to right along the lanes is the fastest way to place a new connector, serving endpoint, or governance surface in the flow of data from source system to business application.

How it works

Data enters through Source and Ingest — ETL, Lakehouse Federation, Lakeflow Connect, Auto Loader, or Structured Streaming — and lands in Storage as Delta Lake or Iceberg tables curated through the medallion architecture. Apache Spark and Photon do every transformation and query from there, whether that’s a declarative Pipelines job, a SQL warehouse, or an MLflow-tracked training run.

Serve is where the platform hands data and models to the rest of the business: Databricks SQL for BI, Model Serving and Unity AI Gateway for ML, Lakebase for OLTP, and OpenSharing/Marketplace/Clean Rooms for sharing data products with partners without granting direct access. Analysis is the consuming layer — AI applications, BI tools, and dashboards.

Unity Catalog and Lakeflow Jobs are not lanes; they are the rails every lane runs on. Governance, lineage, data quality monitoring, and orchestration apply uniformly across Source through Analysis rather than being bolted onto each lane separately.

What to get right

  • Federate before you replicate. Lakehouse Federation and Catalog Federation bring SQL sources and external catalogs under Unity Catalog governance without an ETL copy — reach for them before building a pipeline.
  • Version the medallion. Bronze, Silver and Gold are contracts between teams as much as storage tiers; changing a Silver schema is a breaking change to everything downstream.
  • Put entitlements in Unity Catalog, not in the app. Access policies enforced once at the catalog apply consistently across SQL, ML, and sharing — enforcing them per-application does not.
  • Route model traffic through Unity AI Gateway. It is what makes governance, monitoring, and cost attribution possible across every model serving endpoint instead of per-team.

For the full swim-lane breakdown, see Databricks Data Intelligence Reference Architecture.

How it fits together

  1. Source

    External data enters via ETL, Lakehouse Federation (SQL sources queried in place, still governed by Unity Catalog), or Catalog Federation (external Hive Metastore or AWS Glue catalogs brought under Unity Catalog).

  2. Ingest

    Lakeflow Connect provides built-in, Unity Catalog-governed connectors for enterprise apps and databases. Auto Loader picks up files from cloud storage, partner ingest tools handle batch loads from systems of record, and Structured Streaming ingests Kafka, IoT and CDC events.

  3. Storage

    Cloud storage curated through the medallion architecture — Bronze, Silver, Gold — as Delta Lake or Apache Iceberg tables.

  4. Transform & Query/Process

    Apache Spark and Photon power every transformation and query. Pipelines is the declarative framework for building and testing data pipelines; SQL warehouses and workspace clusters run SQL, Python and Scala workloads; MLflow underpins data science and MLOps.

  5. Serve

    Databricks SQL serves DWH and BI workloads; Model Serving and Unity AI Gateway serve ML models and govern access to them; Lakebase provides an OLTP layer on Postgres; OpenSharing, Marketplace and Clean Rooms serve data products to partners.

  6. Analysis

    The business-application layer — AI apps calling Model Serving for real-time inference, BI tools and the Databricks SQL Editor against the warehouse, and dashboards for shared visualizations.

  7. Data and AI governance

    Unity Catalog is the single governance system for every asset — tables, volumes, features, models — with runtime lineage, access policies, and Data Quality Monitoring across all workspaces.

  8. Automation and orchestration

    Lakeflow Jobs orchestrate data, ML and analytics pipelines end to end; Lakeflow pipelines give declarative ETL; the platform supports CI/CD and MLOps throughout.

Layered diagram of the Databricks Data Intelligence Platform, from Source and Ingest through Storage and Transform to Serve and Analysis, with Unity Catalog governance and Lakeflow orchestration as cross-cutting rails — detailed architecture
Databricks Data Intelligence Platform Reference Architecture — detailed architecture

Typically built with

  • Unity Catalog
  • Delta Lake
  • Apache Spark
  • Databricks SQL
  • MLflow
  • Lakeflow

Go deeper

More reference architectures

Layered diagram of an enterprise graph architecture for agentic search, from ingestion and entry points through retrieval, the enterprise graph context layer, and agentic reasoning to the model and tool ecosystem, beside rails for governance and for trust, security and observabilityEnterprise Context Graph Reference Architecture for Agentic Enterprise SearchLayered diagram of a real-time agentic AI architecture, from triggers and gateways through an event mesh and orchestrator to agents and the tools, data and LLM services they call, beside rails for deployment footprint and for trust, governance and observabilityReal-Time Agentic AI Reference Architecture using an Event MeshLayered diagram of the Salesforce Data 360 agentic enterprise reference architecture, from sources and ingestion through the harmonized C360 data model to segmentation, query and agentic activation, beside a foundational elements rail for governance, security, sandboxes and Data Cloud OneSalesforce Data 360 Agentic Enterprise Reference Architecture