Databricks Data Intelligence Reference Architecture

The Databricks Data Intelligence Platform reference architecture is organized along seven swim lanes: Source, Ingest, Transform, Query/Process, Serve, Analysis, and Storage. Reading a Databricks architecture diagram left to right along these lanes makes it easier to place any given capability — ingestion connector, governance surface, serving endpoint — in the flow of data from source system to business application.

Source

There are three ways to integrate external data into the Data Intelligence Platform:

ETL — the platform integrates with systems that provide semi-structured and unstructured data (sensors, IoT devices, media, files, logs) as well as structured data from relational databases or business applications.
Lakehouse Federation — SQL sources such as relational databases can be integrated into Databricks and Unity Catalog without ETL. The source system data stays governed by Unity Catalog, and queries are pushed down to the source system.
Catalog Federation — external Hive Metastore catalogs or AWS Glue can be integrated into Unity Catalog through catalog federation, letting Unity Catalog control tables stored in Hive Metastore or AWS Glue.

Ingest

Data lands in Databricks via batch or streaming:

Databricks Lakeflow Connect offers built-in connectors for ingestion from enterprise applications and databases. The resulting ingestion pipeline is governed by Unity Catalog and powered by serverless compute and Pipelines.
Files delivered to cloud storage can be loaded directly using Databricks Auto Loader.
For batch ingestion from enterprise applications into Delta Lake, the platform relies on partner ingest tools with adapters built for those systems of record.
Streaming events — from Kafka, sensors, IoT, or change data capture — can be ingested directly using Databricks Structured Streaming.

Storage

Data is typically stored in the cloud storage system, where ETL pipelines use the medallion architecture to curate data as Delta Lake tables or Apache Iceberg tables.

Transform and Query/Process

The platform uses Apache Spark and Photon for all transformations and queries.

Pipelines is a declarative framework for building reliable, maintainable, and testable data processing pipelines. Powered by Spark and Photon, the Data Intelligence Platform supports both SQL queries via SQL warehouses and SQL, Python, and Scala workloads via workspace clusters.

For data science — ML modeling and AI — the Databricks AI and Machine Learning platform provides specialized ML runtimes for AutoML and for coding ML jobs. MLflow underpins all data science and MLOps workflows.

Serve

For data warehousing (DWH) and BI use cases, the platform provides Databricks SQL, powered by SQL warehouses and serverless SQL warehouses.

For machine learning, Model Serving is a scalable, real-time, enterprise-grade model serving capability hosted in the Databricks control plane. Unity AI Gateway governs and monitors access to supported AI models and their model serving endpoints.

Operational databases:

Lakebase is an online transaction processing (OLTP) database based on Postgres, fully integrated with the Data Intelligence Platform. It lets you create OLTP databases on Databricks and integrate OLTP workloads with the platform.
External systems, such as operational databases, can store and deliver final data products to user applications.

Collaboration:

Business partners get secure access to the data they need through OpenSharing. Built on OpenSharing, the Databricks Marketplace is an open forum for exchanging data products. Clean Rooms are secure, privacy-protecting environments where multiple parties can work together on sensitive enterprise data without direct access to each other’s data.

Analysis

The final business applications sit in this swim lane — custom clients such as AI applications connected to Model Serving for real-time inference, or applications that access data pushed from Databricks to an operational database.

For BI use cases, analysts typically use BI tools to access the data warehouse. SQL developers can additionally use the Databricks SQL Editor for queries and dashboarding. The platform also offers dashboards to build data visualizations and share insights.

Integrate

The platform integrates with standard identity providers for user management and single sign-on (SSO).
External AI services like OpenAI, LangGraph, or Hugging Face can be used directly from within the Data Intelligence Platform.
External orchestrators can use the comprehensive REST API or dedicated connectors to tools like Apache Airflow.
Unity Catalog is used for all data and AI governance and can integrate other databases into its governance through Lakehouse Federation. It can also be integrated into other enterprise catalogs — contact the enterprise catalog vendor for details.

Common capabilities across all workloads

Beyond the swim lanes, the platform ships management capabilities that support every workload.

Data and AI governance

Unity Catalog is the central data and AI governance system, providing a single place to manage data access policies across all workspaces. It covers every asset created or used in Databricks — tables, volumes, features (feature store), and models (model registry) — and can capture runtime data lineage across queries run on Databricks.

Databricks Data Quality Monitoring lets you monitor data quality across all tables in your account, detecting anomalies and producing a full data profile for each table.

For observability, system tables provide a Databricks-hosted analytical store of your account’s operational data, useful for historical observability across the account.

Data intelligence engine

The Data Intelligence Platform combines AI with the unification benefits of Databricks so your organization can understand the unique semantics of its data — see Databricks AI assistive features. Genie Code is available in notebooks, the SQL editor, the file editor, and elsewhere as a context-aware AI assistant for users.

Automation and orchestration

Lakeflow Jobs orchestrate data processing, machine learning, and analytics pipelines across the platform. Lakeflow pipelines let you build reliable, maintainable ETL pipelines with declarative syntax. The platform also supports CI/CD and MLOps.

Why the swim-lane view matters

Mapping capabilities to Source, Ingest, Transform, Query/Process, Serve, Analysis, and Storage gives architects a consistent way to reason about a Databricks deployment: every new Databricks feature announcement slots into one of these lanes, and every governance or automation capability sits underneath all of them via Unity Catalog and Lakeflow Jobs. For teams comparing this to a broader data engineering or cloud data platform strategy, or evaluating generative AI workloads that need to sit on top of governed data, this swim-lane structure is the fastest way to see where a new capability belongs.

Source: Databricks — AWS Lakehouse Architecture Reference

Disclaimer: Diagram and content adapted from Databricks’ official documentation. All product names, logos, and trademarks are property of their respective owners (Databricks Inc., Amazon Web Services, Inc.). This reproduction is for educational/reference purposes — refer to the source link above for the authoritative, up-to-date version.