# Databricks Data Intelligence Reference Architecture The Databricks Data Intelligence Platform reference architecture is organized along seven swim lanes: **Source**, **Ingest**, **Transform**, **Query/Process**, **Serve**, **Analysis**, and **Storage**. Reading a Databricks architecture diagram left to right along these lanes makes it easier to place any given capability — ingestion connector, governance surface, serving endpoint — in the flow of data from source system to business application. ## Source There are three ways to integrate external data into the Data Intelligence Platform: - **ETL** — the platform integrates with systems that provide semi-structured and unstructured data (sensors, IoT devices, media, files, logs) as well as structured data from relational databases or business applications. - **Lakehouse Federation** — SQL sources such as relational databases can be integrated into Databricks and Unity Catalog without ETL. The source system data stays governed by Unity Catalog, and queries are pushed down to the source system. - **Catalog Federation** — external Hive Metastore catalogs or AWS Glue can be integrated into Unity Catalog through catalog federation, letting Unity Catalog control tables stored in Hive Metastore or AWS Glue. ## Ingest Data lands in Databricks via batch or streaming: - **Databricks Lakeflow Connect** offers built-in connectors for ingestion from enterprise applications and databases. The resulting ingestion pipeline is governed by Unity Catalog and powered by serverless compute and Pipelines. - Files delivered to cloud storage can be loaded directly using **Databricks Auto Loader**. - For batch ingestion from enterprise applications into Delta Lake, the platform relies on **partner ingest tools** with adapters built for those systems of record. - Streaming events — from Kafka, sensors, IoT, or change data capture — can be ingested directly using **Databricks Structured Streaming**. ## Storage Data is typically stored in the cloud storage system, where ETL pipelines use the **medallion architecture** to curate data as Delta Lake tables or Apache Iceberg tables. ## Transform and Query/Process The platform uses **Apache Spark** and **Photon** for all transformations and queries. **Pipelines** is a declarative framework for building reliable, maintainable, and testable data processing pipelines. Powered by Spark and Photon, the Data Intelligence Platform supports both SQL queries via **SQL warehouses** and SQL, Python, and Scala workloads via **workspace clusters**. For data science — ML modeling and AI — the Databricks AI and Machine Learning platform provides specialized ML runtimes for AutoML and for coding ML jobs. **MLflow** underpins all data science and MLOps workflows. ## Serve For data warehousing (DWH) and BI use cases, the platform provides **Databricks SQL**, powered by SQL warehouses and serverless SQL warehouses. For machine learning, **Model Serving** is a scalable, real-time, enterprise-grade model serving capability hosted in the Databricks control plane. **Unity AI Gateway** governs and monitors access to supported AI models and their model serving endpoints. **Operational databases:** - **Lakebase** is an online transaction processing (OLTP) database based on Postgres, fully integrated with the Data Intelligence Platform. It lets you create OLTP databases on Databricks and integrate OLTP workloads with the platform. - External systems, such as operational databases, can store and deliver final data products to user applications. **Collaboration:** Business partners get secure access to the data they need through **OpenSharing**. Built on OpenSharing, the **Databricks Marketplace** is an open forum for exchanging data products. **Clean Rooms** are secure, privacy-protecting environments where multiple parties can work together on sensitive enterprise data without direct access to each other's data. ## Analysis The final business applications sit in this swim lane — custom clients such as AI applications connected to Model Serving for real-time inference, or applications that access data pushed from Databricks to an operational database. For BI use cases, analysts typically use BI tools to access the data warehouse. SQL developers can additionally use the Databricks SQL Editor for queries and dashboarding. The platform also offers **dashboards** to build data visualizations and share insights. ## Integrate - The platform integrates with standard identity providers for user management and single sign-on (SSO). - External AI services like OpenAI, LangGraph, or Hugging Face can be used directly from within the Data Intelligence Platform. - External orchestrators can use the comprehensive REST API or dedicated connectors to tools like Apache Airflow. - **Unity Catalog** is used for all data and AI governance and can integrate other databases into its governance through Lakehouse Federation. It can also be integrated into other enterprise catalogs — contact the enterprise catalog vendor for details. ## Common capabilities across all workloads Beyond the swim lanes, the platform ships management capabilities that support every workload. ### Data and AI governance **Unity Catalog** is the central data and AI governance system, providing a single place to manage data access policies across all workspaces. It covers every asset created or used in Databricks — tables, volumes, features (feature store), and models (model registry) — and can capture runtime data lineage across queries run on Databricks. **Databricks Data Quality Monitoring** lets you monitor data quality across all tables in your account, detecting anomalies and producing a full data profile for each table. For observability, **system tables** provide a Databricks-hosted analytical store of your account's operational data, useful for historical observability across the account. ### Data intelligence engine The Data Intelligence Platform combines AI with the unification benefits of Databricks so your organization can understand the unique semantics of its data — see Databricks AI assistive features. **Genie Code** is available in notebooks, the SQL editor, the file editor, and elsewhere as a context-aware AI assistant for users. ### Automation and orchestration **Lakeflow Jobs** orchestrate data processing, machine learning, and analytics pipelines across the platform. Lakeflow pipelines let you build reliable, maintainable ETL pipelines with declarative syntax. The platform also supports CI/CD and MLOps. ## Why the swim-lane view matters Mapping capabilities to Source, Ingest, Transform, Query/Process, Serve, Analysis, and Storage gives architects a consistent way to reason about a Databricks deployment: every new Databricks feature announcement slots into one of these lanes, and every governance or automation capability sits underneath all of them via Unity Catalog and Lakeflow Jobs. For teams comparing this to a broader [data engineering](https://vedcraft.com/tech-trends/data-engineering/) or [cloud data platform](https://vedcraft.com/tag/cloud-data-platform/) strategy, or evaluating [generative AI](https://vedcraft.com/tech-trends/gen-ai/) workloads that need to sit on top of governed data, this swim-lane structure is the fastest way to see where a new capability belongs. --- **Source:** Databricks — [AWS Lakehouse Architecture Reference](https://docs.databricks.com/aws/en/lakehouse-architecture/reference) **Disclaimer:** Diagram and content adapted from Databricks' official documentation. All product names, logos, and trademarks are property of their respective owners (Databricks Inc., Amazon Web Services, Inc.). This reproduction is for educational/reference purposes — refer to the source link above for the authoritative, up-to-date version.