Data Engineer to AI Data Engineer Learning Path
Ask why a retrieval-augmented application gives bad answers and the conversation usually turns to the model. It is almost never the model. It is a 4,000-character chunk that split a table in half, an embedding index that has not been refreshed since March, and a document nobody was allowed to read in the first place.
Those are data engineering problems. They are your problems — which is why the shortest route to a working AI system in most organisations runs through the data team, not the ML team.
This path has seven stages, roughly nineteen weeks at an hour a day. It assumes you already run pipelines in production.
Stage 0: what already transfers
| You already do this | It is called this in AI work |
|---|---|
| Modelling a domain into tables | Chunking and context assembly |
| Incremental loads and CDC | Index refresh and re-embedding |
| Freshness and null checks | Retrieval quality and groundedness |
| Column-level access control | Chunk-level access control |
| Lineage and audit | Prompt and answer provenance |
| Warehouse cost tuning | Embedding and token economics |
| A dashboard nobody trusts | An assistant nobody trusts |
That last row is the one to take seriously. The failure mode is identical, and so is the fix: measure the thing, publish the measurement, and give it an owner.
The roadmap
Stage 1: embeddings and vector storage — weeks 1–3
Learn: what an embedding actually is, how similarity search works, chunking strategies and why they decide answer quality more than the model does, the index types (HNSW, IVF) and what each trades away, and how recall degrades as a collection grows.
Build: load one existing table plus one document set into pgvector and measure recall at 5 and 10 against twenty questions you wrote by hand.
Done when: you can explain why your chunk size is what it is, with a number attached.
Start with pgvector in the Postgres you already run rather than a new dedicated store. Most teams never need more, and you will learn more from tuning one index than from evaluating six products. When you do need to compare, the field splits into stores you probably already operate — pgvector, Redis, Elasticsearch — and dedicated engines worth knowing by name: Weaviate, Milvus, Pinecone. The knowledge base has a semantic data layer technology radar for the comparison.
Chunking has its own vocabulary and it is worth using precisely: semantic chunking splits on meaning rather than character count; contextual chunking prepends document and section context so a chunk stands alone; contextual summary stores a generated summary beside the raw text; context compaction folds earlier material down as the window fills; and context rot is the measurable quality decay when a long window fills with marginally relevant text. Those five decide answer quality far more often than the embedding model does.
Stage 2: unstructured data pipelines — weeks 4–6
Learn: PDF and layout-aware parsing, OCR for scans, table extraction, HTML and wiki normalisation, transcript handling, near-duplicate detection, and PII detection and redaction before text reaches an embedding model.
Build: an incremental pipeline that watches a document source, parses, chunks, redacts, embeds and upserts — and that reprocesses only what changed.
Done when: re-running it end to end is cheap enough that you are willing to change the chunking strategy.
This is the stage that most resembles the job you already have, and the one most teams skip. Dataset engineering in the knowledge base covers the ground properly.
Stage 3: retrieval as a data product — weeks 7–9
Learn: hybrid search (dense plus keyword), re-ranking, metadata filters, freshness SLAs, and chunk-level access control so retrieval cannot leak what the user could not have opened.
Build: a retrieval API with its own contract, its own quality dashboard and its own owner. Applications call it; they do not each grow their own copy.
Done when: two different applications use it and neither has its own chunking code.
Treat this as a data product, because it is one: a contract, an SLA, a quality dashboard, an owner. The RAG section and the RAG reference architecture are the reference here; search as code is worth reading for how to version the retrieval config itself.
Stage 4: semantic layer and graphs — weeks 10–13
Learn: ontologies and entity resolution, the metrics layer, text-to-SQL and why it fails without one, knowledge graphs and Graph RAG for relationship questions — the case vector search handles worst — and exposing all of it to agents over MCP rather than through bespoke connectors. Neo4j is the usual first graph store, and the platforms are closing in: Amazon Neptune, graph queries inside BigQuery, and Fabric IQ on the Microsoft side all put the graph next to the warehouse instead of in a separate system to synchronise. Palantir Foundry sells the same idea ontology-first. The technique matters far more than the product — if you cannot name your entities and the relationships between them, no store will rescue the answer.
Build: a semantic layer over one business domain — defined metrics, resolved entities — and a text-to-SQL endpoint that answers only in terms of it.
Done when: “what was net revenue last quarter” returns one number, and it is the number finance uses.
This is where the data team’s real leverage is. A model that can query a governed semantic layer beats a bigger model guessing at raw table names, every time. Databricks Genie and Microsoft Fabric’s 2026 database work are both worth a look at how the platforms are packaging this.
Stage 5: evaluating the data plane — weeks 14–16
Learn: retrieval metrics separated from answer metrics — recall, precision, mean reciprocal rank — plus groundedness, embedding drift as source content changes, and how to build a golden question set that survives contact with real users.
Build: a nightly job that runs 100 golden questions through retrieval only and alerts when recall drops.
Done when: you find out that a source system changed before the business does.
Evaluation frameworks and AI as a judge cover the techniques; the discipline is the same one you already apply to data quality.
Stage 6: governance and cost — weeks 17–19
Learn: lineage that reaches from an answer back to a source document, audit trails for what was retrieved and shown to whom, PII policy at chunk granularity, retention and the right to be forgotten in a vector index, and the economics of re-embedding a large corpus.
Build: an answer-level provenance trail — question, chunks retrieved, documents they came from, who was allowed to see them.
Done when: you can answer “why did it say that, and was this person allowed to see it” from logs alone.
AI governance and cost management in the knowledge base go deeper than this path can.
The two rows an assessment will actually check
Alongside the AI-specific stages, enterprise assessments check two things about your platform. Neither is new work — but the expected surface area is wider than most data engineers assume.
One modern data platform, properly. Snowflake or Databricks — you are expected to know one of them well, not both:
| Snowflake | Databricks |
|---|---|
| Snowpipe, virtual warehouses, Snowsight | Lakehouse and medallion architecture |
| Data sharing, data lineage | Delta Lake, Unity Catalog, data lineage |
| Snowpark, Cortex, Snowflake ML | PySpark, Databricks Workflows, MLflow |
| AgentBricks, workspace and cluster management, adaptive query execution |
Cortex and AgentBricks are the rows that matter most here — they are where each platform puts the AI work, which means the retrieval and semantic-layer stages above may already have a managed answer on the platform you run. Vedcraft’s Snowflake Cortex walkthrough is the worked version of that, and the knowledge base covers Databricks Genie.
The open-source stack around it. Know roughly one per row and what each row is for:
| Concern | Know one of |
|---|---|
| Integration and workflow | Airbyte, Airflow, Spark, Flink, Pandas |
| Query engines | Trino, Dremio, Pinot |
| Table formats | Delta Lake, Iceberg, Hudi |
| Transformation and orchestration | dbt, Dagster |
| Serialisation | Parquet, Avro |
| Events and streaming | Kafka, Confluent, Flink |
| Visualisation | Superset |
The table-format row is the one worth extra attention. Iceberg in particular is becoming the interchange layer between warehouses, lakehouses and the AI tooling that reads them — which makes it the row most likely to change how the rest of your stack fits together.
The platform does not get replaced
Every row on the right is the row on its left applied to text nobody had modelled before. That is the whole shift. If your warehouse, your orchestration and your governance are in decent shape, you are most of the way there — and if they are not, an AI project will surface that faster than anything else you could have run.
Vedcraft’s Snowflake Cortex walkthrough is a good example of the platform-native version of this, and the Google Data Cloud Summit takeaways trace where the warehouse vendors started heading.
What to have built by the end
| # | Project | Proves you can |
|---|---|---|
| 1 | Incremental ingestion for one document source | Handle unstructured data with the rigour you give tables |
| 2 | A retrieval API with a quality dashboard | Run retrieval as a product, not a feature |
| 3 | A semantic layer with defined metrics | Give agents governed access to the business |
| 4 | Provenance and access control end to end | Put it in front of an auditor |
Four traps
- Chunking by character count because it was the default. Structure-aware chunking — by heading, by row, by turn — is usually a bigger win than any model change.
- One vector store per application. You would never let four teams each build their own customer table. This is the same mistake with a newer name.
- Access control bolted on after launch. Retrieval that ignores permissions is a data breach with a chat interface. Filter at query time, not after.
- No retrieval-only evaluation. If you only measure the final answer, you cannot tell a retrieval regression from a model regression, and you will spend weeks on the wrong one.
Where to go next
- Agentic AI Knowledge Base — the maintained reference behind most of the links above
- Building gen AI apps with Snowflake Cortex
- Java/Python Developer to AI Engineer — the application side of the same system
- DevOps Engineer to LLMOps/AIOps Platform Engineer — the platform underneath it
- Developer to Architect — the breadth all three sit inside