The ODI Story

Data infrastructure for agents you trust.

"MDS was optimized for humans. ODI is designed for a future with humans and production agents at scale."

The Managed Data Stack gave teams governed, queryable data that humans could trust. It solved the right problem for its moment. But as production AI agents enter the loop — reading gold tables, writing back derived insights, acting on the results — a new set of requirements emerges. Agents need the same governance guarantees humans relied on, plus provenance, freshness signals, and the ability to ask questions in language rather than SQL.

Open Data Infrastructure is Fivetran's answer to that shift. Open table formats instead of proprietary warehouses. Semantic layers that serve both BI tools and language models. Connectors that agents can call the same way analysts do. The architecture stays the same; the consumers multiply.

How this demo is built

One pipeline. One semantic layer. Two consumers.

Liner Notes is a record collection treated as a data product. The same gold tables power a static UI and dbt-wizard run-time agents. Neither consumer owns the data — they both read from the governed layer.

Step 01

Ingest

Fivetran

Spotify Web API and MusicBrainz connectors pull listening history, liked tracks, top artists, and album metadata on a scheduled cadence into Snowflake-managed Iceberg tables.

Step 02

Store

Snowflake + Iceberg

Raw events land in open table format under MDLS. Schema evolution, time-travel, and zero-copy cloning come for free. No proprietary lock-in on the storage layer.

Step 03

Transform

dbt

A governed semantic layer — staging, intermediate, and gold marts — runs on top of the Iceberg tables. Every number on the Analytics page and every answer on the Ask page is served from a dbt-modeled gold mart.

Step 04

Serve

This app + dbt-wizard agents

A static Next.js site reads gold marts directly at build time. dbt-wizard run-time agents translate natural-language questions into SQL on demand, so the Ask page never hardcodes a query.

The open-format difference

Iceberg tables any engine can read.

The Spotify and MusicBrainz data lands in Snowflake-managed Apache Iceberg tables under MDLS. Because Iceberg is an open format, Snowpark, Spark, Trino, or any future compute engine can query the same bytes without a migration. The data outlives the warehouse vendor.

dbt sits on top and builds staging → intermediate → gold marts. The gold layer is what this app and the run-time agents both read. Changing a mart definition propagates to every consumer at the next run — no manual API updates.

The agent layer

Humans and agents read the same gold layer.

The Ask page uses dbt-wizard run-time agents — four sub-agents (Explorer, Summary, Worker, Verification) that translate a plain English question into a SQL query, run it against the gold mart, and return a sentence plus a number. The agents never see raw Spotify event data; they only see the governed semantic layer dbt built.

That's the ODI promise: agents with the same governance guarantees humans relied on. Same freshness signals. Same access controls. Same lineage. The trust transfers.

What each page demonstrates

Every view is a different lens on the same pipeline.

Read the full ODI Story →View architecture →