The Agentic Data Development Lifecycle in Practice

In Coalesce 2.0: The Agentic Data Engineering Platform, we called out that we believe the future DataOps lifecycle has four stages: Define, Build, Govern, and Observe. In this post, we get to the fun part and build an end-to-end pipeline on the NYC taxi dataset with Coalesce, Snowflake, and Claude Code.

Over the last year, we've rebuilt Coalesce for agentic workflows. The Coalesce CLI (coa) lets your agent author, validate, and run pipelines locally. The Coalesce MCPs let it use advanced functionality from Transform, Catalog, and Quality. Platform skills teach it best practices, and Coalesce Desktop lets you iterate fast with the UI.

If you want to get your hands dirty, we've put together a full guide with setup steps and copy-paste prompts you can follow.

Why this matters to you

With an agent, writing code is no longer the bottleneck. But agreeing on modeling standards, designing the architecture, deciding what governance applies, and knowing whether the numbers are right are still hard problems. We've added a set of new tools to make Coalesce work well for agentic development, while still enforcing standards such as naming conventions, node types, and SQL templates. The agent's work shows up as a visual pipeline you can review before it's pushed to production.

We believe the 80% should be automated, but it should still be easy for you to tweak the last 20% in ways you're comfortable with, whether that's in code or in the UI. Below, we walk through an example across the four stages.

Define

The operations team needs a data model around trip distance for their pricing work, so they set up a call with the data engineer. The first draft of the spec comes straight from the call transcript, as a Markdown file in Git that the agent will later build from.

Next, the agent picks the tables and sources to build from. At most companies, context for finding the most relevant tables is scattered or lives in people's heads. The Catalog MCP, which connects the agent to Coalesce Catalog, puts it in one place, so the agent can check for

• who owns the table and is accountable for it
• how many people and queries use it, and which dashboards depend on it
• whether its data quality checks are passing
• its tier, domain, and deprecation notes, so it can skip a look-alike table with a different trip distance definition

The agent then combines it all into a final spec, with metadata from Catalog, quality expectations (e.g., a trip distance can't be zero or negative), and your answers to the questions it raised. The more context and detail the spec captures, the more likely the agent is to build the right pipeline on the first try.

Build

Once the spec is signed off, Coalesce can build the pipeline from it automatically.

The agent builds a pipeline from the spec, and the view shows up in Coalesce Desktop with its flag columns.

The agent builds a view on top of the trusted fact table, reuses the existing dimensions, and adds a flag column for each quality expectation. All of that is inferred from standards built into Coalesce, which you can easily extend with your own.

• Built-in patterns. For example, node types enforce consistent naming and give every dimension the same prefix, keys, and system columns.
• Best practices as skills. Coalesce ships best practices as skills, like never building a dimension straight from a source without a staging node in between for the logic.
• Your own standards. Anything specific to your team, added as new skills.

You can review the result as a visual pipeline in Coalesce Desktop, where small edits are often faster than in code, and you stay in control of what gets merged into production. The spec and the pipeline live in the same repository, and aren't pushed to production without your review.

Govern

Once the pipeline is built, it needs the right level of governance. Tests catch data issues before changes are merged and before they reach production. Owners and descriptions help people understand the data, and PII tags control who can see sensitive data. Skills describe the standard, like a governance rollout skill that assigns owners, and the agent drafts the rest for you to approve.

A new pipeline gets its tests from the spec. Older, hand-built pipelines rarely have that, so we point the agent at one of those instead.

With the Quality MCP, the agent walks the lineage, reads each node's code, and recommends relevant tests and anomaly monitors. Reading the code also surfaces harder-to-catch issues, like a join on the wrong ID that gives every trip the same drop-off zone as its pickup.

The recommendations follow our best practice testing approach, which goes much further than what most teams do today. When we've analyzed workspaces before working with a team, 97% of tests were null or uniqueness checks, which hardly represent the real world. Our recommended approach tests pipelines layer by layer rather than node by node. It focuses on the sources and the final layer, and avoids testing the same data twice, such as on passthrough columns.

What's left is making sure the pipeline has the right guardrails and is monitored continuously in production.

Observe

With the acquisition of SYNQ earlier this year, Coalesce now has native data observability through Coalesce Quality. It monitors the pipeline continuously in production, flagging issues in near real time.

Say an anomaly monitor flags that one vendor's daily revenue has dropped to zero. The issue opens in Coalesce Quality and is routed to the table's owners. With the Quality MCP, the agent investigates using metadata such as Git commits, schema changes, lineage, and similar past issues.

The agent works through a set of checks, from code and schema changes to upstream dependencies and similar segments. If it finds a likely root cause, it can propose a fix you can implement in Transform.

You still provide input to fine-tune the monitors and fix errors, creating a self-improving loop. For example, you can mark an issue as 'Expected' to tell the monitor not to alert on a similar issue next time. You also approve the fix and decide which important issues become incidents, which capture context about what went wrong that isn't in the metadata. That feedback, and every incident description, is saved to the context layer and shown in Catalog.

What the agent does, and what you decide

At every stage, the agent now does most of the work, from research and drafting to SQL and the first pass on every issue. The human's job shifts to making calls that need business context or accountability, where a data engineer's time has always been most valuable.

Try it yourself

The full guide walks through every step with prompts you can paste into your own agent. It runs on a free Snowflake trial and a free Coalesce Transform trial, so anyone can try it.

Resources