Agentic data engineering and self-healing pipelines

What is agentic data engineering?

Agentic data engineering is putting an AI agent on the work of keeping data pipelines running: reading a failed run, working out what caused it, and writing a fix that a person then approves. Self-healing pipelines is the phrase vendors use for the older half of the same promise, a pipeline that gets itself out of trouble without anyone waking up.

The two arrive in the same sentence in almost every product announcement, and they are not the same thing. Automatic recovery deals with a failure whose answer you decided in advance: try again three times, accept the new column, drop the rows that break this rule. An agent deals with a failure nobody wrote a rule for, by reading the error, the changes around it and the code, and proposing something that did not exist before it looked.

What actually breaks in a pipeline

Six causes cover nearly everything that goes wrong with a nightly load.

1. A source schema change. A field renamed, split, dropped or retyped.

2. An expired credential. A token past its expiry, a rotated password, a service account that lost a permission.

3. A late or missing delivery. Nothing on the drop at midnight.

4. A source API change. A retired version, a moved endpoint, a tighter rate limit.

5. A data quality violation. Duplicate keys, empty values where the model needs one, an amount outside anything plausible.

6. A real bug in your own logic. The join fans out, the filter drops a category, the currency conversion runs twice.

The first four are the bulk of it, and they share a property: the answer is mechanical. Nothing about the business changed.

You can read that ranking off the tools themselves. Every mainstream platform ships a retry setting, a schema-change policy and a re-authorise button. None of them ships a button for a wrong margin calculation.

The last two behave differently. A quality violation is mechanical to detect and a judgement to resolve, because the open question is whether the data is wrong or the rule is. A logic bug is the one an agent is least equipped to find on its own, because nothing failed.

What a vendor means by self-healing

When a product page says self-healing, it is nearly always one of three mechanisms, and each is documented plainly if you go and read it.

Retry. The simplest and the most useful. In Apache Airflow a task that fails with retry attempts left goes into the up_for_retry state and gets rescheduled instead of being marked failed. That covers the network blip and the API that was briefly unwell.

Schema auto-adjust. Azure Data Factory calls this allowing schema drift. Switch it on in the source and every incoming field is read and passed through to the sink, including columns that are not in your projection. Microsoft names the price in the same document: drifted columns arrive as a string type unless you also tick infer drifted column types, and the flow becomes late binding, so those column names are missing from the schema views while you build. Databricks Auto Loader defaults to addNewColumns, where a new column makes the stream update the schema and then stop with an UnknownFieldException, and restarting resumes with the updated schema.

Quarantine. Keep the run alive and put the bad rows to one side. Lakeflow declarative pipelines give every expectation one of three actions: warn, where invalid records are still written to the target, ON VIOLATION DROP ROW, where they are dropped before the write, and ON VIOLATION FAIL UPDATE, where invalid records prevent the update from succeeding and manual intervention is required before reprocessing.

All three are automatic recovery. All three act on a decision you made before the failure happened, and all three are older than the current wave of agents. Two of the three do not even heal: the Auto Loader stream stops and waits to be restarted, and a failed expectation asks for a person. Good engineering with an oversold name.

Automatic recovery versus agent diagnosis

Put them next to each other on one dimension: did anything get worked out?

The third row carries the rest of the table. A retry does not know why the call failed and does not need to. A schema setting that adopts a new column has no idea what the column means; it knows only that you said yes to new columns.

An agent that reads the stack trace, notices the connector logged a schema change forty minutes before the failure, opens the model that references the missing column and comes back with a mapping has formed a hypothesis about cause. That is worth an hour of somebody's morning, and it is the exact part that can be confidently and plausibly wrong. Recovery buys you the run. Diagnosis buys you the hour somebody would have spent finding the cause, and you only keep that hour if a person still reads the answer.

A schema change that breaks the load at two in the morning

An installation company with forty technicians runs its work orders through a field service app. A connector pulls the work-order lines into the warehouse every night at two, a transformation turns them into an hours model, and that model feeds two Power BI reports and the weekly invoicing export.

At 02:12 the transformation fails. It selects duration_minutes and the column is no longer there. The app vendor released a new version overnight: duration_minutes is gone and two fields have taken its place, duration and duration_unit, where the unit is either minutes or hours.

Here is what an agent with access to the logs, the schema history and the lineage can have ready before anybody is awake.

• The failure, named: the model reads a column the source stopped sending.

• The correlation: the connector logged the schema change at 01:47, twenty-five minutes before the failure, and nothing was deployed on your side that day.

• The blast radius from lineage: the work-order model feeds the hours model, which feeds two reports and the invoicing export.

• A look at the new data: the source now returns the last ninety days in the new shape, and of those 9,800 lines, 260 carry the unit hours and the rest carry minutes.

• A proposed change, run against a copy of the data rather than against production: take duration where the unit is minutes, multiply it by sixty where the unit is hours, plus a test that total hours per month for closed periods match the previous run.

Now the part that matters, which is the fix the agent nearly wrote instead. The shortest way to make the error go away is to map duration straight onto duration_minutes. Every run goes green. Those 260 lines average about three hours, so 780 hours arrive as 13. Total invoiceable hours over ninety days read about 18,733 instead of 19,500, a shortfall close to four percent.

Nothing about that number looks wrong. It is not zero, it is not doubled, it moves in the right direction week to week. The report refreshes, the Monday meeting uses it, the invoicing export goes out four percent light, and the person who finds it is a technician arguing about an invoice a month later. This is where pipelines differ from application code. A bad change in an application usually throws, or the page breaks and the phone rings. A bad change in a pipeline produces plausible numbers, and those get used.

Three things in this fix are not the agent's to settle.

• The meaning. Billing rounds up to the quarter hour. Does the new field carry the same rounding as the old one, or is it raw? That question belongs to the person who owns the model, or to the vendor's release note.

• The history. Ninety days of rows were built on the old field. Reprocess them, or leave them and record where the break sits? Both answers are defensible and only one is what your finance colleague expects.

• The consequence. The invoicing export leaves the building and turns into money. That is a signature, and a signature has a name on it.

So the fix goes out the way every other change goes out: a branch, a review, a deploy. What the agent removed was the hour of reading logs at seven in the morning and the guesswork about which upstream change did it. What it did not remove is the decision.

Diagnose freely, write to production through the same door

The vendors with the most to gain from selling autonomy have written the approval gate into their own product descriptions. Databricks describes Genie Code, in an April 2026 write-up, like this: when a pipeline or job fails it analyses errors, proposes updates across the relevant files, and shows you the diffs before applying any changes, and in agent mode it asks for your approval before it runs code or updates a pipeline.

Genie ZeroOps, announced in June 2026 and rolled out first as a private preview, is the more autonomous product and keeps the same gate. It monitors in the background and uses Unity Catalog lineage to trace a failure to a code bug, a schema change three tables upstream, or bad data from another pipeline, then verifies its fix in a sandbox with zero-copy clones of your data. Databricks states the boundary in one sentence: nothing gets applied to production without your approval.

That gives you a split you can hold any vendor to.

• Read anything. Logs, run history, schema versions, lineage, sample data, under the same access rules a person on that team would have.

• Propose anything. A diff, a test, a backfill plan, or a note saying it cannot work this one out. Proposals are cheap and a wrong one costs a read.

• Write to production only through the door a person uses. The same branch, the same review, the same deploy. Not a second path with fewer checks because the author is a machine.

The reason is not caution for its own sake. A pipeline that rewrites its own transformation and carries on has broken the property you lean on when a number is disputed: being able to say what produced it and when that changed. If the honest answer is that the agent adjusted a mapping on the ninth of March at 02:14 and nobody read it, your data lineage now contains a step nobody can account for. That is a defect in your lineage, not a feature with a governance question attached.

The same split tells you which pipelines an agent can help with at all. It needs tests, so a proposed fix can be shown to be right instead of asserted; dbt's generic tests are select statements that look for failing records, and zero rows back means the assertion held. It needs contracts, so a changed source has a before and an after instead of somebody's memory. It needs lineage, so the agent can see what a change touches. And it needs a rerun that gives the same answer twice, so a fix can be tried on a copy. That is the same list that makes a pipeline maintainable by a person who did not build it.

What to watch out for with agentic data engineering

If you run a smaller company you are not going to run a fleet of agents over your data platform. You will meet this as a feature inside a tool you already pay for, arriving switched on in a release note. That makes the question narrow enough to actually ask: what does it change on its own, and where is that written down?

Get the list of automatic actions. Retry, add a column, drop rows, change a mapping, rerun a backfill. For each one, ask whether it happens without a person, where it is logged, and who gets told. A vendor who answers with the word intelligent has not thought about it.

Buy it for triage first. What is real today is the failing run, the change that caused it and a candidate fix sitting in front of somebody at seven in the morning rather than at nine. The first draft of a fix is often right and always needs reading. Bought as a way to stop watching your pipelines, it will disappoint, and the disappointment will be four percent light rather than loud.