Article
Data pipelines fail in predictable ways: a crash mid-batch, a lost checkpoint. Here's why durable execution is becoming the fix, and why it's moving into the database itself.
Anyone who has run a data pipeline in production knows the specific way it fails. A batch job dies partway through, and now you don't know which rows finished and which didn't. A worker crashes after writing the results but before marking the source record as processed, so the next run either skips real work or duplicates it. An API call to an embedding service times out mid-batch, and there's no shared checkpoint telling you where to resume.
Microsoft's answer, an open-source PostgreSQL extension called pg_durable, tackles this by moving the fix directly into the database rather than adding another service to operate. Released in June, it lets developers define a data workflow as a graph of SQL steps that PostgreSQL checkpoints as it executes. If the database crashes or a step fails, execution resumes from the last saved checkpoint automatically, without a separate orchestration layer or a hand-built status table tracking retries.
Why this problem usually gets solved badly
Most teams solve pipeline reliability one of two ways, and both come with real costs. The first is standing up a dedicated workflow orchestration tool, something like Temporal or Airflow, which works but adds a whole separate system to run, monitor, and keep in sync with the database that actually holds the data. The second is hand-rolling the reliability logic: a status column, a retry counter, a dead-letter table for rows that failed too many times. That approach is what most teams building their first data pipeline end up with, and it's also the first thing that breaks under real load, because retry logic written under deadline pressure rarely accounts for every failure mode until one of them happens in production.
pg_durable's pitch is that Postgres already solved this problem for your data. Every row you write is protected against a crash by the database's write-ahead log. pg_durable extends that same guarantee to the workflow itself, so the steps of your pipeline get the same durability as the data they're producing.
Where this matters most: AI-driven data pipelines specifically
The use case Microsoft highlights most directly is one a lot of businesses are building right now without realizing how fragile it is: a pipeline that chunks documents, calls an embedding API, and writes the results into a vector database for search or retrieval. That pipeline has multiple points of failure, an external API call, a batch write, a dependency between two systems, and no natural checkpoint unless someone builds one in.
The project's early reception backs up that there's real appetite for this. It reached the front page of Hacker News on launch day and passed 1,700 GitHub stars within its first few days, with independent engineers publishing their own walkthroughs and stress tests within a week of release.
The tradeoff worth knowing before adopting it
An independent evaluation of pg_durable found real rough edges: recovery after certain failures required the workflow's own logic to be written carefully to handle being resumed, and in-flight workflows didn't always behave predictably if the underlying SQL function definitions changed while a workflow was paused. It's still a preview-stage project, and teams adopting it now are choosing to be early rather than choosing something fully proven at scale.
What this means for building reliable data infrastructure
The pattern here matters more than this specific tool. As more data and reporting infrastructure gets built around AI-driven steps, embeddings, extraction, enrichment, the pipelines running them need the same reliability guarantees that databases have offered for decades. Whether that durability lives inside Postgres, in a dedicated orchestration tool, or somewhere else, treating it as optional is how a working dashboard turns into a support ticket the first time a batch job crashes at 2 a.m.