We cut our indexing failure rate from about 20% to below 0.1% by moving StackAI's ingestion pipeline from Celery to Temporal, one stage at a time. Along the way, parsing costs fell to roughly a tenth of what they were. This post covers what the original pipeline looked like, why it broke, and how we rebuilt it. For the interview version of this story, see Temporal's case study, How StackAI used Temporal to cut indexing failures from 20% to below 0.1%.

"As we scaled, it became clear we couldn't keep patching our document pipeline. We had to rethink it from first principles, both the architecture and the tools, and build something that could keep up with our growth, with reliability as a hard requirement. Temporal became the foundation of that."

- Gabriel Mermelstein, Tech Lead, Knowledge Bases at StackAI

From raw drive to searchable knowledge

StackAI, now part of Asana, gives regulated enterprises a no-code way to build AI workflows with governance and guardrails. A core piece of that is the knowledge base: customers connect Google Drive, SharePoint, or Dropbox, and we turn those files into searchable context for their agents.

From the customer's side, connecting a drive is one click. Behind that click is a pipeline. A typical drive holds a thousand files or more, from small text files to large PDFs, images, and Word documents. Each file has to pass four stages, and each stage fails in its own way:

Every stage has to succeed for a file to become searchable. With a thousand or more files per drive, some of them will hit at least one of those failures.

Where we started: Celery and Flower

Our original indexing pipeline was built on Celery, with Flower providing monitoring and visibility. This was a pragmatic choice for an early-stage Python backend, and it served StackAI well as the platform grew. However, what worked well for an MVP was no longer sufficient for the reliability, observability, and operational guarantees we needed at scale.

The problem: one file in five failed to index

About 20% of files failed to index on the first attempt. Indexing was the most common thing customers complained to us about. On Celery, three things made each failure expensive:

• Retries repeated finished work. When a later step failed, the retry re-ran steps that had already succeeded, including paid parser calls.

• We couldn't see where things broke. Flower told us a task failed. It didn't tell us which file, at which stage, as part of which customer's ingest. Debugging meant adding logging and waiting for it to happen again.

• Resources ran out. The pipeline is I/O-heavy, and our Celery setup ran synchronously, so we used threads for parallelism. Under load we could run out of workers or memory.

Why Temporal

More logging, more retry rules, and more workers each bought less as volume grew. The underlying issue was that the pipeline's state lived nowhere durable. So we set one requirement for the rebuild: every stage had to be durable, observable, and retried on its own.

Temporal fits that requirement directly. It records the progress of each workflow, so a failed stage retries on its own: if the download succeeded, we don't download again; if extraction succeeded, we don't pay a parser again. The Temporal UI shows how far each execution got and where it stopped, which is the per-file, per-stage view Flower couldn't give us.

Security was the other non-negotiable. Customer data has to stay encrypted, and we can't have access to it. Temporal let us keep that guarantee while still getting durable, observable workflows.

How we built it

Migrate incrementally

We moved the pipeline to Temporal Cloud stage by stage, with no big-bang rewrite. Think of the knowledge-base pipeline as ten stages. We made stage one durable on Temporal, then stage two, and kept going until every stage ran there. Each step shipped on its own, so we never bet the product on a cutover.

Route work before it starts

A router splits each ingest before any workflow starts. Files are grouped into workflows by type and size, and anything over 50 MB gets a workflow of its own, so one huge PDF can't stall a batch of small ones. Inside a workflow, each stage fans out per file, and one bad file doesn't fail the batch.

Decide what not to retry

Retrying only the failed stage is half the story. The other half is not retrying at all when it can't help. A single error classifier decides:

• Non-retryable: corrupt, oversized, or unsupported files, and API requests rejected outright. These fail once.

• Retryable with backoff: rate limits, timeouts, network errors.

We don't spend worker time or paid API calls on work that can't succeed.

Make parsing its own service

We pulled parsing out of the monolith into a separate service, also on Temporal. Each file type maps to an ordered chain of parsers, each suited to different documents. We always use the top parsers on the market, such as LlamaParse and Reducto. Every parser in the chain runs as its own child workflow, so when one provider fails, the file moves to the next.

Results are cached by a hash of the file's contents plus the parser settings. The service checks that cache before fetching anything, so an unchanged file is never parsed twice.

Wait for slow parsers without holding a worker

Some external parsers take minutes. Here's how a parsing workflow waits for one:

1. Submit the job with a correlation ID that points back to this exact workflow.

2. Park on a durable wait with a timeout. It costs nothing while the provider works, and it survives restarts.

3. When the job finishes, the provider calls our webhook. We validate the request, use the correlation ID to find the waiting workflow, and signal it.

4. The signal carries only a small status, not the document. The workflow wakes, confirms the status, and fetches the full result in an Activity.

5. If the webhook never arrives, the timeout fires and the chain moves to the next parser.

Scale each workload separately

Downloads mostly wait on the network; parsing a large PDF needs a lot of memory. So parsing runs with its own Temporal workers and task queues. We size that fleet for memory and the main backend for I/O, and scale one without the other.

A few more rules keep things fast:

• Nothing heavy runs in the application's event loop, so the product stays responsive during large ingests.

• Files and parsed documents stay in object storage. Activities pass signed URLs, which keeps workflow histories lean.

• New indexing work is admitted in a controlled way, so a burst from one customer can't starve everyone else. A workflow waiting its turn sleeps on a durable timer instead of holding a worker.

• Temporal task priorities dispatch in-flight work ahead of newly enqueued work.

Results

Indexing errors dropped from about 20% to below 0.1%, and customer complaints about indexing stopped. Parsing costs fell to roughly one-tenth of their previous level.

Temporal wasn't the only change. Alongside our own work optimizing the parsing service, Gabriel Napadenschi tuned the pipeline end to end, and Erik Wrede optimized how we use external parsers such as LlamaParse and Reducto. What Temporal did was make those gains stick: completed work is preserved, and retries happen at the right level.

What's next

Next, we're making the workers smarter about load. Our demand is spiky, and keeping a large fleet running in case a spike arrives is expensive. Our infra team, led by Lokesh Challagulla, is tuning how workers scale with demand.

"Temporal is the silent hero of our stack: once you put it in, it works so reliably that you forget it’s even there."

— Frank Hidalgo, CTO of StackAI