← all news

The Zero-ETL RAG Stack: Stop Duct-Taping Your AI Infrastructure

The Zero-ETL RAG Stack: Stop Duct-Taping Your AI Infrastructure

Imagine you are trying to put together a modern AI application, but every time you want to connect your data to a Large Language Model, you end up building a Rube Goldberg machine.

To get an LLM to read a simple company document, platform teams are routinely duct-taping together an S3 bucket, a standalone chunking microservice, an event bus like Kafka or SQS, and an expensive, siloed vector database. This setup duplicates state across multiple environments, breaks constantly during synchronization, and introduces a massive operational tax in data egress and compute overhead.

The fastest RAG pipeline is not a better ETL tool. It is eliminating the ETL entirely.

This post will break down a better way to tackle this challenge using the distributed-object-store-rag repository. This complete, production-grade RAG pipeline transforms your existing object storage into an intelligent AI agent. It utilizes the Model Context Protocol (MCP) to plug directly into Claude Desktop or your chat interface of choice. More importantly, it requires zero standalone database servers and zero middleware.

Here is how we built a true Zero-ETL RAG stack on OneBucket, and how you can run it yourself.

The Architecture: Storage as the Database

The core philosophy of this pipeline is that your storage layer should be the single source of truth and the orchestration point for your data.

When you drop a file into OneBucket, it triggers a serverless cascade:

  1. The Chunker: A lightweight function processes the raw document into chunks.
  2. The Embedder: Dense (BGE-small) and sparse (SPLADE) embeddings are generated.
  3. The Index: Vectors are written to an embedded index.

To achieve this without spinning up external infrastructure, we made two critical architectural decisions.

1. We use LanceDB for serverless, embedded indexing

Rather than exporting data to a managed vector silo, we use LanceDB, an open-source, embedded vector database. LanceDB operates directly on top of OneBucket and AWS S3. By treating LanceDB as an indexing query layer that lives alongside the compute, we completely eliminate the database server.

2. The index stores pointers, not text

Our LanceDB index stores only the vector embeddings and a lightweight URI pointer back to the storage layer. It never stores the raw chunk text. When the MCP server executes a hybrid search, it retrieves the highest-scoring URIs and relies on OneBucket to fetch the actual text payloads. This keeps the index footprint near zero and prevents your data from being duplicated and trapped inside a vector silo.

Solving State Without a Database

In a traditional microservice architecture, managing the state of a document as it moves through a pipeline requires a tracking database to prevent race conditions. If an embedder tries to process a document before the chunker is finished, the pipeline breaks.

To maintain our Zero-ETL standard, we moved the atomic commit point to the file system itself. The chunker writes every individual chunk file to storage first, and it writes a MANIFEST.json file last. The appearance of this manifest is the sole barrier signal that triggers the embedder. The downstream pipeline never sees a half-written document. We enforce strict state management without deploying a single tracking table.

From Local Prototyping to Multi-Cloud Production

Developers despise writing code twice. This pipeline is designed so that the exact same Python phase functions execute regardless of where they run.

If you want to test the pipeline on your laptop, the repository includes a local-daemon execution mode that polls your local directory.

However, the real power unlocks when you point it at OneBucket. By switching to the onebucket-events execution mode, your local MCP server subscribes directly to OneBucket's global events server. You get multi-cloud placement, intelligent cache tiers, and a per-request telemetry feed without changing a single line of pipeline code.

The Speed Track to AI and Future Scaling

This Zero-ETL approach, combined with OneBucket's policies for creating a unified data lake across multiple clouds and regions, is the ultimate speed track to kick off your AI initiatives. It does come with trade-offs. For instance, natively scaling a sparse vector index currently requires the text to reside within the database. Because our index stores only pointers, this solution calculates and re-ranks the hybrid search directly in the MCP layer. This works brilliantly to get you started, but it will not scale indefinitely for massive datasets.

This architecture is designed to get your AI agents off the ground instantly, before you commit to a fully optimized, heavy-duty semantic pipeline. Once you roll in specific business logic nuances and hit strict performance ceilings, you will likely lean into dedicated data warehousing and scalable vector solutions tailored for those specific needs. Until then, this approach provides the fastest path from raw data to a working AI agent.

Run It Today

You can stop managing fragile synchronization scripts and bloated vector silos.

Check out the distributed-object-store-rag repository on GitHub. To see the zero-infrastructure magic firsthand:

  1. Get your free account at app.onebucket.io and create an API key.
  2. Authenticate locally with uv run python scripts/onebucket-login.py.
  3. Run bash scripts/start-event-consumer.sh.

In a matter of minutes, you will have a production-grade, multi-cloud RAG pipeline exposing your unstructured data directly to your AI agents.

get your free account at app.onebucket.io.

sign up by December 31: 90 days free, including up to 5 TB of Wasabi storage. then $10 per TB per month. cancel anytime.

get your free account