Here's a compliance question that stops most RAG implementations: when your RAG system retrieves a document and sends it as context in a prompt to a cloud API, that document just left your perimeter.
For non-sensitive documents, this is fine. For patient records, financial statements, legal work product, or any data covered by HIPAA, GDPR, SOC 2, or legal privilege, it's a problem that your legal team will eventually notice.
Rackspace's 2026 private cloud AI trends report identified this as Trend 6: RAG pipelines and sensitive workloads are moving on-premises. Healthcare, financial services, and legal are leading the shift.
This post covers the technical architecture, the compliance argument, and what a private RAG deployment actually requires to run in production.
Why Cloud RAG Creates a Compliance Problem
Standard RAG architecture has three steps at query time:
- Take the user's question, convert it to a vector, search the vector database for similar chunks
- Take the retrieved chunks and inject them into a prompt as context
- Send the full prompt (question + retrieved context) to the LLM for generation
In cloud-based RAG, step 3 sends the retrieved context to an external API. That context contains your actual documents, the specific passages the retrieval system found relevant to the query.
For most enterprise knowledge bases, those documents contain business-sensitive information. For healthcare organizations, they contain clinical notes, patient records, care protocols. For financial services, they contain trading strategies, client portfolios, proprietary analysis. For law firms, they contain client communications and work product.
Under HIPAA: Patient health information (PHI) that leaves the covered entity's infrastructure without a proper Business Associate Agreement triggers reporting requirements and potential penalties. Using a third-party LLM API for clinical RAG without a signed BAA: specific to PHI processing, which most major LLM providers do not offer, is a compliance violation.
Under GDPR: Personal data flowing to a US-based LLM API is a cross-border transfer that requires documented legal basis, often Standard Contractual Clauses, and the ability to demonstrate data minimization. When the data is in the context window of a RAG query, minimization is structurally difficult: the whole point of RAG is to provide relevant context.
Under EU AI Act (Article 10): Data governance requirements for high-risk AI systems include knowing where your training and retrieval data went and who processed it. Cloud API processing adds a third party to that chain.
For legal privilege: Attorney-client communications sent through a third-party LLM may weaken or waive privilege in some jurisdictions. Law firms are conservative about this for good reason.
The Performance Argument (This One Often Gets Ignored)
Compliance is the forcing function, but the performance argument for on-premise RAG is real too.
Latency: RAG systems have two round-trips at query time: retrieval from the vector store, then generation from the LLM. Cloud API generation adds network latency on top of model inference latency. On-premise inference runs on local hardware with no external network hop. For high-frequency RAG applications, this is measurable.
Throughput consistency: Cloud APIs have rate limits and can experience degraded performance during high-demand periods. A well-configured on-premise inference cluster has predictable throughput that you control.
Cost at scale: Cloud APIs price per token. RAG pipelines can be token-intensive: a retrieved context of 2,000 tokens, multiplied across 10,000 queries per day, adds up. Fixed-cost on-premise infrastructure breaks even faster than people expect at this volume.
Rackspace specifically noted that hosting RAG pipelines in private cloud supports lower latency, more consistent inference performance, and cost stability as retrieval frequency and knowledge base size grow.
What a Private RAG Deployment Actually Requires
On-premise RAG has three technical components, each of which needs careful configuration for production use.
1. Vector Database
Stores the embeddings of your document chunks. Options for private deployment:
- Qdrant: Open-source, high performance, runs in Docker or Kubernetes. Strong choice for most private deployments. Can handle tens of millions of vectors.
- Weaviate: Similar capability to Qdrant with a different query interface. Also runs on-premise.
- pgvector: PostgreSQL extension. Lower operational overhead if you already run Postgres. Lower peak performance than dedicated vector databases, but more than sufficient for most knowledge base sizes.
- Chroma: Simpler setup, better for smaller deployments. Not the right choice for high-query production environments.
2. Embedding Model
Converts your documents and queries into vectors. This must also run on-premise for full data sovereignty: if you use an external embedding API, your documents still leave your perimeter during the indexing step.
Good options in 2026:
- nomic-embed-text: Strong performance, open-source, runs efficiently on CPU or GPU
- all-MiniLM: Smaller, faster, sufficient for many English-language use cases
- BGE (BAAI General Embedding): Strong multilingual performance
Run the embedding model on the same infrastructure as your vector database to eliminate the external API dependency.
3. LLM for Generation
This is where most of the privacy risk sits in standard cloud RAG. On-premise, you run an open-weight model on your own hardware.
For regulated industries in 2026, Llama 4 is the most commonly deployed model for private RAG because of its instruction-following ability, context length, and permissive commercial license. For use cases requiring smaller context windows and lower GPU cost, Mistral variants are common.
The serving stack: vLLM is the standard inference serving framework for production on-premise LLM. It handles GPU memory efficiently, supports batch inference, and is production-stable.
The Ingestion Pipeline
Beyond the three core components, a private RAG deployment needs a robust ingestion pipeline: the process that takes raw documents and converts them to indexed embeddings.
For regulated industries, this pipeline is where your document governance happens:
Document classification: Not all documents in your knowledge base may be at the same sensitivity level. A clinical note is HIPAA data; a publicly available drug reference is not. Your ingestion pipeline should apply document-level metadata that your retrieval logic can filter on.
Access control: Who can retrieve what? In most enterprise RAG deployments, the answer should be: the same people who could read the source documents. This means your retrieval system needs to be aware of document-level permissions and apply them at query time. This is harder than most teams expect and easy to skip in the initial build.
Version control: When a document is updated, what happens to the old embeddings? You need a process for re-indexing on update and deprecating stale chunks. In healthcare, a care protocol that changes needs its old version flagged or removed, not left in the retrieval pool.
Audit logging: For HIPAA and GDPR compliance, you need to log what was retrieved, by whom, and when. This applies to both the indexing step and the query step.
The Architecture That Works
For a private RAG deployment in a regulated environment, here's the architecture that covers the compliance requirements:
[User Query]
↓
[Query Embedding, on-premise embedding model]
↓
[Vector Search, on-premise vector DB, with access control filter]
↓
[Retrieved Chunks, logged, with document provenance]
↓
[Prompt Construction, context + question]
↓
[On-premise LLM inference, vLLM serving]
↓
[Generated Response, logged with source attribution]
Everything runs within your infrastructure boundary. No document content, no query content, and no response content leaves your perimeter. The audit log covers the full chain from query to response.
What Xenith RAG and Xenith Private AI Provide
Xenith RAG is our private, enterprise-grade retrieval-augmented generation product. It runs on your infrastructure. The entire pipeline, embedding, vector storage, retrieval, generation, is deployed within your environment.
Xenith Private AI is the underlying infrastructure layer: the LLM deployment, serving, monitoring, and security controls.
For regulated industries, we build in the compliance components from day one: document-level access control, full audit logging, HIPAA/GDPR-compliant data handling, and the documentation trail needed for compliance audits.
The common alternative is to start with a cloud-based RAG system and plan to move it on-premise "when needed." In practice, moving a production RAG system on-premise is significantly harder than building it that way from the start, because the access control and audit logging have to be redesigned, not added.
If you're in a regulated industry and evaluating RAG, start the conversation with us before you build anything. The architectural decision you make now determines your compliance options later.