The most common question we get from enterprise clients in regulated industries isn't "which AI should we use?" It's "can we use AI at all without sending our data to someone else's cloud?"
The answer in 2026 is yes — and it's becoming more practical, not less, as the gap between frontier closed-source models and open-weight alternatives narrows faster than most people realize.
But "yes, you can" and "here's exactly how" are different conversations. This is the second one.
Who Actually Needs Private AI
Let's be clear about who this guide is for, because "private AI" has become a marketing term that gets applied to things it doesn't describe.
You need private AI — truly air-gapped or on-premise deployment — if any of the following apply:
Healthcare: Patient data, clinical notes, diagnostic records, and anything covered by HIPAA cannot be sent to external APIs without a BAA in place. Some healthcare organizations have BAAs with OpenAI or Azure OpenAI. Most don't, and for those, every query that contains PHI is a compliance violation.
Financial services and banking: Trading strategies, client portfolios, M&A deal data, proprietary models. Any of these hitting an external LLM API is a potential material non-public information (MNPI) exposure. Regulators are paying attention.
Defense and government: Classified and sensitive government data often cannot leave specific network environments by law. FedRAMP, IL4/IL5/IL6 requirements, and ITAR compliance all constrain where compute can happen.
Legal: Attorney-client privilege is extraordinarily sensitive. Sending privileged communications through a third-party API creates real exposure — both reputational and potentially legal.
High IP-sensitivity companies: Pharmaceutical formulations, proprietary manufacturing processes, trade secrets. If your competitive advantage lives in your data, running it through someone else's API is a real risk.
If you're in none of these categories — if you're a SaaS startup or a digital agency or a marketing team — you almost certainly don't need on-premise AI. Use the managed APIs. They're cheaper, more capable, and constantly improving. Don't add infrastructure complexity you don't need.
The Capability Landscape in 2026
Two years ago, the case for on-premise LLMs was complicated by a genuine capability gap. The open-weight models available were noticeably worse than GPT-4 for complex reasoning, instruction following, and most knowledge-intensive tasks.
That gap has narrowed dramatically. Here's where things stand:
Llama 3.x (Meta): The strongest general-purpose open-weight family. Llama 3.1 70B and 405B are genuinely competitive with GPT-4o on most enterprise tasks. The 70B model is the sweet spot for most deployments — capable enough for complex workflows, efficient enough to run on reasonable hardware. The 405B model is frontier-level but requires serious GPU infrastructure.
Mistral and Mixtral: Strong on reasoning and code. Mixtral 8x22B (a sparse mixture-of-experts architecture) gives you performance close to a 70B dense model at much lower inference cost. Excellent for use cases where query volume is high and you need to control cost per query.
Qwen 2.5 (Alibaba): Don't overlook this. The 72B model has become competitive with Llama on most benchmarks, and it has strong multilingual capability — important if you're deploying across languages.
Code Llama and related: If your use case involves code generation, security analysis, or technical document processing, the code-specialized models are worth evaluating separately from general-purpose models.
The emerging picture: For most enterprise document processing, Q&A, summarization, and agentic workflow use cases, a well-configured 70B model on adequate hardware is within 5-10% of frontier closed models. That gap is shrinking with each model release.
Architecture Choices
Before getting into hardware and software, get clear on which deployment pattern fits your situation.
Option 1: On-Premise (Fully Air-Gapped)
The model weights, inference engine, and all data stay entirely within your network. No external API calls. Suitable for classified environments, healthcare with strict PHI requirements, and organizations with data that absolutely cannot leave their perimeter.
Pros: Maximum data sovereignty, no external dependencies, no API cost.
Cons: Highest infrastructure cost, your team owns all operations, model updates require internal process, no access to frontier capabilities (GPT-4o, Claude 3.5) without Azure/AWS private endpoints.
Option 2: Private Cloud (VPC Deployment)
Model runs in your own cloud account (AWS, GCP, Azure) — not on shared infrastructure, not routed through vendor APIs. Your VPC, your access controls, your data never leaves your cloud account.
Pros: Flexible scaling, no hardware capex, easier operations than on-premise, can often satisfy compliance requirements for HIPAA/SOC2/GDPR.
Cons: Still cloud-dependent, ongoing compute cost, some organizations have data residency requirements that even VPC deployment doesn't satisfy.
Option 3: Hybrid (Tiered by Sensitivity)
Route queries by data sensitivity. Public or low-sensitivity queries go to managed APIs for best performance. Sensitive queries with PII or proprietary data stay on private infrastructure. This requires a classification layer but gives you the best of both.
Pros: Balances capability and compliance, often the most practical for mixed-workload environments.
Cons: Requires classification logic, more complex architecture, doesn't work for organizations that need all AI to be air-gapped.
The Inference Stack
Once you've chosen your deployment pattern, you need an inference stack. This is the software that loads the model weights and serves requests efficiently. Three options dominate enterprise deployments:
vLLM: The most widely deployed open-source inference engine. Built by researchers at Berkeley, now used in production at some of the largest AI deployments in the world. Key advantages: PagedAttention (dramatically improves throughput for concurrent requests), continuous batching, support for most major model architectures. Best for: high-throughput deployments where many users are making requests simultaneously.
Ollama: Simpler to set up, excellent for smaller deployments and development environments. Not designed for high-throughput production — it handles one request at a time well, struggles with concurrent load. Best for: development environments, small-team internal tools, POCs.
TGI (Text Generation Inference — Hugging Face): A solid production option, especially if you're working closely with Hugging Face's model ecosystem. Good observability, supports quantization, active development. Best for: teams already in the Hugging Face ecosystem.
SGLang: A newer entrant that's showing impressive throughput numbers, especially for structured output (JSON mode) and multi-turn conversations. Worth watching and evaluating for high-throughput use cases.
For most enterprise production deployments, we recommend starting with vLLM. It's battle-tested, has excellent documentation, and the throughput characteristics are the best in class.
Hardware Requirements (Real Numbers)
This is where many guides get vague. Here are concrete numbers for the models we deploy most:
Llama 3.1 8B (great for simple tasks, fast):
- Minimum: 1x NVIDIA A10G (24GB VRAM)
- Good: 1x NVIDIA A100 40GB
- Throughput: ~400-600 tokens/second at typical batch sizes
- Cost (cloud): ~$3-4/hour for A10G, ~$10-12/hour for A100
Llama 3.1 70B (the enterprise sweet spot):
- Minimum: 4x NVIDIA A100 40GB (or 2x A100 80GB with quantization)
- Good: 2x NVIDIA A100 80GB (full precision)
- Better: 4x NVIDIA A100 80GB (higher throughput)
- Throughput: ~80-120 tokens/second
- Cost (cloud): ~$20-40/hour depending on configuration
- For on-premise: 4x H100 gives you much better economics at scale
Llama 3.1 405B (frontier capability, serious commitment):
- Minimum: 8x NVIDIA A100 80GB
- Better: 8x H100 80GB
- Throughput: ~20-40 tokens/second
- Not for most enterprise deployments — this is for organizations where frontier quality is genuinely necessary and budget is not the primary constraint
Quantization note: INT4 and INT8 quantization (reducing model precision) can cut memory requirements by 50-75% with a 5-15% quality hit on most tasks. For many enterprise use cases, INT8 is a reasonable trade. For high-stakes applications (medical, legal decisions), evaluate quantized vs full-precision on your specific task before committing.
The Deployment Checklist
This is what "production-ready" actually means for a private LLM deployment:
Infrastructure
- GPU cluster provisioned and tested
- Networking configured: firewall rules, VPC routing, no external egress for AI traffic
- Storage: fast NVMe for model weights, separate volume for logs
- Redundancy: what happens if one GPU node fails?
- Backup and recovery for model weights and configuration
Model Serving
- Inference engine deployed (vLLM / TGI)
- API gateway in front of inference engine (authentication, rate limiting, logging)
- Health checks and auto-restart configured
- Load balancing if running multiple inference instances
Security
- Network-level isolation confirmed (can the inference server reach the internet? It shouldn't.)
- Authentication on all API endpoints (never expose a bare vLLM endpoint)
- Audit logging: who queried what, when (not the content of queries if sensitive, but the metadata)
- Secrets management for any API keys used by the system
- Data at rest encryption for model weights and logs
- Penetration test before go-live
Observability
- GPU utilization and memory monitoring (Prometheus + Grafana, or Datadog)
- Request latency tracking (P50, P95, P99)
- Error rate monitoring
- Token throughput tracking (so you can right-size your infrastructure)
- Alerting: who gets paged if the inference server goes down?
MLOps
- Model weight versioning (which version of Llama 3.1 are you running exactly?)
- Rollback procedure if a model update degrades performance
- Update process for security patches
- Prompt versioning (changes to your prompts should be tracked like code)
RAG on Private Infrastructure
Most private AI deployments need retrieval — the model needs to answer questions about your internal documents, not just its training data. This requires a private vector database and embedding model as well.
For the vector database, three options dominate private deployments:
pgvector (Postgres extension): If you already run Postgres, this is often the right answer. Production-ready, familiar operations, ACID guarantees, no new infrastructure. Handles tens of millions of vectors well. Starts to have performance issues at hundreds of millions of vectors at high query rates.
Weaviate: Purpose-built vector database with strong filtering capabilities, good performance at scale, can run fully on-premise. Good choice if you need more than pgvector can handle or need more sophisticated filtering.
Qdrant: Rust-based, very high performance, excellent for high-throughput use cases. Growing enterprise adoption. Strong choice if query volume is high.
For embeddings, you also need a private model — sending your documents to OpenAI's embeddings API defeats the purpose. The best options:
BGE (BAAI General Embedding): The community standard. BGE-large-en produces embeddings that match or beat text-embedding-ada-002 on most retrieval benchmarks. Runs on a single GPU, even a modest one.
E5-mistral-7b: Stronger than BGE for complex reasoning tasks, but requires more compute. Worth the cost if your retrieval accuracy is the limiting factor.
Common Failure Modes
GPU memory management: vLLM's PagedAttention is excellent, but if you misconfigure KV cache settings or max_model_len, you'll hit OOM errors under load. Test with realistic concurrent load before going live.
Latency surprises: A 70B model serves tokens at ~80-120/second. That's fast in aggregate, but a single long response (2,000 tokens) takes 20-25 seconds. Design your UX for streaming — display tokens as they're generated, don't make users wait for the complete response.
Context window management: Llama 3.1 supports 128K context, but running at full context length is expensive. Most enterprise use cases work well at 4K-8K context if you design your RAG chunking correctly. Don't assume you need full context — measure what actually matters for your use case.
Model drift from prompts: When you update your prompts, your model's behavior changes — sometimes in ways you didn't expect. Version control your prompts, test before deploying, and have a rollback procedure.
Overconfidence in model outputs: Open-weight models, like all LLMs, hallucinate. Don't remove human review because the model seems good in testing. Build validation steps for any output that drives a real decision.
The Total Cost Picture
Private AI isn't free. Let's be honest about what it costs:
Cloud deployment (typical 70B model, 50 concurrent users):
- Compute: ~$15-25K/month (2x A100 80GB instances)
- Storage: ~$500/month
- Operations: 0.25 FTE DevOps/MLOps
- Total: ~$20-35K/month
On-premise deployment (amortized over 3 years):
- Hardware: $150-300K initial (4x H100 cluster)
- Power and cooling: ~$3-5K/month
- Operations: 0.5 FTE
- Total: ~$8-15K/month at scale
API cost comparison:
- GPT-4o: $5-15/million tokens (input/output varies)
- At 50 concurrent users doing meaningful work: $30-80K/month at typical usage
For high-volume use cases, private infrastructure has a clear cost advantage over API usage at scale. For low-volume use cases, the API is almost always cheaper after accounting for infrastructure and operations.
The crossover point is typically around 50-100M tokens/month. Below that, managed APIs win on economics. Above that, private infrastructure starts looking attractive on cost alone — before accounting for compliance requirements.
Getting Started
The right first step isn't building a production cluster. It's a structured proof of concept that answers three questions:
- Does the open-weight model you're considering actually perform adequately on your specific task?
- What does the infrastructure cost look like at your actual usage patterns?
- What are the operational requirements and do you have the team to handle them?
Run a 3-week POC: one GPU node, a small model (8B or 14B), your real data, your real queries. Measure accuracy, latency, and cost per query. Use that to extrapolate to production requirements.
Don't sign a hardware purchase order before you've done this. The model that sounds best on paper may not be best for your specific domain. The infrastructure that benchmarks best may not fit your operational capabilities.
Private AI is genuinely viable for regulated enterprise environments in 2026. But like all infrastructure decisions, the cost of getting it right upfront is a fraction of the cost of getting it wrong.