Why SDK-Based AI Security Beats AI Proxies

If your AI agents run Ollama locally, your proxy sees nothing. Zero. The model never makes a network call. An SDK running inside the process sees every call — cloud or local. That single fact is the most important structural difference between the two architectures, and it shapes every other tradeoff.

The proxy model is intuitive — you already route HTTP traffic through a gateway, so the analogy is natural. But it fails in ways that are structurally difficult to work around: local model blindness, per-call latency overhead, and a data privacy paradox where securing prompts requires routing them through an additional third party.

1. The Proxy Dead Zone: Local Models

The fastest-growing segment of enterprise AI deployment is local models: Ollama running Llama 3.1 on developer workstations, vLLM serving Mistral on-premises in healthcare environments, llama.cpp compiled into an application binary for edge deployments. These models make no network calls. Inference happens in a Python process, in GPU memory, with no HTTP request leaving the machine.

A network proxy is completely blind to this traffic by construction — there is no packet to intercept. This isn't an edge case. Healthcare organizations deploy local models to keep PHI off cloud APIs. Defense contractors use air-gapped inference. Developer tools use on-device models for code completion. In all of these deployments, a proxy-based governance architecture provides zero coverage.

2. Latency: Every Call Pays the Toll

A network proxy adds a round-trip to every single inference call. The actual overhead depends on your network topology, but the ranges are consistent:

• Same datacenter: 2–5ms (adds up at scale)
• Cross-region within cloud: 15–40ms
• External SaaS proxy: 50–200ms (internet RTT to proxy plus inspection overhead)
• Enterprise on-prem proxy: 10–30ms (LAN RTT plus inspection)

At 1,000 LLM calls per day through an external proxy, you're adding 50–200 seconds of pure overhead daily — overhead that users experience as the agent "thinking" longer. For an agent that makes 10–20 LLM calls to complete a task, a 100ms proxy adds 1–2 seconds of wait per task. SDK-based enforcement evaluates policy in-process in under 1ms because there is no network call involved.

Proxy latency also hits calls that would pass policy immediately. A benign prompt still pays the full proxy RTT. In-process enforcement skips the network entirely for the common case — the budget check (for shared spend state) is the only network call the SDK makes on the hot path, and only when a budget cap applies.

3. The Privacy Paradox

An AI security proxy inspects your prompts to govern them. That means your prompt text — including customer PII, patient PHI, proprietary source code, or trade secrets embedded in the context — flows through the proxy infrastructure.

If you're using an external SaaS proxy, you've added a third party to your data processing chain. That party needs BAAs, DPAs, and vendor risk assessments. You've traded one trust boundary (the LLM provider) for two (the proxy plus the LLM provider). An on-premises proxy avoids the third-party concern but still routes sensitive data through an additional network hop and process that logs and inspects it.

SDK-based enforcement never routes prompt text anywhere. Policy evaluation happens in-process in the same memory space as your application. Only audit metadata — event type, model, verdict, detected entity types — needs to leave the process. The raw prompt text stays local.

4. SDK Architecture: How In-Process Interception Works

The key insight behind SDK-based enforcement is that LLM framework libraries share common base classes. In Python's LangChain ecosystem, every chat model — ChatOpenAI, ChatAnthropic, ChatOllama, Bedrock, Vertex, and models inside agent frameworks like LangGraph — inherits from BaseChatModel. The LLM-specific logic lives in leaf methods (_generate, _stream), but the control flow funnels through shared inherited methods.

An SDK wraps those inherited funnels once, at the class level, and intercepts every call regardless of which provider is used. The patch is applied when the SDK is imported, requires no per-model configuration, and is transparent to application code. A healthcare team switching from OpenAI to a local Ollama model for PHI reasons gets identical governance coverage without changing any application code.

5. Code Example: SDK vs Proxy

With a proxy: you change every model's base URL, manage separate API key routing for each provider, and still get zero coverage for local models. Each provider has a different parameter name for the base URL — it's manual and error-prone.

# Proxy approach: reconfigure every model to point to proxy
from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic

# Must change base URL on every model — manual, error-prone
openai_llm = ChatOpenAI(
    model="gpt-4o",
    base_url="https://your-ai-proxy.example.com/v1",  # manual
    api_key="your-proxy-key"  # separate key management
)

# Anthropic has different base_url param name
anthropic_llm = ChatAnthropic(
    model="claude-3-5-sonnet-20241022",
    base_url="https://your-ai-proxy.example.com",
)

# Ollama: proxy can't intercept this — it's localhost
from langchain_ollama import ChatOllama
ollama_llm = ChatOllama(model="llama3.1:8b")  # UNGOVERNED

import inferencefort  # activates governance in-process

# Set identity context for this request/session
inferencefort.set_context(
    user="alice@acme.com",
    agent_id="support-bot",
    thread_id="conv_xyz"
)

# Every model is governed — no per-model changes needed
from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from langchain_ollama import ChatOllama

openai_llm = ChatOpenAI(model="gpt-4o")        # governed
anthropic_llm = ChatAnthropic(model="claude-3-5-sonnet-20241022")  # governed
ollama_llm = ChatOllama(model="llama3.1:8b")  # governed (local!)

# PolicyViolation raised on block — same exception regardless of provider
try:
    result = ollama_llm.invoke("My SSN is 123-45-6789, help me with...")
except inferencefort.PolicyViolation as e:
    print(f"Blocked: {e}")

6. Performance Numbers

In-process policy evaluation runs in the same memory space as your application. The numbers are not in the same order of magnitude as proxy latency:

The budget check is the one exception where the SDK makes a network call — daily spend is shared team state and cannot be decided from a per-process cache. That call goes to a regional endpoint and costs roughly 15–40ms, only when a budget cap is configured for the matching policy. Calls without a budget cap never touch the network on the enforcement path.

7. Enterprise Deployment Patterns

On-premises deployment. Regulated industries — healthcare, finance, defense — often cannot route AI traffic through external proxies. A self-hosted proxy trades one operational burden for another. An SDK has no additional infrastructure: it runs in the application process.

Air-gapped environments. Defense and intelligence applications may operate in networks with no internet connectivity. A proxy that phones home for policy updates doesn't work. An SDK with a cached policy bundle — refreshed when connectivity is available and enforcing the last-known policy when offline — works correctly in air-gapped environments.

Multi-tenant SaaS. A platform where each customer's AI agent must enforce that customer's policy needs per-request policy evaluation. A shared proxy multiplexes customer contexts across all traffic — both a performance bottleneck and a cross-tenant isolation risk. An SDK applies each customer's policy from a local cache using per-request context variables, with no shared mutable state between concurrent requests.

8. The On-Prem and Sovereign AI Case

Healthcare, defense, and financial services are converging on sovereign AI — models running in the organization's own infrastructure, under the organization's own security controls. The models are either self-hosted open-weight models or cloud models accessed under dedicated tenancy with guaranteed data residency.

In this architecture the entire inference stack is on-premises: GPU cluster, model serving layer (vLLM or Triton), and the application. There is no external API call to govern. A proxy that intercepts HTTP to api.openai.com provides zero coverage here.

9. When Proxies Still Make Sense

Brownfield applications you don't control. If you're responsible for governing AI calls from a third-party application where you cannot modify the code or install an SDK, a proxy is your only option. Network-level interception is better than none.

Non-Python/TypeScript environments. If the application runs on a runtime where the SDK isn't available, a proxy is the fallback. This is a temporary situation for most enterprise stacks.

Network-level audit requirements. Some compliance frameworks require network-level logging of all traffic independent of application-level controls. A proxy can provide this as a defense-in-depth layer even when an SDK is also present. The critical word is "also" — a proxy as a supplement to SDK enforcement is reasonable. A proxy as the primary governance mechanism has the structural weaknesses described above.

10. Choosing the Right Approach