Documentation

Integration is one package and one environment variable. The rest of this page is about the harder question — knowing, at any moment, exactly what is being governed and what is not.

Install

Python. The .pth file ships in the wheel, so the interpreter activates governance at startup and each framework is patched when your app imports it.

pip install inferencefort
export INFERENCEFORT_KEY=your-team-key
python app.py

How activation works

Four ways in, easiest first. They all reach the same place.

Patching is lazy: a sys.meta_path hook patches each framework the moment your app imports it — after the module executes, before the import statement binds it, so even from litellm import completion gets the governed reference.

Your first block

from langchain_ollama import ChatOllama
from inferencefort import PolicyViolation

llm = ChatOllama(model="gemma3:4b")
llm.invoke("What is the capital of France?")     # allowed

try:
    llm.invoke("My SSN is 123-45-6789")           # a content rule blocks this
except PolicyViolation as e:
    print(e)             # blocked by content rule
    print(e.findings)    # [{'rule': 'ssn', 'source': 'rule', ...}]
    print(e.approval)    # True when a human gate would clear it

Python

Governs LangChain and LangGraph (every provider, through the inherited funnels), LiteLLM, CrewAI, MCP tool calls, and the raw OpenAI and Anthropic clients — streaming included.

import inferencefort

inferencefort.set_context(
    user="alice@acme.com",
    agent_id="support-bot",
    thread_id=request.conversation_id,
    customer_id="acme",
)

# Crossing a raw thread boundary (a sync .invoke under an async server)?
# Wrap the work so identity is not dropped:
from inferencefort.core.context import bind_context
executor.submit(bind_context(do_work))

TypeScript

npm install @inferencefort/ai-sdk
export INFERENCEFORT_KEY=your-team-key

import { activate, setContext, PolicyViolation } from '@inferencefort/ai-sdk';

activate();
setContext({ user: 'alice@acme.com', threadId: conversationId, customerId: 'acme' });

try {
  await generateText({ model: openai('gpt-4o'), prompt });
} catch (e) {
  if (e instanceof PolicyViolation) console.error(e.message, e.findings);
}

MCP under ESM

// ESM apps that `import` the MCP SDK need the explicit wrap: the require-based
// hook resolves a different module instance under ESM.
import { wrapMcpClient, wrapMcpServer } from '@inferencefort/ai-sdk';
const client = wrapMcpClient(new Client(...));

Identity & sessions

Identity is optional for enforcement and required for attribution. thread_id is what groups a conversation, and it is what session state accumulates against.

When no thread_id is supplied the SDK mints one scoped to the current async context. There is deliberately no process-global default, so concurrent requests never collapse into one session.

Where to call setContext

Identity lives in a contextvar (Python) / AsyncLocalStorage (TypeScript), not a global. It propagates down from where you set it — through await, into tasks your framework spawns, and into every LLM and tool call underneath. It does not propagate sideways or backwards. Get the placement wrong and calls are still governed, they just carry no identity: no user on the audit event, and a fresh session per call, so the trifecta guard never accumulates its legs.

The rule

# FastAPI / Flask / any server: set it per REQUEST, inside the handler.
@app.post("/chat")
async def chat(req: Request, body: ChatIn):
    with inferencefort.context_scope(
        user=body.user_email,
        customer_id=body.tenant,          # required if you serve many tenants
        agent_id="support-bot",
        thread_id=body.conversation_id,
    ):
        return await agent.ainvoke(body.message)
    # restored on exit, so the next request cannot inherit this identity

import { withContext } from '@inferencefort/ai-sdk';

app.post('/chat', async (req, res) => {
  await withContext(
    { user: req.user.email, customerId: req.tenant, threadId: req.body.conversationId },
    async () => {
      const out = await generateText({ model: openai('gpt-4o'), prompt: req.body.message });
      res.json(out);
    },
  );
});

Where it silently does nothing

Module scope is the common mistake

# WRONG — module scope. Runs once, in the import context.
inferencefort.set_context(user="alice@acme.com")   # <-- every request is "alice"

app = FastAPI()

@app.post("/chat")
async def chat(body: ChatIn):
    return await agent.ainvoke(body.message)

# WRONG — after the call. The verdict was already decided.
answer = llm.invoke(prompt)
inferencefort.set_context(user=u)   # <-- too late, that call had no identity

Crossing a raw thread boundary

Contextvars survive await automatically. They do not survive run_in_executor, asyncio.to_thread, or ThreadPoolExecutor.submit — and LangChain’s sync .invoke() under an async server frequently runs the model call in exactly such a worker.

from inferencefort.core.context import bind_context

# WRONG — a bare thread pool does not inherit contextvars.
loop.run_in_executor(pool, lambda: agent.invoke(msg))       # empty context

# RIGHT — the worker runs under a snapshot of the caller's context.
loop.run_in_executor(pool, bind_context(lambda: agent.invoke(msg)))

Frameworks that spawn their own tasks

LangGraph runs each node body in its own task. Because contextvars propagate into child tasks, setting identity at the request boundary — outside graph.invoke() — reaches every node. Setting it inside a node does not reach the others: the LLM node and the tool node would land in different sessions, and the trifecta guard would never see all three legs together.

Checking it worked

Attribution shows up on the audit event. If user is empty or every call has a different thread_id, the context is not reaching the call site.

import inferencefort
print(inferencefort.get_context())   # {'user': 'alice@acme.com', 'thread_id': 'conv_9', ...}

Controls

Every one of these is decided in-process from a cached policy bundle. Only budget requires a round trip.

Policy & rules

The control plane is the system of record and ships policy to SDKs in a bundle, refreshed on a TTL. A policy change reaches a running process within one TTL — there is no push invalidation.

Offline / air-gapped

A policy file is a complete configuration. With no key the core makes zero network calls.

if-policy init > policy.json          # a starter policy that blocks something
export INFERENCEFORT_POLICY_FILE=./policy.json

if-policy validate policy.json        # boots the core against the file
if-policy learn audit.ndjson          # propose a policy from real traffic

Debugging

First question: is it actually on?

A layer that fails open is indistinguishable from a working one if you only check that calls succeed. Check explicitly, at startup.

import inferencefort, json
print(json.dumps(inferencefort.governance_status(), indent=2))

{
  "initialised": true,          // false means NOTHING is being governed
  "offline": false,
  "bundle_version": "0125f09f",
  "capability_gaps": {},        // configured, but could not run
  "fail_mode": "fail_open"
}

Why didn't my rule fire?

Usually the text was never in scope. Every verdict and every audit event carries scanned_scopes, so a clean result can never be confused with a screen that never ran.

"scanned_scopes": ["user-prompt", "tool-output"]

system-prompt and assistant-history are opt-in per tenant. If you expected them and they are absent, they were not enabled. Then check capability_gaps — a detector that is configured but could not be built is reported there rather than silently doing nothing.

Which gate decided this, and what did it cost?

export INFERENCEFORT_STAGE_LOG=1
export INFERENCEFORT_STAGE_LOG_FILE=./stages.ndjson

{"kind":"tool","subject":"send_email","decision":"block",
 "decided_by":"content-rule","decided_tier":"deterministic","total_ms":118.4,
 "stages":[
   {"stage":"detector","tier":"probabilistic","ms":104.2,"hit":true},
   {"stage":"session-risk","tier":"semantic","skipped":true,
    "detail":"no trigger fired"}]}

Fail modes

What happens when the control plane is unreachable. The default keeps your product up; change it when a coverage gap is worse than an outage.

export INFERENCEFORT_FAIL_MODE=fail_open     # default: allow if we are unreachable
export INFERENCEFORT_FAIL_MODE=fail_closed   # refuse instead
export INFERENCEFORT_FAIL_MODE=fail_cached   # replay the last known verdict

Environment reference