How to build your context API
A practical guide to building an LLM-facing context API.
An incident fires minutes after a production deploy. checkout-api is failing, but the alert does not tell us who should own the bug.
To route it, an agent has to connect the incident to the service, the service to the deploy, the deploy to the code change, and the code change to the owning team.
How do you provide this context to the LLM?
You need a proper context API.
A bad pattern finding adoption is giving the LLM access to internal systems without carefully auditing what resources and actions should be allowed for a probabilistic and unaccountable agent workflow.
A better design would look something like this -
Source systems → context service → declared tools (thin wrapper around endpoints) → LLM
In this blog we're going to dig into the "context service" side of the problem, and how to build it.
We're not going to explore MCPs or A2A protocols here, for brevity.
What is an LLM-facing context API?
A useful context API does the following:
Identifies the caller
- the human or workflow running the agent.
Resolves the task:
- the facts and relationships.
Composes context:
- across databases and files.
Applies policy:
- authorization and redaction happen before data reaches the model.
Controls the budget:
- latency, traversal depth and result size are bounded.
Returns metadata:
- sources, missing data, and conflicting signals are explicit.
Some guidelines for building a good context API
1. LLMs should not know about your internal systems
The easiest mistake is to mirror your backend as tools:
get_service(service_id)
search_deploys(filters)
execute_graphql(query)
query_graph(expression)
Now the LLM must discover which systems matter, make several calls, and understand your ownership rules.
Design tools around the actual task instead:
get_incident_routing_context(incident_id)
get_change_risk_context(change_id)
get_customer_impact_context(incident_id)
explain_service_blast_radius(service_id)
Describe the desired outcome without naming your internal systems
“Tell me which team probably owns this incident and show me why” is a good capability.
“Let me run arbitrary traversals over the service graph” is not.
2. Design your tool contracts
The model-facing input can remain simple:
{
"incident_id": "inc-4821"
}
That is the only argument the model supplies.
Identity and authorization context should come from the trusted runtime, not from model-generated fields.
The context API should receive data from the runtime like:
principal_id · organization_id · scopes · agent_id · request_purpose
For example, the handler might execute with this trusted request context:
{
"principal_id": "user-173",
"organization_id": "acme",
"scopes": ["incidents:read", "services:read", "deploys:read"],
"agent_id": "incident-router-v2",
"request_purpose": "route_incident"
}
The response should contain the conclusion and the evidence required to explain or challenge it:
{
"status": "partial",
"suggested_owner": {
"team_id": "payments-platform",
"basis": "The affected service is owned by this team and the latest deploy contains a change from its repository."
},
"evidence": [
{
"type": "affected_service",
"id": "checkout-api",
"source": "incident:inc-4821",
"observed_at": "2026-07-20T09:31:12Z"
},
{
"type": "recent_deploy",
"id": "deploy-1842",
"source": "deploy:deploy-1842",
"observed_at": "2026-07-20T09:26:44Z"
},
{
"type": "service_owner",
"id": "payments-platform",
"source": "service:checkout-api",
"observed_at": "2026-07-20T08:58:03Z"
}
],
"ambiguities": [
"The current on-call rotation could not be verified."
],
"freshness": {
"evaluated_at": "2026-07-20T09:31:19Z",
"oldest_source_age_seconds": 1996
}
}
This is more useful than a bare answer such as {"team":"payments-platform"}.
It is also better than a dump of raw records.
Each material fact should carry a stable source identifier and timestamp.
This contract also makes failures diagnosable. A wrong answer can be traced back to the reason.
3. Separate context tools from action tools
Reading context and changing state are different risk classes. Do not hide an assignment, deployment, or ticket update inside a context call.
get_incident_routing_context(incident_id) # read only
propose_bug_assignment(issue_id, team_id) # creates a reviewable proposal
assign_bug(issue_id, team_id, approval_id) # performs the action
A safe flow is:
read → propose → approve → execute
The read tool should be side-effect free and safe to retry.
The action tool should be independently authorized, idempotent, and fully audited.
High-impact actions can require human approval or a policy-issued approval token.
4. Bound and observe every context request
Policy decisions should be observable.
Log the caller, tool, arguments, policy decision, sources accessed, redactions, response status, latency, and outcome.
So the real question is not only:
Will the model vendor see this data?
It is also:
Should this specific model call see this data at all?
5. Choose the interface behind the tool boundary
GraphQL is a layer that can exist between the tool and your storage database, exposes a typed schema.
The schema says which objects exist, which fields they have, and which relationships clients can follow.
A query can ask for exactly the fields and nested relationships it wants. For "who is working on what?", that is a natural fit:
| Interface | Best role | Main strength | Main risk |
|---|---|---|---|
| GraphQL | Compose flexible, nested context | Typed relationships and flexibility | An open query surface can become difficult to govern |
| REST | Predictable operations | Familiar HTTP semantics, caching, and observability | Relationship-heavy workflows can create endpoint sprawl |
| RPC | The LLM-facing boundary | Intent based with clear scope and authorization | Maintainability |
GraphQL, REST, and RPC/tool calls solve different interface problems and can coexist in the same context API.
The best one depends on your scope, reliability, security and flexibility requirements.
6. (Optional) Graph Databases as the storage layer
GraphQL is storage-agnostic. It works with relational databases such as PostgreSQL or MySQL, or even REST services.
A graph database on the other hand stores relationships as first-class data. They come with their own query languages that let backend code describe paths and patterns directly:
Incident -> Service -> Deploy -> Change -> Team
They're specially useful when graph traversal and relationship mapping is the core workload. GraphRAG, for example, uses them.
The danger is giving the LLM raw graph query access. Even if you take the necessary precautions, the model can still be tricked. The user can still ask for data they should not see. A Jira comment can still contain prompt injection. A retrieved document can still include secrets.
Use a graph database when you need graph storage. Do not make it the LLM tool surface.
LadybugDB for in-memory graphs
For agent builders, LadybugDB is worth a look.
It can run fully in memory or persist to disk when needed. Because it runs inside the application process, it is useful for prototypes and local graph workloads without deploying a separate database server.
In-memory mode is ephemeral, so treat it as working context rather than the system of record.
Takeaway
Do not expose your internal systems, open GraphQL endpoint, or raw graph DB surface directly to the model.
Expose auditable context tools that answer a task, and enforce policy before any data reaches the LLM.
Use a graph database (as storage) when relationship traversal needs to be fast.