New: FixBugs for VS Code is now available. Install from Marketplace 

BYOK

FixBugs is now Free Forever

How implemented the free BYOK tier in FixBugs, and learned while qualifying providers from frontier AI labs.

Kirti Rathore··15 min read
FixBugs is now Free Forever
FixBugs is now Free Forever

Recently I realized that having a free forever bring-your-own-key plan just fits perfectly with our self-hosted zero data retention philosophy.

So, without wasting too much time business-thinking and thumb-twiddling, we added it.

Doing so was a learning experience in how LLM APIs have evolved over the last year.

Why an LLM API adapter layer was necessary

FixBugs was originally built and tested over the DeepSeek model. Why? it was cheap and quite good.

DeepSeek, and almost all other inference providers at the time had an OpenAI-Compatible API, which gave developers a consistent request/response contract to work with.

FixBugs VSCode Extension v0.0.1 gave users access to hundreds of models through one OpenRouter API key. Initially we thought this was good enough.

It wasn't. There were so many reasons that this deserves its own post: Why FixBugs switched away from OpenRouter.

Extending our OpenRouter based implementation to our target set of 20+ providers in 2026 turned into four different wire protocols, strategies for efficient prompt caching, and other surprises.

A successful "Hi" to an LLM API is not enough

FixBugs pushes LLM APIs to its limits. When you are implementing complex workflows at production level reliability, you need a clear and reliable API contract. Most providers do not provide this.

This meant that before adding a LLM provider, we had to qualify it first.

We wrote a script that automated this qualification process, sending requests to providers and checking:

  • structured JSON schema output, strict tool contracts, and the combination of both;
  • streaming text, reasoning, prompt caching, terminal state, and consistent error statuses;
  • the complete premium, worker, multimodal, reasoning, and fallback profile for each tier.
  • usage reporting, output limits, and model-specific fields;

The most reliable validation however, were our E2E tests, which revealed a lot of model and provider specific insufficiencies.

Nine Providers made the cut for FixBugs

ProviderWhy we wanted itLessons learned
OpenRouterOne key provides a broad model market plus routing and per-request privacy controls.Llama Maverick and Seed models passed basic calls but sometimes ignored a required tool. ZDR eligibility, cache affinity, price, and tool support are provider specific.
OpenAIIt is still the clearest baseline for strict schemas and tools.The API surface is consistent; the models are not. Optional fields and tool-reasoning levels vary per model.
AnthropicHaiku is a useful fast worker, while Fable, Opus and Sonnet supply a stronger fallback and multimodal path.Its Messages API is not OpenAI-compatible. You need a separate adapter layer for their API.
Google GeminiMultimodality is native, and Flash/Flash-Lite provide useful role separation.The Models API listed Gemini 2.5 models that returned 404 for inference on our new account.
xAIGrok supplied a complete coding-oriented premium, worker, and multimodal family.Again, they have their own Responses API which is not OpenAI-compatible.
Mistral AIOpen weights, EU-hosted APIs, multimodal Large/Medium models, and small edge-capable Ministral workers.Raw generation was often fast; Response quality even for premium models is variable.
Together AIIt is a deployment platform as much as an inference vendor: serverless models, fine-tuning, dedicated endpoints.Two models behind the same OpenAI-compatible endpoint can implement schemas and tools differently.
Fireworks AIIts product is optimized inference for open models, making it attractive for latency- and cost-sensitive agents.Prompt cache entries are replica-local. Session affinity and provider-specific metadata matter.
Moonshot AI / KimiKimi targets long-horizon coding, reasoning, and large-context agent work.Moonshot's docs advertised direct K3 while our funded account still rejected a minimal K3 call. They should focus more on building out their model serving capacity.

One contract, several dialects

Within FixBugs, we use two types of LLM API adapters: LLMDriver (optimized for throughput) and StreamingLLMDriver (optimized for interactivity).

These adapters have been made robust with bounded retries, schema validation, attempt resets, usage accounting, and several user-centric failure mode handling.

Replacing those drivers for BYOK would have thrown away the difficult lessons.

Instead, we chose to add a provider translation layer that sits below them.

flowchart LR
    D[LLMDriver and StreamingLLMDriver] -->|Request| E[BYOK execution boundary]
    E --> T{Provider transport}
    T --> O[OpenAI-compatible]
    T --> X[xAI Responses]
    T --> H[Anthropic Messages]
    T --> G[Gemini GenerateContent]
    O -->|wire response or SSE| N[Normalizer]
    X -->|Responses items or events| N
    H -->|content blocks or events| N
    G -->|parts or events| N
    N -->|text, tools, usage, errors| D

The drivers still ask for text, tools, or a typed result. The lower layer decides what that means on a particular wire.

Why Groq, Cerebras, Deepseek and Z.AI are missing

Groq did not give us complete set of premium, worker, multimodal, and fallback models we needed.

Its strict structured-output mode also could not be combined with streaming or tool use, a limitation still stated in Groq's documentation.

Cerebras had even fewer models: our inventory returned three models, and we had no qualified multimodal fallback. Very fast text inference is not enough when you try to make it a part of a complex workflow.

Deepseek had the requisite model set but did not support JSON schema compliance on its API. Same with Z.ai. Both model families remain available through OpenRouter.

OpenAI: few surprises to note

  • gpt-5-mini rejected temperature, so the adapter omits it.
  • gpt-5-nano at high reasoning did not adhere to prompts requiring it to call context-gathering tools.

While the API contract says the provider supports reasoning and tools this is not guaranteed as you try a combination of the model/reasoning level/tool choice parameters.

This is why a provider-level capability flag is not sufficient. For reliable behavior the harness has to be knowledgeable about model limitations.

Gemini: OpenAI compatibility is partial

Gemini is not fully OpenAI API compatible. So we switched to use their own API for reliability.

Implementation note - Gemini API's tool calls are not stateless. Each tool call has to carry an ID and a thought signature that must be provided again on the next turn:

Request:

{
  "functionCall": {
    "id": "call-7",
    "name": "inspect_trace",
    "args": "..."
  },
  "thoughtSignature": "opaque-provider-state"
}

Response:

{
  "functionResponse": {
    "id": "call-7",
    "name": "inspect_trace",
    "response": {"result": "..."}
  }
}

Dropping the thought signature can break the next model turn and dropping the function ID can mispair parallel calls.

xAI: Another Provider Specific API

xAI's provides an OpenAI-compatible API, but we found it unreliable. We switched to using their recommended Responses API and abstracted away the transport part for both Gemini and xAI.

{
  "model": "grok-4.5",
  "input": [{"role": "user", "content": [{"type": "input_text", "text": "..."}]}],
  "store": false,
  "prompt_cache_key": "opaque"
}

Interestingly -

  • They give you API level switches for specifying whether the prompt/response may be stored. This defaults to "true".
  • An unique prompt cache key must be provided by the user to enable prompt caching.

Kimi: one model family, three different contracts

Together

Together's Kimi K2.6 and K2.7 sometimes ignored JSON response schemas or returned fenced JSON.

Since fenced JSON was actually reliable, we translated into structured JSON internally.

Fireworks

Fireworks had the inverse problem: schema-only and tool-only calls worked, but they did not work together. We had to write an adapter for this.

Moonshot

Direct Moonshot was different again. During qualification, Moonshot's K2.6 and K2.7 gave us JSON-object mode rather than strict JSON. We add a translation layer here as well.

FeatureJSON Object Mode (json_object)Strict JSON Schema Mode (json_schema, strict: true)
PurposeEnsures the output is valid JSONEnsures the output follows a predefined JSON schema
Valid JSON✅ Guaranteed under normal completion✅ Guaranteed under normal completion
Schema Enforcement❌ No schema enforcement✅ Enforces supported JSON Schema constraints
Required Fields❌ Not guaranteed✅ Enforced
Data Types❌ Not guaranteed✅ Enforced (string, number, boolean, array, etc.)
Field Names❌ Model may generate different field names✅ Must match the schema
Nested Objects & Arrays⚠️ Structure depends on model instructions✅ Structure validated against the schema
Provider Compatibility✅ Generally widely supported⚠️ Varies by provider, model, and supported schema subset

Mistral: Fast and high quality

ModelOutput Speed (tokens/s)Median Time to First Token (TTFT)
Mistral Medium 26041650.51s
Ministral 8B134~0.5s
Ministral 14B120~0.5s
Mistral Large 2512536.6s

Benchmark methodology: Measurements were collected via direct API calls after a warm-up, using a single fixed short prompt and a 256-token output limit.

Mistral Large 3 passed our complete FixBugs matrix, which was rare for a frontier open weights model during our evaluations.

Beyond keys: subscription inference

After qualifying the nine API providers, we also added Codex and Claude subscription inference support to FixBugs. This lets developers use an existing subscription instead of configuring a separate API key, where the provider allows it.

Point 3 in our OpenRouter post explains the motivation. If you already pay for ChatGPT or Claude, creating another account, buying API credits, and managing another key adds extra setup and cost just to use a debugging tool.

There is an important difference between the two providers:

  • Codex (OpenAI): FixBugs supports Codex subscription inference through OpenAI's official OAuth-based integration. Eligible users can authenticate with their ChatGPT account and use their existing subscription for supported inference requests, subject to plan limits and eligibility requirements. No separate OpenAI API key or API credits are required for this path.
  • Claude (Anthropic): We added Claude subscription integration, but Anthropic has since restricted third-party use of subscription credentials. Under its current policy, third-party applications cannot directly reuse Claude subscription credentials for API inference. As a result, Claude subscription inference is not currently a supported option in FixBugs. Developers who want to use Claude through FixBugs should use an Anthropic API key instead.

In short, FixBugs supports Codex subscription inference today. Claude is available through API-key-based access, but not through direct subscription reuse under Anthropic's current rules.

Separation of Responsibilities

We kept the sign-in flow in a layer above the code that translates model requests. FixBugs still owns prompts, conversation state, tool execution, and result validation. The provider-specific adapters are the final layer before the model call. We do not launch Codex or Claude Code to run another agent inside ours.

How to setup BYOK in FixBugs

Important things first: Privacy

The privacy boundary is deliberately boring and auditable.

VS Code stores one key per provider in SecretStorage.

OpenAI API key entered as masked characters beside Validate and Save in FixBugs Settings

Choose a premium model for complex reasoning, a worker model for lighter tasks, and a multimodal model for images and videos.

OpenAI premium, worker and multimodal model choices with reasoning levels and fallback models in FixBugs

The selected key stays in memory, and goes directly to the vendor.

Validated OpenAI provider marked Ready and Active provider in FixBugs, with its stored-key preview redacted

We collect provider and plan status, never the key.

Caveat: You may lose Zero Data Retention guarantees that come with your FixBugs Plan.

We can't guarantee zero data retention in the BYOK model. You have to set this up yourself.

When you're subscribed to FixBugs, we enforce Zero Data Retention (ZDR) across all models.

Direct BYOK inherits the account policy the user negotiated or enabled. The details are uneven:

Should you build your own LLM provider layer?

We would not tell every team to do this.

FixBugs needed an ad-hoc API driver layer because we observed a lot of errors and unreliable behavior from LLM APIs; a workflow provided validation functor was therefore crucial.

I'd encourage you to give already mature and open source LLM adapter frameworks a try.

LiteLLM is the obvious first stop for this with broad provider coverage, gateway routing, budgets, fallbacks, and observability built in.

Sergey Inozemtsev's llm-api-adapter is interesting: it handles an extensive model provider set (including local models) while being open-source and avoiding bloat. We are looking to run some evaluations with it.