← David Veznik

OpenServ Reasoning

AI · LLM infrastructure

A commercial LLM inference gateway, drop-in compatible with OpenAI and Anthropic SDKs, that wraps every model call in a reasoning harness of prompt optimization, safety guardrails, and structured-output validation with auto-repair, all metered against prepaid credits.

I helped develop an optimization layer that improves the accuracy and cost of frontier LLM models.

Context

OpenServ Reasoning is an LLM inference gateway. Customers point their existing OpenAI- or Anthropic-compatible SDKs at it without rewriting any code, and the gateway transparently improves and governs each request before forwarding it to the underlying model providers. Every call passes through a reasoning harness: versioned prompt-engineering strategies, safety filtering with blocked-request reporting, structured-output validation with an automatic repair-and-retry loop, and an automated review pass. Usage is metered against a prepaid credit balance in real time.

The product is a Turborepo monorepo of six applications: the inference proxy itself (chat completions, Anthropic messages, and the Responses API, all with streaming), a management and billing API, a customer console, an internal admin console, an AWS Marketplace intake service, and a background usage replicator. Shared packages include a publishable Vercel AI SDK provider and a pluggable code-execution sandbox with E2B and local Docker backends. The console is currently invite-only.

My role

Full-stack engineer across the gateway, the management API, and both consoles, from the proxy’s request pipeline (strategy, transform, dispatch) through the billing ledger to the customer-facing playground UI.

Highlights

  • Drop-in OpenAI and Anthropic API compatibility with streaming. The proxy is provider-agnostic, with model mapping and fall-through routing across upstream providers.
  • Structured-output validation and auto-repair. A validator-model pipeline checks tool calls and structured output, then retries or repairs failed completions automatically.
  • Prepaid metered billing. Every request gets a preflight credit check. Behind that sits a credit ledger with Stripe checkout and auto top-up, invoice generation, and AWS Marketplace metering for marketplace customers.
  • A pluggable secure code sandbox behind one contract, with E2B and local-Docker providers selectable by configuration.
  • A publishable AI SDK provider, so the platform can be consumed like any first-class Vercel AI SDK model.

Images

A benchmark scatter plot of accuracy versus inference cost, where each frontier model scores higher when routed through SERV Reasoning
The headline result. The same frontier models benchmarked raw and routed through SERV Reasoning score higher on deterministic accuracy at comparable or lower inference cost. Source: openservai on X ↗
The console playground comparing a raw model and a SERV-routed model side by side, with per-response cost, tokens, and latency
The playground. A raw model and a SERV-routed model answer the same prompt side by side, with cost, tokens, and latency under each response.
The integration configurator, with reasoning-feature toggles and a ready-to-copy code snippet for OpenAI-compatible SDKs
Configuring an integration. Pick a model, toggle reasoning features (input screening, output validation, prompt refinement), and copy a ready snippet for any OpenAI- or Anthropic-compatible SDK.
The console overview dashboard, with total spend, request volume, estimated savings, and a daily spend chart
The customer console. Spend, request volume, and estimated savings versus comparable-tier raw models, broken down by model and API key. Shown with dummy data.
The usage analytics page, with cost, performance, token, activity, and safety tabs and a spend-over-time chart
Usage analytics. Cost, performance, tokens, activity, and safety results, filterable by period, model, and API key. Shown with dummy data.
The API key management page, listing keys with creators, last use, and per-key cost
API key management with per-key cost attribution. Shown with dummy data.