/ Context
An AI Gateway sits on the request path of every application that uses it, so its behaviour under sustained load is a platform concern rather than an application one. The question was how the gateway path itself behaves when driven hard — separate from any question about which model or framework an application should use.
/ Role & decision scope
- Designed and ran the synthetic load test against the AI Gateway.
- Built the test harness that generated the request load.
- Kept qualitative framework evaluation separate from the gateway load test.
/ Constraints
- Synthetic traffic only — this was a controlled test, not observation of production usage.
- The test exercises the gateway request path, not end-user application behaviour.
- Frameworks and platforms differ in purpose, so they are not directly comparable as performance targets.
/ Architecture & key decisions
- D01
Test the gateway, not the vendors
The load test targets the AI Gateway request path. It is deliberately not a head-to-head performance ranking of OpenAI, Amazon Bedrock, LiteLLM, AgentKit, LangChain and AWS Strands — those serve different roles and are not equivalent targets.
- D02
Framework evaluation kept qualitative and separate
Those frameworks and platforms were assessed on architecture fit, integration surface and operability as a separate exercise, feeding the platform ADRs rather than a latency table.
- D03
Synthetic load, stated as synthetic
More than 69,000 requests were generated by the harness during the test. That is a property of the test run, not evidence of adoption, enterprise scale or production traffic.
/ Trade-offs
- Synthetic load versus real traffic realism
- A synthetic harness gives a repeatable, controllable signal about the gateway path; it does not substitute for production observability, and is not presented as such.
- A single comparable number versus honest comparison
- Non-equivalent platforms were compared qualitatively on architecture fit instead of being forced onto one benchmark axis.
/ Evidence & outcomes
- Artifact
- Load-test harness generating 69,000+ synthetic requests against the AI Gateway
- Decision
- Qualitative architecture evaluation of OpenAI, Amazon Bedrock, LiteLLM, AgentKit, LangChain and AWS Strands, recorded separately from load testing