feat(ai-openrouter): capture per-request cost from chat responses#469
Open
season179 wants to merge 2 commits intoTanStack:mainfrom
Open
feat(ai-openrouter): capture per-request cost from chat responses#469season179 wants to merge 2 commits intoTanStack:mainfrom
season179 wants to merge 2 commits intoTanStack:mainfrom
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds OpenRouter per-request USD cost capture by cloning/parsing SSE chat completions in an HTTPClient response hook, buffering results in a CostStore, and attaching optional Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant Adapter as OpenRouter Adapter
participant Hook as HTTPClient Hook
participant SSE as SSE Stream
participant CostStore as CostStore
participant App
Client->>Adapter: chatStream(request)
Adapter->>Hook: construct client with cost-capture hook
Adapter->>SSE: open chat completion stream (HTTP response)
SSE->>Hook: HTTP Response (text/event-stream)
Hook->>Hook: clone Response body and parse clone (out-of-band)
Hook->>CostStore: store extracted usage.cost & usage.cost_details for response id
par streaming
SSE->>Adapter: yield content/token usage chunks
Adapter->>Adapter: accumulate finalUsage and finishReason
and
Hook->>CostStore: store parsed cost when found
end
SSE->>Adapter: trailing usage-only chunk (finalUsage)
Adapter->>CostStore: take(responseId)
CostStore-->>Adapter: {cost, costDetails} | undefined
Adapter->>Adapter: buildRunFinishedUsage(finalUsage, costInfo)
Adapter->>App: emit RUN_FINISHED with usage (+ cost when present)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
packages/typescript/ai/src/types.ts (1)
796-806:costDetailscomment says "loosely typed" but the shape is actually locked to OpenRouter's two fields.The JSDoc argues the type must be loose to accommodate provider divergence (BYOK upstream, cache discounts, per-tier rates, ...), but
costDetailsonly declaresupstreamInferenceCostandcacheDiscount. Any future adapter that reports, say, a tier-specific rate would be forced to either omit it oras any-cast — exactly what the comment says should be avoided.If the intent is genuinely "loose", consider adding an index signature so additional provider-specific keys are type-legal without a cast. Otherwise, tighten the comment to match the actual (narrow, OpenRouter-shaped) contract.
♻️ Suggested adjustment
costDetails?: { upstreamInferenceCost?: number | null cacheDiscount?: number | null + [key: string]: number | null | undefined }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai/src/types.ts` around lines 796 - 806, The JSDoc promises a "loosely typed" costDetails but the declared shape only allows upstreamInferenceCost and cacheDiscount; update the type for costDetails (in packages/typescript/ai/src/types.ts, symbol: costDetails) to include an index signature (or use Record<string, number | null>) while keeping the existing named keys so provider-specific numeric fields are allowed without casting; this ensures adapters can add tier/rate/BYOK fields legally without changing the comment.packages/typescript/ai-openrouter/src/adapters/cost-capture.ts (1)
161-167: Minor: content-type check is case-sensitive.
content-typeheader values are case-insensitive per RFC; OpenRouter today uses lowercasetext/event-stream, but a proxy on the path could legitimately return e.g.Text/Event-Streamand cost capture would silently skip. Cheap to harden:♻️ Proposed tweak
- const contentType = res.headers.get('content-type') ?? '' + const contentType = (res.headers.get('content-type') ?? '').toLowerCase() // Cost capture is only wired for streaming chat completions. Non-SSE // responses on `/chat/completions` (e.g. `structuredOutput()` which // calls `chat.send({ stream: false })`) never consume `costStore` — // skipping them here avoids cloning the response and second-parsing // potentially large JSON bodies for no downstream consumer. if (!contentType.includes('text/event-stream')) return🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts` around lines 161 - 167, The content-type check in the cost capture branch is case-sensitive and may miss valid SSE responses; update the check around the contentType variable so it compares case-insensitively (e.g. normalize contentType with toLowerCase() or use a case-insensitive regex) before testing for 'text/event-stream' and keep the early return behavior unchanged to avoid extra parsing.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/typescript/ai-openrouter/tests/openrouter-adapter.test.ts`:
- Around line 6-19: Merge the two separate type imports from
'../src/adapters/text' into one line (exported types OpenRouterTextAdapter and
OpenRouterTextModelOptions) to remove import/no-duplicates, and replace the
typed vi.importActual call in the vi.mock helper (currently using await
vi.importActual<typeof import('@openrouter/sdk')>('@openrouter/sdk')) with an
untyped runtime import (await vi.importActual('@openrouter/sdk')) or cast the
result to any so you avoid the banned import() type annotation; keep the other
type imports (CostStore, StreamChunk, Tool) and the module-scope mockSend
variable unchanged.
---
Nitpick comments:
In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts`:
- Around line 161-167: The content-type check in the cost capture branch is
case-sensitive and may miss valid SSE responses; update the check around the
contentType variable so it compares case-insensitively (e.g. normalize
contentType with toLowerCase() or use a case-insensitive regex) before testing
for 'text/event-stream' and keep the early return behavior unchanged to avoid
extra parsing.
In `@packages/typescript/ai/src/types.ts`:
- Around line 796-806: The JSDoc promises a "loosely typed" costDetails but the
declared shape only allows upstreamInferenceCost and cacheDiscount; update the
type for costDetails (in packages/typescript/ai/src/types.ts, symbol:
costDetails) to include an index signature (or use Record<string, number |
null>) while keeping the existing named keys so provider-specific numeric fields
are allowed without casting; this ensures adapters can add tier/rate/BYOK fields
legally without changing the comment.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 90bbcd91-81f2-44c8-9463-74a171a180c3
📒 Files selected for processing (9)
.changeset/openrouter-cost-tracking.mdREADME.mddocs/adapters/openrouter.mdpackages/typescript/ai-openrouter/src/adapters/cost-capture.tspackages/typescript/ai-openrouter/src/adapters/text.tspackages/typescript/ai-openrouter/tests/cost-capture.test.tspackages/typescript/ai-openrouter/tests/openrouter-adapter.test.tspackages/typescript/ai/src/activities/chat/middleware/types.tspackages/typescript/ai/src/types.ts
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 18, 2026
Merge the two separate type imports from '../src/adapters/text' into
one statement (import/no-duplicates) and replace the inline `typeof
import('@openrouter/sdk')` type annotation with a top-level
`import type * as OpenRouterSDK` (@typescript-eslint/consistent-type-imports).
Picked up in CodeRabbit review of TanStack#469.
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 18, 2026
Content-Type header values are case-insensitive per RFC 9110. OpenRouter today serves lowercase `text/event-stream` but a proxy on the path could return a different casing, which would make cost capture silently skip a real SSE response. Lowercase the header before the substring match. Picked up in CodeRabbit review of TanStack#469.
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 18, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 20, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
ab571ba to
7a0d1ff
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts`:
- Around line 64-90: The fallback in take(id) uses this.pendingParses which
still contains parses that have already announced a different id, causing
head-of-line blocking; fix by removing the parse from the pending set when it
announces an id. In announceId(id, parse) call this.pendingParses.delete(parse)
(e.g., right after creating the ParseEntry / this.idToParse.set) so announced
parses are no longer awaited by the Promise.allSettled fallback; apply the same
removal change to the other announce-path (the similar block referenced at lines
~115-124) so all announce flows remove their parse from pendingParses.
In `@packages/typescript/ai-openrouter/src/adapters/text.ts`:
- Around line 110-111: The adapter narrows finish reasons and drops
'content_filter'; change the declared finalFinishReason variable type and the
Generator return type to use RunFinishedEvent['finishReason'] instead of the
union 'stop'|'length'|'tool_calls', and update the mapping logic that maps
OpenRouter finish reasons to RunFinishedEvent to include an explicit case for
'content_filter' (preserving that value rather than converting it to 'stop'),
ensuring finalFinishReason is assigned the mapped 'content_filter' where
appropriate.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e23b2f7b-821f-4b1d-97a8-f2262c11a3c4
📒 Files selected for processing (9)
.changeset/openrouter-cost-tracking.mdREADME.mddocs/adapters/openrouter.mdpackages/typescript/ai-openrouter/src/adapters/cost-capture.tspackages/typescript/ai-openrouter/src/adapters/text.tspackages/typescript/ai-openrouter/tests/cost-capture.test.tspackages/typescript/ai-openrouter/tests/openrouter-adapter.test.tspackages/typescript/ai/src/activities/chat/middleware/types.tspackages/typescript/ai/src/types.ts
✅ Files skipped from review due to trivial changes (2)
- README.md
- docs/adapters/openrouter.md
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/typescript/ai/src/types.ts
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 20, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
7a0d1ff to
e98120e
Compare
season179
added a commit
to season179/ai
that referenced
this pull request
Apr 20, 2026
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
e98120e to
e6b1aab
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts`:
- Around line 326-344: extractCostFromUsage currently allows returning
costDetails when usage.cost is missing; change it so costDetails are only
included when the authoritative total cost (usage.cost) is present. Update
extractCostFromUsage to treat undefined cost as a reason to return undefined (or
at minimum only attach costDetails when cost !== undefined), so that the
returned object never contains costDetails without a numeric cost; refer to
extractCostFromUsage, the local variables cost/details/upstream/cacheDiscount,
and helper pickNumberOrNull to implement this conditional.
- Around line 219-223: The isChatCompletionsRequest function currently tests the
full URL string which can match query params like "?next=/chat/completions";
instead parse the url into a URL object and run the regex (or a pathname
comparison) against the URL.pathname only (i.e., use new URL(url).pathname and
then test that string) so the hook only triggers for actual /chat/completions
pathnames; update isChatCompletionsRequest to use URL.pathname for the check.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 9dae5fc2-0ed4-4a5c-b2d3-4c751d20543d
📒 Files selected for processing (9)
.changeset/openrouter-cost-tracking.mdREADME.mddocs/adapters/openrouter.mdpackages/typescript/ai-openrouter/src/adapters/cost-capture.tspackages/typescript/ai-openrouter/src/adapters/text.tspackages/typescript/ai-openrouter/tests/cost-capture.test.tspackages/typescript/ai-openrouter/tests/openrouter-adapter.test.tspackages/typescript/ai/src/activities/chat/middleware/types.tspackages/typescript/ai/src/types.ts
✅ Files skipped from review due to trivial changes (2)
- README.md
- docs/adapters/openrouter.md
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/typescript/ai-openrouter/tests/openrouter-adapter.test.ts
- packages/typescript/ai-openrouter/src/adapters/text.ts
Comment on lines
+219
to
+223
| function isChatCompletionsRequest(url: string): boolean { | ||
| // Match path segment to avoid false positives on hosts whose name happens | ||
| // to end in "/chat/completions". The SDK always sends absolute URLs. | ||
| return /\/chat\/completions(?:[/?#]|$)/.test(url) | ||
| } |
Contributor
There was a problem hiding this comment.
Match /chat/completions against the URL pathname only.
Line 222 runs the regex over the full URL, so an unrelated SSE request like /generation?next=/chat/completions would be treated as a chat-completions response. Parse the URL and check pathname to keep the hook scoped to chat responses.
🐛 Proposed fix
function isChatCompletionsRequest(url: string): boolean {
// Match path segment to avoid false positives on hosts whose name happens
// to end in "/chat/completions". The SDK always sends absolute URLs.
- return /\/chat\/completions(?:[/?#]|$)/.test(url)
+ return /\/chat\/completions(?:\/|$)/.test(new URL(url).pathname)
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts` around lines
219 - 223, The isChatCompletionsRequest function currently tests the full URL
string which can match query params like "?next=/chat/completions"; instead
parse the url into a URL object and run the regex (or a pathname comparison)
against the URL.pathname only (i.e., use new URL(url).pathname and then test
that string) so the hook only triggers for actual /chat/completions pathnames;
update isChatCompletionsRequest to use URL.pathname for the check.
Comment on lines
+326
to
+344
| function extractCostFromUsage( | ||
| usage: Record<string, unknown>, | ||
| ): CostInfo | undefined { | ||
| const cost = typeof usage.cost === 'number' ? usage.cost : undefined | ||
| const details = usage.cost_details as Record<string, unknown> | undefined | ||
| const upstream = pickNumberOrNull(details, 'upstream_inference_cost') | ||
| const cacheDiscount = pickNumberOrNull(details, 'cache_discount') | ||
| const hasDetails = upstream !== undefined || cacheDiscount !== undefined | ||
| if (cost === undefined && !hasDetails) return undefined | ||
| return { | ||
| ...(cost !== undefined && { cost }), | ||
| ...(hasDetails && { | ||
| costDetails: { | ||
| ...(upstream !== undefined && { upstreamInferenceCost: upstream }), | ||
| ...(cacheDiscount !== undefined && { cacheDiscount }), | ||
| }, | ||
| }), | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Do not store cost details without the authoritative total cost.
extractCostFromUsage() currently returns { costDetails } when usage.cost is missing but usage.cost_details is present. That can surface a billing breakdown without the provider’s total usage.cost, which contradicts the PR’s “authoritative per-request USD cost” contract.
🐛 Proposed fix
function extractCostFromUsage(
usage: Record<string, unknown>,
): CostInfo | undefined {
const cost = typeof usage.cost === 'number' ? usage.cost : undefined
+ if (cost === undefined) return undefined
+
const details = usage.cost_details as Record<string, unknown> | undefined
const upstream = pickNumberOrNull(details, 'upstream_inference_cost')
const cacheDiscount = pickNumberOrNull(details, 'cache_discount')
const hasDetails = upstream !== undefined || cacheDiscount !== undefined
- if (cost === undefined && !hasDetails) return undefined
return {
- ...(cost !== undefined && { cost }),
+ cost,
...(hasDetails && {
costDetails: {
...(upstream !== undefined && { upstreamInferenceCost: upstream }),
...(cacheDiscount !== undefined && { cacheDiscount }),🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/typescript/ai-openrouter/src/adapters/cost-capture.ts` around lines
326 - 344, extractCostFromUsage currently allows returning costDetails when
usage.cost is missing; change it so costDetails are only included when the
authoritative total cost (usage.cost) is present. Update extractCostFromUsage to
treat undefined cost as a reason to return undefined (or at minimum only attach
costDetails when cost !== undefined), so that the returned object never contains
costDetails without a numeric cost; refer to extractCostFromUsage, the local
variables cost/details/upstream/cacheDiscount, and helper pickNumberOrNull to
implement this conditional.
Capture OpenRouter's per-request cost from the SSE chat response and surface it on UsageTotals.costDetails so callers can track spend. Implementation notes: - Cost capture is isolated per-request so concurrent chats don't cross-contaminate; SSE parsing matches separators directly (handles LF and CRLF) and matches the content-type case-insensitively. - Skipped on non-streaming chat responses to avoid needless overhead. - Omits usage entirely when token counts are missing, even if a cost was captured, so downstream consumers never see partial usage. - Does not downgrade a RUN_FINISHED event to RUN_ERROR on a late stream abort; the deferral trade-off is documented in the adapter. - Flushes the trailing SSE frame and resolves no-cost takes fast.
…etails The JSDoc promised "loosely typed" costDetails to accommodate provider divergence (BYOK upstream costs, cache discounts, per-tier rates) but the declared shape only allowed two OpenRouter-specific keys, so any other adapter would have been forced to `as any`-cast to report its own breakdown. Add a numeric index signature so additional keys are type-legal without a cast, matching the documented intent. Picked up in CodeRabbit review of TanStack#469.
e6b1aab to
3d50ef4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Changes
Surface OpenRouter's authoritative per-request USD cost on
RUN_FINISHED. OpenRouter returnsusage.costandusage.cost_detailsinline in every chat response (docs), but the@openrouter/sdkZod parser strips those fields because the SDK schema doesn't declare them. Cost also can't be reconstructed locally (different upstream routes → different prices, plus cache discounts the SDK can't see).Closes #468
How
@tanstack/ai-openrouterattaches a hook on the SDK's publicHTTPClient(addHook('response', …)). The hook callsResponse.clone()to tee the body and parses the clone out-of-band to pullcost/cost_detailsbefore Zod strips them. The SDK's stream consumer reads the other branch untouched — no extra HTTP request, no added latency.text/event-streamchat responses, so structured-output and non-streaming paths aren't cloned.OpenRouterTextAdapterreads the captured cost when the stream ends and emits it onRUN_FINISHED.usage.{cost, costDetails}.RUN_FINISHEDis deferred until the upstream stream fully drains so the trailing usage-only chunk (emptychoices) is included inusage.httpClientis preserved: the adapter clones the caller's client (inheriting their fetcher, retries, tracing, and any pre-registered hooks) and appends cost capture to the clone. The caller's instance is never mutated.Types (
@tanstack/ai, additive + backwards-compatible)UsageTotalstype with optionalcostandcostDetails(upstreamInferenceCost,cacheDiscount).RunFinishedEvent.usage, middlewareUsageInfo(consumed byonUsage), andFinishInfo.usage(consumed byonFinish) all reuseUsageTotalsso they can't drift. No required field changes; adapters that don't populate cost keep working without modification.Correctness
The out-of-band parse runs concurrently with the main stream, so the implementation handles:
take(id)— one slow stream doesn't block another'sRUN_FINISHED.\n\n,\r\n\r\n, and\r\revent separators including when\r/\nstraddle read-chunk boundaries, plus EOF-terminated frames with no trailing separator.finishReasonis captured does not downgrade the run toRUN_ERROR.take(id)fast-paths toundefinedas soon as the matching parse settles without recording cost; cost-less providers or non-cost responses aren't penalized.Docs + changeset
docs/adapters/openrouter.md.@tanstack/ai-openrouterand@tanstack/aias minor.Tests
cost-capture.test.ts(19 new tests): SSE parse w/ and w/o details, non-streaming skip, CRLF separators, EOF flush, preceding-hook body-disturbed path, per-id isolation, race regressions.openrouter-adapter.test.ts(+7 tests): basic attach, cost w/o details, trailing usage-only chunk, consume-once, customhttpClientpreservation, late-abort still emitsRUN_FINISHED, zero-token not fabricated.@tanstack/ai-openrouterunit tests green; full PR suite (lint, types, build, tests, docs, knip, sherif) green across 40 projects.✅ Checklist
pnpm run test:pr.🚀 Release Impact
Summary by CodeRabbit
New Features
Documentation
Tests
API / Types