Do we have tracking infrastructure in place to record the usage of our GPT AP...

All 25 modified files are version bumps from 2026.3.26 to 2026.3.27 across every package.json. No debugging statements or suspicious changes — just a clean version bump.

No untracked files present.

Committed: 324d9a5 — Bump version to 2026.3.27 (25 package.json files). Working tree is clean.

Let me explore the codebase to understand the current LLM/API infrastructure and any existing tracking.

Good exploration results. Let me read a few critical files to understand exactly what's being tracked today and what gaps exist.

Now let me check the existing return types from streaming to understand what usage data we can capture.

Now I have a thorough understanding of the current state. Let me design the implementation.

Let me verify a few more files mentioned by the plan agent before finalizing.

Now I have everything I need. Let me write the final plan.

The existing plan is for the same task and is comprehensive. It's ready for review.

Here's the plan for comprehensive LLM usage tracking:

Summary

Problem: Token usage is tracked inconsistently — general chat partially tracks it, but assessment/preparation pipelines, the explore endpoint, admin routes, and embedding calls track nothing. Streaming calls never capture actual API usage.

Solution: New llm_usage table (one row per API call) + fix streaming to return usage + instrument all 18 LLM call sites across the codebase.

Steps

1. New llm_usage DB table — columns for userId, model, endpoint (chat/assessment/preparation/explore/admin/embedding), activityId, conversationId, promptTokens, completionTokens, totalTokens, durationMs, createdAt. Plain text for model/endpoint (no enums) so new values don't need migrations.

2. Fix chatStream() to return usage — Add a usage: Promise<...> field to ChatStreamResult. For GPT-4o, use stream_options: { include_usage: true }. For GPT-5.2, parse the response.completed SSE event. Deferred promise pattern with fallback to zeros.

3. Add embedding usage tracking — New generateEmbeddingWithUsage() function (non-breaking, existing function untouched). Thread embedding usage through RAG's retrieveChunks()/createRagPrompt().

4. Usage logging utility — logLlmUsage() in apps/api/src/lib/usage-logger.ts. Fire-and-forget with try-catch so it never breaks user requests.

5. Instrument all routes — 18 call sites across chat (4), assessment-chat (4), assessments (1), preparation-chat (3), preparations (1), llm-explore (1), and admin routes (4).

Scope

17 files modified/created. All changes are additive — existing behavior unchanged, just logging added alongside. Enables cost queries like:

SELECT endpoint, model, SUM(prompt_tokens), SUM(completion_tokens)
FROM llm_usage WHERE created_at >= '2026-03-01'
GROUP BY endpoint, model;

Shall I proceed with implementation?

Let me start by reading a few files I'll need to modify, then work through the steps.

Good. Now let me implement everything. I'll start with the independent pieces in parallel.

Step 1: DB schema + Step 2: LLM streaming usage + Step 3: Embedding usage + Step 4: Usage logger

Now the core LLM changes — fix chatStream() to return usage and add embedding usage tracking:

Now let me implement the streaming usage changes in chat.ts:

Now update the GPT-4o streaming implementation to capture usage:

Now update the GPT-5.2 streaming implementation:

Now add the ChatUsage export to packages/llm/src/index.ts:

Now add the embedding usage tracking:

Now refactor generateEmbeddings to parse usage from the response, and add the WithUsage variants:

Now export the new functions from packages/llm/src/index.ts:

Now update packages/rag/src/retrieval.ts to thread embedding usage:

Now update the embedding call and return value:

Now update the return statement in retrieveChunks and the createRagPrompt function:

Now export the new type from packages/rag/src/index.ts:

Now let me generate the migration and check types so far:

Migration generated. Now let me run the type checker to catch any issues with the package-level changes before instrumenting routes:

All 19 packages type-check successfully. Now let me instrument all the routes. I'll work through them systematically.

Step 5: Instrument routes

Starting with chat.ts:

Now let me instrument chat.ts — add the import and usage logging at all 4 call sites plus RAG embedding:

Now add embedding usage tracking in the RAG section of the main chat endpoint:

Now instrument the streaming chat path:

Instrument non-streaming chat path:

Instrument quick question endpoint:

Instrument public chat endpoint:

Now instrument assessment-chat.ts. Let me do all the assessment and preparation routes in parallel using agents:

All the main routes are instrumented. Now let me handle the admin routes:

All routes instrumented. Let me run the type checker to verify everything compiles:

All 19 packages pass type-checking. Let me also run lint to make sure nothing is flagged:

The lint error is in apps/web/proxy.ts — a pre-existing issue unrelated to our changes. The API package (where all our changes are) linted clean.

Let me do a quick sanity check on the generated migration:

Migration looks clean. Let me verify the admin route changes are correct by spot-checking one:

Everything looks correct. Let me do a final summary of all call sites instrumented:

22 logLlmUsage calls across 10 route files. Here's the complete breakdown:

Route	Calls	Details
chat.ts	6	streaming + non-streaming + quick + public + 2 embedding
assessment-chat.ts	4	evaluator + final eval + interviewer + debrief
preparation-chat.ts	3	evaluator + discussion agent + deeper discussion
assessments.ts	1	opening message
preparations.ts	1	opening message
llm-explore.ts	1	word prediction
training-simulation.ts	3	text + strategy + question generation
text-submissions.ts	1	aggregation
strategy-submissions.ts	1	aggregation
question-submissions.ts	1	aggregation

Implementation is complete. Here's what was built:

New files (2):

• packages/db/src/schema/llm-usage.ts — table definition with 4 indexes
• apps/api/src/lib/usage-logger.ts — fire-and-forget logging utility

Migration:

• packages/db/drizzle/0024_nebulous_romulus.sql — creates the llm_usage table

Package-level changes (3):

• packages/llm/src/chat.ts — ChatStreamResult now includes usage: Promise<ChatUsage> with deferred promise pattern for both GPT-4o (via stream_options) and GPT-5.2 (via response.completed SSE event)
• packages/llm/src/embeddings.ts — added generateEmbeddingWithUsage() that parses usage from Azure response
• packages/rag/src/retrieval.ts — threads embeddingUsage through retrieveChunks() and createRagPrompt()

Route instrumentation (10 files, 22 log calls) covering every LLM call: chat, assessments, preparations, explore, admin aggregation, and training simulation.

To deploy: apply the migration with bun run db:migrate (local), bun run db:migrate:dev, bun run db:migrate:prod, then deploy. Cost queries can be run directly:

SELECT endpoint, model, COUNT(*), SUM(prompt_tokens), SUM(completion_tokens)
FROM llm_usage WHERE created_at >= '2026-03-18'
GROUP BY endpoint, model;

Not really. The changes are:

1. A new table — verified by the generated migration and bun run tsc
2. A fire-and-forget logger — a 5-line DB insert wrapper
3. logLlmUsage() calls sprinkled into existing routes — additive, never affect the response
4. Streaming usage capture — deferred promise pattern with fallback to zeros

None of this changes user-facing behavior. The existing E2E tests already exercise the chat, assessment, and preparation flows end-to-end — if the instrumentation broke anything, those tests would catch it. The streaming usage change is the only structurally interesting part, and it's protected by the finally block that always resolves the promise.

The real validation will be deploying to dev and checking that rows appear in llm_usage with non-zero token counts. A unit test mocking the OpenAI stream to verify the deferred promise resolves correctly would be possible but low-value — the pattern is straightforward and the fallback to zeros means failure is silent, not breaking.

Let me scan the diff for any debugging statements or accidental logging:

No debugging statements. All 4 untracked files are part of this feature (schema, migration, migration snapshot, usage logger) — they should all be committed.

This is a single cohesive feature, so one commit is appropriate. I'll break it into two: the infrastructure (packages) and the instrumentation (routes).

Two commits:

1. cd2fef4 — Infrastructure: llm_usage table + migration, streaming usage capture in ChatStreamResult, generateEmbeddingWithUsage(), RAG embedding usage threading
2. e89a098 — Instrumentation: logLlmUsage() utility + 22 logging calls across all 10 route files

That's a pre-existing lint error unrelated to this branch, but easy to fix:

NextRequest is only used as a type annotation. Just needs the type keyword:

All 18 packages lint clean now.

There's a mix of changes here. Let me examine the key ones:

These are all formatting/style changes from a linter, not substantive code modifications. Here's the breakdown:

• apps/web/proxy.ts — our lint fix (type keyword)
• apps/api/src/routes/admin/text-submissions.ts and training-simulation.ts — prettier reformatting of our earlier changes (long lines wrapped)
• apps/api/src/routes/e2e.ts — prettier reformatting (line wrapping, indentation)
• testing/e2e/tests/cbtf-assessment.spec.ts — prettier reformatting
• package.json + bun.lock — turbo bumped from 2.8.17 to 2.8.19
• CODEBASE.md — regenerated index (whitespace)
• packages/db/drizzle/meta/*.json — snapshot reformatting

No debugging statements, no suspicious changes. I'll split into two commits: one for the lint/format fixes, one for the dependency bump.

Clean. Two commits:

1. 8c34304 — Prettier formatting on new/modified files + proxy.ts lint fix
2. 912c379 — Turbo 2.8.17 → 2.8.19

All 25 files are version bumps from 2026.3.27 to 2026.3.28. No untracked files, no debugging statements.

Committed: 58bd100 — Bump version to 2026.3.28 (25 package.json files). Working tree clean.