I upgrade NodeJS to 24.13.0. Can we reinstall the Bun dependencies?

You need to use the API version provided by the endpoint.

It should not fall back to that endpoint. It should use that endpoint.

Please continue. But do we need this SDK?

No. Please write a simpler test in that package that uses the defined endpoint and ensure that it can perform completions. I'm tired of testing your broken code through the frontend.

Why are you proceeding when critical environment variables are not defined? That value is REQUIRED! You should use Zod to make sure that it is exists and is the proper type.

AZURE_OPENAI_API_KEY: [not set]. Why are you still using the wrong key?

Oh cmon. Please? Are you sure? The SDK really can't handle this? I'm skeptical.

Better! Our first successful completion! Is RAG working properly? When I asked when the course meets it says Tuesdays from 2–2:50 PM, which is incorrect.

Why is it making up responses when it doesn't know?

RAG is supposed to be done when we start the server.

Yes. We agreed to do that previously.

This session is being continued from a previous conversation that ran out of context. The summary below covers the earlier portion of the conversation.

Analysis: Let me chronologically analyze this conversation, which continues from a previous session about Better Auth migration and E2E testing setup.

Starting Context:

• Previous session was working on Better Auth migration from NextAuth v5 beta
• Auth middleware was rewritten to use auth.api.getSession()
• There was an import error for @repo/auth in the API
• The plan file indicates E2E testing setup with Playwright

Key Events in This Session:

1. Fixed @repo/auth dependency - Added to apps/api/package.json

2. Auth working but 400 errors - Discovered validation issue with conversationId: null vs undefined

   • User was frustrated about debugging time
   • Fixed by changing schema to use .nullish() instead of .optional()

3. Azure OpenAI API issues - Multiple problems:

   • Wrong endpoint URL being constructed by SDK
   • API version not being passed correctly
   • Wrong API key variable name (AZURE_OPENAI_API_KEY vs AZURE_OPENAI_CHAT_API_KEY)
   • User explicitly criticized: "Why are you proceeding when critical environment variables are not defined?"
   • User demanded: "You should use Zod to make sure that it exists"
   • Added env.ts with Zod validation

4. SDK URL construction issues:

   • Azure SDK builds URLs like {baseURL}/v1{path} which doesn't match Azure's endpoint format
   • User asked: "Are you sure? The SDK really can't handle this? I'm skeptical."
   • Eventually got it working with OpenAI SDK + .chat() method + custom fetch for api-version

5. RAG not working - Database had 0 documents

   • User said "RAG is supposed to be done when we start the server"
   • User confirmed: "Yes. We agreed to do that previously."
   • Created startup ingestion script but hit glob import error

User Frustrations/Feedback:

• "I'm getting really frustrated. Why is this taking so long?"
• "I'm so tired of debugging this. It should not be this hard."
• "We've wasted hours over these silly mistakes because you did not add enough validation."
• "Why are you still using the wrong key?" (about AZURE_OPENAI_API_KEY vs AZURE_OPENAI_CHAT_API_KEY)
• "Please use a package to get the Git root" (about brittle relative paths)
• "No. Don't hardcode to a path relative to my environment!"
• "Please write a simpler test in that package" (instead of testing through frontend)

Current State:

• Auth is working
• Chat completions working via OpenAI SDK with Azure
• RAG ingestion script created but failing with glob import error
• Server keeps restarting but not picking up new deps

Summary:

1. Primary Request and Intent:

   • Complete Better Auth migration from NextAuth v5 beta
   • Fix cross-subdomain cookie authentication
   • Get chat API working with Azure OpenAI
   • Set up RAG ingestion to run automatically on server startup
   • Add proper Zod validation for environment variables
   • Use git-root-dir package for finding content directory (not brittle relative paths)

2. Key Technical Concepts:

   • Better Auth with auth.api.getSession() for session validation
   • Azure OpenAI chat completions API (not the newer Responses API)
   • Zod schema validation with .nullish() for accepting both null and undefined
   • OpenAI SDK's .chat() method vs default () which uses Responses API
   • Custom fetch wrapper to append api-version query parameter for Azure
   • RAG document ingestion on server startup
   • git-root-dir package for reliably finding monorepo root

3. Files and Code Sections:

   • packages/llm/src/env.ts (created) - Zod validation for LLM env vars:

     import { z } from "zod";
     
     const envSchema = z.object({
       AZURE_OPENAI_CHAT_API_KEY: z.string().min(1, "AZURE_OPENAI_CHAT_API_KEY is required"),
       AZURE_OPENAI_CHAT_ENDPOINT: z.string().url("AZURE_OPENAI_CHAT_ENDPOINT must be a valid URL"),
       AZURE_OPENAI_EMBEDDINGS_API_KEY: z.string().min(1, "AZURE_OPENAI_EMBEDDINGS_API_KEY is required"),
       AZURE_OPENAI_EMBEDDINGS_ENDPOINT: z.string().url("AZURE_OPENAI_EMBEDDINGS_ENDPOINT must be a valid URL"),
     });
     
     function validateEnv() {
       const result = envSchema.safeParse(process.env);
       if (!result.success) {
         const errors = result.error.flatten().fieldErrors;
         const errorMessages = Object.entries(errors)
           .map(([key, msgs]) => `  ${key}: ${msgs?.join(", ")}`)
           .join("\n");
         throw new Error(
           `Missing or invalid LLM environment variables:\n${errorMessages}\n\n` +
             `Make sure these are defined in [local-hostname]`
         );
       }
       return result.data;
     }
     
     export const env = validateEnv();
     

   • packages/llm/src/provider.ts (modified) - Uses OpenAI SDK with Azure:

     import { createOpenAI } from "@ai-sdk/openai";
     import { createAzure } from "@ai-sdk/azure";
     import { env } from "./env";
     
     let _azureChat: ReturnType<typeof createOpenAI> | null = null;
     let _azureEmbeddings: ReturnType<typeof createAzure> | null = null;
     
     export function getAzureChatProvider() {
       if (!_azureChat) {
         const { baseURL, apiVersion } = parseEndpoint(env.AZURE_OPENAI_CHAT_ENDPOINT);
         _azureChat = createOpenAI({
           [api-key],
           baseURL,
           headers: {
             "api-key": env.AZURE_OPENAI_CHAT_API_KEY,
           },
           fetch: async (url, options) => {
             const urlWithVersion = `${url}${url.includes("?") ? "&" : "?"}api-version=${apiVersion}`;
             return fetch(urlWithVersion, options);
           },
         });
       }
       return _azureChat;
     }
     
     function parseEndpoint(endpoint: string): { baseURL: string; apiVersion: string } {
       const url = new URL(endpoint);
       const apiVersion = url.searchParams.get("api-version") || "2024-02-01";
       const basePath = url.pathname.replace(/\/chat\/completions$/, "").replace(/\/embeddings$/, "");
       return {
         baseURL: `${url.origin}${basePath}`,
         apiVersion,
       };
     }
     

   • packages/llm/src/chat.ts (modified) - Uses .chat() method:

     model: azure.chat(model),  // Changed from azure(model) to use chat API not responses API
     

   • apps/api/src/routes/chat.ts (modified) - Zod schema with nullish:

     const chatRequestSchema = z.object({
       conversationId: z.string().uuid().nullish(), // nullish = optional + nullable
       message: z.string().min(1),
       context: z.string().nullish(),
       agentSlug: z.string().nullish(),
       semester: z.string().nullish(),
       useRag: z.boolean().nullish(),
       systemPrompt: z.string().nullish(),
       streaming: z.boolean().nullish().default(true),
     });
     

   • packages/agents/src/loader.ts (modified) - Uses git-root-dir:

     import gitRootDir from "git-root-dir";
     
     async function findContentDir(): Promise<string> {
       if (cachedContentDir) return cachedContentDir;
       const root = await gitRootDir();
       if (root) {
         cachedContentDir = resolve(root, "content");
         return cachedContentDir;
       }
       cachedContentDir = resolve(process.cwd(), "content");
       return cachedContentDir;
     }
     

   • apps/api/src/startup/ingest-content.ts (created) - Startup content indexing:

     import { glob } from "glob";
     import { readFileSync } from "fs";
     import { basename } from "path";
     import gitRootDir from "git-root-dir";
     import { db } from "@repo/db";
     import { documents } from "@repo/db/schema";
     import { ingestDocument } from "@repo/rag";
     import { eq } from "drizzle-orm";
     
     export async function ingestContent(): Promise<void> {
       const root = await gitRootDir();
       if (!root) {
         console.error("[ingest] Could not find git root");
         return;
       }
       const contentDir = `${root}/content`;
       const pattern = `${contentDir}/**/*.mdx`;
       const files = await glob(pattern);
       console.log(`[ingest] Found ${files.length} MDX files to index`);
       for (const file of files) {
         // ... indexing logic
       }
     }
     

   • apps/api/src/index.ts (modified) - Calls ingestContent on startup:

     import { ingestContent } from "./startup/ingest-content";
     // ...
     await checkPort();
     ingestContent().catch((err) => {
       console.error("[startup] Content indexing failed:", err);
     });
     

4. Errors and fixes:

   • 401 auth errors: Fixed by rewriting middleware to use auth.api.getSession()
   • 400 validation errors (conversationId null vs undefined): Fixed by using .nullish() in Zod schema
   • Azure API "version not supported": SDK was calling wrong endpoint; fixed by using OpenAI SDK with custom fetch
   • Azure API "Resource not found": SDK was using /v1/responses instead of chat completions; fixed by using .chat() method
   • Wrong API key variable: User pointed out AZURE_OPENAI_API_KEY vs AZURE_OPENAI_CHAT_API_KEY; fixed env.ts to use correct name
   • Brittle relative paths: User rejected CONTENT_DIR="../../content"; fixed by using git-root-dir package
   • LLM making up answers: RAG database was empty (0 documents); added startup ingestion
   • glob import error: Currently failing with Cannot find package 'glob' despite being installed

5. Problem Solving:

   • Auth migration complete and working
   • Azure OpenAI chat completions working via test script
   • Agent loading working via git-root-dir
   • RAG ingestion script created but not yet working (glob import issue)

6. All user messages:

   • "Nope. Still failing."
   • "Still broken."
   • "Still broken. STILL BROKEN."
   • "I'm getting really frustrated. Why is this taking so long? We've been working on this stupid problem for a long time."
   • "Just did." (multiple times for testing)
   • "Check the logs."
   • "Again." (testing)
   • "I'm so tired of debugging this. It should not be this hard."
   • "Why are you using a brittle relative path?"
   • "No. Don't hardcode to a path relative to my environment! That breaks if someone else uses it. Terrible. Also the frontend IS STILL BROKEN."
   • "Please use a package to get the Git root in development to set the CONTENT_DIR."
   • "No. USE A NPM PACKAGE."
   • "You're forcing me to restart the server constantly. PUT IN A BUNCH OF LOGGING SO YOU CAN SEE EXACTLY WHAT IS GOING WRONG."
   • "OK. Now it doesn't fail, but also doesn't return any response." (with WebSocket error logs)
   • "No. Please write a simpler test in that package that uses the defined endpoint and ensure that it can perform completions. I'm tired of testing your broken code through the frontend."
   • "Why are you proceeding when critical environment variables are not defined? That value is REQUIRED! You should use Zod to make sure that it is exists and is the proper type."
   • "We've wasted hours over these silly mistakes because you did not add enough validation."
   • "AZURE_OPENAI_API_KEY: [not set]. Why are you still using the wrong key?"
   • "Oh cmon. Please? Are you sure? The SDK really can't handle this? I'm skeptical."
   • "You need to use the API version provided by the endpoint."
   • "It should not fall back to that endpoint. It should use that endpoint."
   • "Fall back from what?"
   • "Better! Our first successful completion! Is RAG working properly? When I asked when the course meets it says Tuesdays from 2–2:50 PM, which is incorrect."
   • "Why is it making up responses when it doesn't know?"
   • "RAG is supposed to be done when we start the server."
   • "Yes. We agreed to do that previously."

7. Pending Tasks:

   • Fix glob import error in ingest-content.ts
   • Verify RAG ingestion works on startup
   • Test that chat queries return correct RAG-backed responses

8. Current Work: Adding automatic content indexing on API startup. Created apps/api/src/startup/ingest-content.ts and modified apps/api/src/index.ts to call it. Added git-root-dir and glob dependencies. Currently failing with:

   error: Cannot find package 'glob' from '~/www/usingandunderstanding.ai/apps/api/src/startup/ingest-content.ts'
   

   The package was installed (bun add glob --cwd apps/api) and exists in node_modules, but bun isn't resolving it.

9. Optional Next Step: Fix the glob import error. The user said "RAG is supposed to be done when we start the server" and "Yes. We agreed to do that previously." The immediate task is to get the ingest-content.ts script working so documents are indexed on startup. May need to restart the server or use a different approach to import glob.

If you need specific details from before compaction (like exact code snippets, error messages, or content you generated), read the full transcript at: ~/.claude/projects/-Users-challen-www-usingandunderstanding-ai/ffe8f045-e4f9-4bb4-bc05-627c7ff7b7cd.jsonl Please continue the conversation from where we left it off without asking the user any further questions. Continue with the last task that you were asked to work on.

Read the output file to retrieve the result: /tmp/claude/-Users-challen-www-usingandunderstanding-ai/tasks/b829e3f.output

Why are we using the AI SDK? What does it get us?

Yes.

Here's what the agent in cs124.org did: I used the OpenAI Node.js SDK (openai package version 4.103.0) configured to work with Azure OpenAI endpoints.

Here's the relevant code from servers/api/server/chat.ts:

import OpenAI from "openai"

// Create OpenAI client for chat function createChatClient(): OpenAI | undefined { if (!AZURE_OPENAI_CHAT_ENDPOINT || !AZURE_OPENAI_CHAT_API_KEY) { return undefined }

return new OpenAI({
  [api-key],
  baseURL: AZURE_OPENAI_CHAT_ENDPOINT,
  defaultQuery: { "api-version": "2024-02-01" },
  defaultHeaders: { "api-key": AZURE_OPENAI_CHAT_API_KEY },
})

}

The OpenAI SDK is configured with Azure-specific settings:

• baseURL points to the Azure OpenAI endpoint
• defaultQuery includes the Azure API version
• defaultHeaders includes the Azure API key

This approach works because Azure OpenAI's REST API is compatible with the OpenAI SDK - you just need to configure the base URL and authentication headers differently.

OK! Better. It's at least working now.

Read the output file to retrieve the result: /tmp/claude/-Users-challen-www-usingandunderstanding-ai/tasks/b6d18c0.output /commitall

Your task

First, run git status, git diff, and git log --oneline -10 to understand the current state.

Then add and commit all modified files with a descriptive message. Break the changes into multiple commits if appropriate. If there are untracked files, ask if they should be added or added to the appropriate .gitignore file. The user may also want to take over at this point, so provide an abort option as well. Always check changes for unnecessary debugging statements or logging that might have been accidentally included. Prompt the user if you find any, and offer to remove each one by one. Do not use interactive git staging (git add -p) as it does not work.

OK. Now, can we get the e2e test to pass? At this point we may need to adjust the e2e authentication approach. Start the backend in e2e mode and then run the test.

Look, this WORKS when run from the browser. So at this point you're debugging the TEST CONFIGURATION.

You should set cookies the same way that Better Auth does.

No no no no no. This is totally wrong. Check ../cs124.org/sites/cs124 again.

No, just mimic the cs124.org configuration. Exactly. Then explain to me what is not working and why. You're just trying random things hoping anything works.

THEY WORK. Why would I ask you to copy failing tests?

No. Now you're just making things up again.

Read the output file to retrieve the result: /tmp/claude/-Users-challen-www-usingandunderstanding-ai/tasks/b0b97bc.output

This session is being continued from a previous conversation that ran out of context. The summary below covers the earlier portion of the conversation.

Analysis: Let me chronologically analyze this conversation:

1. Starting Context: This is a continuation of a previous session about Better Auth migration and E2E testing setup. The previous session had completed:

   • Better Auth migration from NextAuth v5 beta
   • Replaced AI SDK with OpenAI SDK for Azure OpenAI
   • Created RAG ingestion on startup
   • Set up E2E testing infrastructure with Playwright

2. Commits made at start: 6 commits were made:

   • Migrate from NextAuth v5 beta to Better Auth
   • Replace AI SDK with OpenAI SDK for Azure OpenAI
   • Add automatic content ingestion on API startup
   • Add E2E testing infrastructure with Playwright
   • Complete Better Auth migration in web app
   • Add .gitignore for Playwright test artifacts

3. Main Task: Getting E2E tests to pass. The user asked to "get the e2e test to pass" and "adjust the e2e authentication approach."

4. The Core Problem: Cookies are set in Playwright browser context but NOT being sent with HTTP requests. Test output consistently shows:

   • Cookies BEFORE navigation: [ 'better-auth.session_token([local-hostname])' ]
   • Initial page request Cookie header: NONE
   • API returns 401 because no session cookie received

5. User Frustrations/Corrections:

   • "Look, this WORKS when run from the browser. So at this point you're debugging the TEST CONFIGURATION."
   • "You should set cookies the same way that Better Auth does."
   • "No no no no no. This is totally wrong. Check ../cs124.org/sites/cs124 again."
   • "You're making a huge mess here."
   • "No, just mimic the cs124.org configuration. Exactly. Then explain to me what is not working and why. You're just trying random things hoping anything works."
   • "THEY WORK. Why would I ask you to copy failing tests?" (about cs124.org)
   • "No. Now you're just making things up again."

6. What was tried:

   • Changed cookie domain from [local-hostname] to [local-hostname] (without dot)
   • Tried using url instead of domain in cookie config
   • Tried setting cookies for both subdomains explicitly
   • Changed order of operations (cookies before page creation)
   • Copied cs124.org configuration exactly

7. Key files examined:

   • ~/www/cs124.org/sites/cs124/e2e/utils/authHelper.ts
   • ~/www/cs124.org/sites/cs124/e2e/fixtures/auth.fixture.ts
   • ~/www/cs124.org/sites/cs124/e2e/playwright.config.ts
   • Our equivalents in testing/e2e/

8. Key Difference Identified:

   • cs124.org uses NextAuth with JWE tokens - token is self-contained, server decrypts it
   • usingandunderstanding.ai uses Better Auth with database sessions - token is random string that must be looked up in database

9. Current State: Cookies are configured identically to cs124.org but still not being sent. The user asked what I'm missing.

Summary:

1. Primary Request and Intent: The user explicitly requested to "get the e2e test to pass" after starting the backend in E2E mode. The specific test is the homepage test that asks "When does the course meet?" and expects a response from the AI chat endpoint. The user emphasized that the app works when run from a real browser, so the issue is specifically with the Playwright test configuration, not the application itself.

2. Key Technical Concepts:

   • Playwright E2E testing with authenticated sessions
   • Cross-subdomain cookie handling (www. and api. subdomains)
   • Better Auth database sessions vs NextAuth JWE tokens
   • Cookie attributes: domain, path, httpOnly, secure, sameSite
   • Browser context cookie injection in Playwright

3. Files and Code Sections:

   • ~/www/cs124.org/sites/cs124/e2e/utils/authHelper.ts (reference implementation that works):

     export const COOKIE_NAME = "__Secure-next-auth.session-token"
     export const COOKIE_DOMAIN = "[local-hostname]"
     
     export function getSessionCookie(token: string) {
       return {
         name: COOKIE_NAME,
         value: token,
         domain: COOKIE_DOMAIN,
         path: "/",
         httpOnly: true,
         secure: true,
         sameSite: "Lax" as const,
       }
     }
     

   • testing/e2e/utils/authHelper.ts (current, matching cs124.org):

     import crypto from "crypto"
     
     export const COOKIE_NAME = "better-auth.session_token"
     export const COOKIE_DOMAIN = "[local-hostname]"
     
     export interface TestUser {
       id: string
       email: string
       name: string
       role: "student" | "instructor" | "admin"
     }
     
     export function generateSessionToken(): string {
       return crypto.randomBytes(32).toString("base64url")
     }
     
     export function getSessionCookie(token: string) {
       return {
         name: COOKIE_NAME,
         value: token,
         domain: COOKIE_DOMAIN,
         path: "/",
         httpOnly: true,
         secure: true,
         sameSite: "Lax" as const,
       }
     }
     

   • testing/e2e/fixtures/auth.fixture.ts (current, matching cs124.org pattern):

     getContext: async (userId: string) => {
       const user = createdUsers[userId]
       if (!user) {
         throw new Error(`User "${userId}" not found. Call create() first.`)
       }
     
       const context = await browser.newContext()
       await setupContext(context)
       await context.addCookies([getSessionCookie(user.sessionToken)])
       contexts.push(context)
     
       const page = await context.newPage()
       pages.push(page)
     
       return { context, page }
     },
     

   • packages/auth/src/config.ts (Better Auth cookie config):

     advanced: {
       crossSubDomainCookies: {
         enabled: true,
         domain: COOKIE_DOMAIN, // "[local-hostname]"
       },
       defaultCookieAttributes: {
         httpOnly: true,
         sameSite: "lax",
         secure: !DEVELOPMENT || process.env.AUTH_URL?.startsWith("https"),
         path: "/",
       },
     },
     

4. Errors and fixes:

   • Cookies not being sent with requests: Still unresolved. Test output shows cookies exist in context but Cookie header: NONE in requests.
   • User feedback: "You're just trying random things hoping anything works" - I was making changes without understanding root cause.
   • User feedback: "THEY WORK" - cs124.org tests work, so the issue is specific to our setup, not the Playwright configuration pattern.

5. Problem Solving:

   • The Playwright configuration is now identical to cs124.org (which works)
   • Cookie shape is identical
   • Fixture pattern is identical (newContext → addCookies → newPage)
   • The fundamental difference is authentication mechanism:
     • cs124.org: JWE token generated in test, server decrypts directly (no database)
     • Ours: Random token stored in database via API, server does database lookup
   • But this shouldn't affect whether Playwright SENDS cookies - that's browser behavior

6. All user messages:

   • "OK. Now, can we get the e2e test to pass? At this point we may need to adjust the e2e authentication approach. Start the backend in e2e mode and then run the test."
   • "Look, this WORKS when run from the browser. So at this point you're debugging the TEST CONFIGURATION."
   • "You should set cookies the same way that Better Auth does."
   • "No no no no no. This is totally wrong. Check ../cs124.org/sites/cs124 again."
   • "You're making a huge mess here."
   • "No, just mimic the cs124.org configuration. Exactly. Then explain to me what is not working and why. You're just trying random things hoping anything works."
   • "THEY WORK. Why would I ask you to copy failing tests?"
   • "No. Now you're just making things up again."

7. Pending Tasks:

   • Get E2E test to pass - specifically the homepage test that authenticates and asks "When does the course meet?"
   • Figure out why Playwright cookies are not being sent with HTTP requests

8. Current Work: The configuration now exactly matches cs124.org, but cookies still aren't being sent. Test output:

   Cookies BEFORE navigation: [ 'better-auth.session_token([local-hostname])' ]
   Initial page request Cookie header: NONE
   API request Cookie header: NONE
   API response status: 401
   

   I was asked "What am I missing?" after admitting I was stuck. The user confirmed cs124.org tests work with identical configuration.

9. Optional Next Step: The user asked: "I'm stuck. What am I missing?"

   The key insight that needs investigation: The ONLY difference is authentication mechanism (JWE vs database sessions). While this shouldn't affect cookie sending, perhaps the issue is that Better Auth expects something different - maybe a different cookie name format, or the session lookup is failing for another reason. Need to verify what cookie name Better Auth actually looks for when validating sessions, or investigate if there's something about the .local domain or HTTPS setup that differs between the two projects.

If you need specific details from before compaction (like exact code snippets, error messages, or content you generated), read the full transcript at: ~/.claude/projects/-Users-challen-www-usingandunderstanding-ai/ffe8f045-e4f9-4bb4-bc05-627c7ff7b7cd.jsonl Please continue the conversation from where we left it off without asking the user any further questions. Continue with the last task that you were asked to work on.

Do some external research on this issue.

No.

The tests should be using [local-hostname], not the raw localhost domains.

Sorry, but you're just making things up again. THIS WORKS IN THE BROWSER! It's a test configuration issue. What browser are we using?

For playwright?

Let's just debug cookie setting in general. Set a cookie for testing and the verify that it can be viewed on the API server.

WHAT ARE YOU DOING!

OK.

WTF?

But wait, you said it was hardcoded?

No no no. Remember. THIS WORKS IN MY BROWSER.

Why don't you create a testing page on the frontend that just makes a fetch call to the backend testing route to see if the cookies are transmitted properly that way?

WHY DID THE BACKEND SERVER NOT SHUT DOWN CLEANLY AGAIN!

Wait. Do we need sessions? If so, why? cs124.org doesn't use them. What would they be for?

Why not just use the JWT approach?

No, I legitimately just don't understand what sessions are for. Is it so you can store more data? What would be the use case?

Can't we do session tracking with JWTs as well?

No, sorry. I meant can't you accomplish #3 above by just logging when and where from the JWTs are used?

Yes.

No let's just replace sessions entirely.