Implement the following plan:

Plan: Automated K8s Database Backup CronJob

Context

The manual bun run db:backup script works, but we need automated daily backups running in the K8s cluster. Key challenges:

• rclone Box OAuth tokens expire (~60 min access, ~60 day refresh) and rclone needs to write refreshed tokens back to its config file — a read-only K8s Secret won't work
• Backup failures should send email notifications without hardcoding recipient addresses
• The backup container needs pg_dump, gzip, rclone, and an SMTP client

New/Modified Files

Task 1: Docker Image (k8s/db-backup/Dockerfile)

Lightweight Alpine image with all required tools:

FROM alpine:3.21
RUN apk add --no-cache postgresql16-client gzip rclone msmtp bash
COPY backup.sh /backup.sh
RUN chmod +x /backup.sh
CMD ["/backup.sh"]

• postgresql16-client provides pg_dump (matches the pg16 server image)
• msmtp is a minimal SMTP client for failure notifications
• Image pushed to docker.io/usingandunderstandingai/db-backup:latest

Task 2: Backup Script (k8s/db-backup/backup.sh)

Connects directly to postgres in-cluster (no kubectl needed). On failure, sends notification via the mail service.

Flow:

1. For each env (prod, dev):
   • pg_dump -h postgres -U postgres -d $DB_NAME | gzip | rclone rcat box:usingandunderstanding-backups/{date}/{filename}.sql.gz
   • PGPASSWORD from DB_PASSWORD env var (injected from shared-secrets)
   • --rclone-config points to the PVC mount at /rclone/rclone.conf
2. On any failure, send notification email via msmtp to [email] through the internal mail service ([redacted])
3. Exit with appropriate code so K8s tracks job success/failure

msmtp config (inline in backup.sh):

account default
host mail-service
port 2525
from [email]

Task 3: K8s Manifests (k8s/db-backup.yaml)

PVC — small persistent volume for rclone config (so refreshed OAuth tokens survive):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rclone-config
  namespace: usingandunderstandingai
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: rook-ceph-block-doublereplicated-discard
  resources:
    requests:
      storage: 1Mi

CronJob:

• Schedule: 0 8 * * * (2 AM Central = 8:00 UTC, since cluster likely uses UTC)
• concurrencyPolicy: Forbid
• successfulJobsHistoryLimit: 3, failedJobsHistoryLimit: 3
• Container mounts:
  • rclone-config PVC at /rclone (read-write for token refresh)
• Env vars:
  • DB_PASSWORD from shared-secrets
• restartPolicy: OnFailure
• imagePullSecrets: docker-registry

Task 4: rclone Setup Script (k8s/rclone-setup.sh)

One-time script to initialize the rclone PVC from local ~/.config/rclone/rclone.conf:

1. Ensure PVC exists (apply db-backup.yaml first)
2. Run a temporary Alpine pod with the PVC mounted
3. kubectl cp local rclone.conf into the pod at /rclone/rclone.conf
4. Delete the temporary pod

This only needs to run once. If the PVC is lost, re-run the script.

Task 5: Mail List (content/mail-lists.ts)

Add backup list so notification recipients are configurable without touching K8s manifests:

backup: {
  address: "backup",
  displayName: "Backup Notifications",
  description: "Database backup failure notifications",
  recipients: { emails: ["[email]"] },
  allowedSenders: ["[email]"],
},

Works on both usingandunderstanding.ai and dev.usingandunderstanding.ai since mail-lists.ts is shared.

Task 6: Integration

• Add kubectl apply -f "$SCRIPT_DIR/db-backup.yaml" to k8s/deploy.sh after postgres is ready
• Add image build/push to scripts/deploy.ts or document as a manual step (the image changes rarely)
• Add db:backup:build and db:backup:push scripts to root package.json for manual image management
• Document the CronJob in CLAUDE.md

Verification

1. Build and push the image: bun run db:backup:build && bun run db:backup:push
2. Run rclone setup: ./k8s/rclone-setup.sh
3. Apply the CronJob: kubectl apply -f k8s/db-backup.yaml
4. Trigger a manual run: kubectl create job --from=cronjob/db-backup db-backup-test -n usingandunderstandingai
5. Check logs: kubectl logs job/db-backup-test -n usingandunderstandingai
6. Verify files appear in Box under usingandunderstanding-backups/
7. Test failure notification by temporarily using a wrong postgres hostname, confirm email arrives

If you need specific details from before exiting plan mode (like exact code snippets, error messages, or content you generated), read the full transcript at: ~/.claude/projects/-Users-challen-www-usingandunderstanding-ai/72399be9-a1f6-4e4d-b3a5-94603a50127d.jsonl

Will this deploy if I run deploy:dev?

Manual is fine. And so this is cron container run by k8s?

Nice! I probably need to rebuild the dev and prod containers, right, to pick up the mail fix?

Your task

First, run git status, git diff, and git log --oneline -10 to understand the current state.

Then add and commit all modified files with a descriptive message. Break the changes into multiple commits if appropriate. If there are untracked files, ask if they should be added or added to the appropriate .gitignore file. The user may also want to take over at this point, so provide an abort option as well. Always check changes for unnecessary debugging statements or logging that might have been accidentally included. Prompt the user if you find any, and offer to remove each one by one. Do not use interactive git staging (git add -p) as it does not work.

1

Your task

First, run git status, git diff, and git log --oneline -10 to understand the current state.

Then add and commit all modified files with a descriptive message. Break the changes into multiple commits if appropriate. If there are untracked files, ask if they should be added or added to the appropriate .gitignore file. The user may also want to take over at this point, so provide an abort option as well. Always check changes for unnecessary debugging statements or logging that might have been accidentally included. Prompt the user if you find any, and offer to remove each one by one. Do not use interactive git staging (git add -p) as it does not work.

I'm wondering if we can create an /instructors page? This would be something I could send to other instructors that would highlight the great and very interesting work we're doing. It should probably provide an overview of the syllabus (probably at the bottom), a summary of the course, links to sample activities and discussion of the activity framework, and examples or links to some of the more interesting bits like conversational assessment and discussion.