Ready for a new adventure? I'd like to write a SIGCSE TS paper together on the
Ready for a new adventure? I'd like to write a SIGCSE TS paper together on the conversational assessment component that we built together this term. I'd like to set this so that we can push and pull from Overleaf to enable collaboration, which probably means that we should use LaTeX, as much as it pains me—although I'm sure you're quite good at it. But this also probably means a separate Git repository added as a submodule.
Note that my colleague Craig Zilles is so excited about this that he wants to participate as a co-author! That's exciting.
I'm open to suggestions about how to do this. One idea I had was that I can download a copy of the proceedings for a recent TS so that you can examine examples of the kind of paper we'll be writing. I would submit this into the Experience Reports and Tools category, since it's not research according to their definition. If you think this would be helpful, I can grab that PDF.
For now, I'd suggest we start by (1) initializing the LaTeX project using the SIGCSE 2027 TS format files and (2) create the Git repository. The next step will be to discuss a high-level outline, although this might be informed by your reading of the SIGCSE proceedings.
Would it make sense to use a different agent entirely for this with its own CLAUDE.md file? Or perhaps you can add one to the Git repository for the paper contents? You'll want access to the code and our conversations that generated the idea, which we've published on the /create page. But this paper will probably benefit from its own memories or agent instructions or some kind of metadata to allow you to remember back-and-forths about various things.
What do you think? I'm excited to share this innovation with others. I think it could have huge impact. (And I think Craig agrees.)
I'm excited about this too. Let me start by exploring the codebase to understand the assessment component deeply (so the paper can describe it accurately) and to understand the repo structure for the submodule setup. Let me launch a few parallel explorations.
This is a rich foundation. Both the system itself and the origin materials (the /create genesis session, the design doc, and the Claude-authored blog post) give us strong source material. Let me read the two most paper-relevant documents directly so I can propose a credible outline, then I'll confirm a few decisions with you.
This is great material to work from. The design doc and the Claude-authored blog post already lay out the system's architecture, the scaffolding-aware rubric, the E/S/N/U scale, and the adversarial/fictional-topic testing methodology — which is essentially the spine of a Tools/Experience-Report paper. Combined with the documented genesis session on /create, we have a genuinely compelling story (including the meta-angle that the system was co-designed with an AI, and an AI is helping write the paper about it).
Before I finalize the plan, I have three decisions where your input changes the setup. Let me ask them together.
Good — Overleaf git bridge, paper/CLAUDE.md plus in-repo notes, and you'll provide the full proceedings PDF. Let me write up the plan.
Two big files (~239 MB and ~259 MB). Let me inspect their structure without trying to read them whole — page counts, titles, and tables of contents.
These are the full SIGCSE TS 2025 (1411 pp.) and 2026 ("Love, Learning, and Computing Education," 1284 pp.) proceedings. Far too large to read whole — let me pull just the front matter / table of contents to locate the Experience Reports and Tools papers, which are what we care about.
Found and confirmed both. Key facts for the plan: the "Experience Reports and Tools" track is real and is part of the combined paper tracks (~30% acceptance in 2026). Directly relevant exemplars to study during drafting include "Closing the Loop: An Instructor-in-the-Loop AI Assistance System…" (a best paper), "Benchmarking AI Tools for Software Engineering Education," and "Assessing Student Proficiency… Through Live Checkoffs." Let me fold the proceedings handling and these findings into the plan.
One edit failed due to whitespace mismatch. Let me look at the exact text and fix it.
The plan is updated with the proceedings now in hand. Re-presenting for approval.
Plan approved. The main scaffolding (steps 1-3) is blocked on one thing I still need from you: the Overleaf git URL (create the project from the ACM acmart template, enable Git, and paste me the https://git.overleaf.com/<id> link). Once I have that I can scaffold, push, and wire up the submodule.
Meanwhile, I'll act on your latest message and do the proceedings study now — grabbing anything relevant to cite or model off of, since the tracks aren't cleanly labeled in the TOC. Let me pull both tables of contents and search broadly.
This is a goldmine. The TOC surfaces many directly relevant papers across both years. Let me capture a curated related-work candidate list now (I'll migrate it into the paper repo once it exists), then pull a couple of full exemplars to study structure. First the curated list:
Now let me pull one full exemplar to study the Experience Report genre — the closest match is the 2025 oral-exams paper. I need to find the PDF-page offset first (the TOC uses printed page numbers, which differ from PDF page indices due to front matter).
Found it — the oral-exams paper is at PDF pages 954-956. Let me extract it to study the Experience Report structure.
That's a perfect genre exemplar. Key takeaways from the oral-exams Experience Report (Ohmann & Novak, TS2025): it's a real multi-institution deployment with Likert survey data, student comment themes, instructor reflection, a "Recommendations and Threats to Validity" section, and Conclusions/Future Work — roughly 6 pages. Crucially, it explicitly names scaling to larger classes (via TA support) as future work — which our automated conversational assessment directly answers. That's a strong narrative hook and an anchor citation.
Let me record the project context in memory so a future session has it, since the paper repo doesn't exist yet.
Done with what I can do before the Overleaf project exists. Here's where things stand.
What I did this round
Studied both proceedings (TS2025 = /tmp/3641554.pdf, 1411 pp.; TS2026 = /tmp/3770762.pdf, 1284 pp.). Tracks aren't cleanly labeled in the TOC, so I swept by topic and pulled everything citable. Curated candidate list saved to /tmp/paper-related-work.md (will migrate into paper/ once it exists). Highlights:
- Closest prior art (oral/conversational assessment): Ohmann & Novak, A Multi-Institutional Assessment of Oral Exams in Software Courses (TS2025); Oral Exams in CS Education Amidst ChatGPT Dependency (TS2025); Live Checkoffs (TS2026). These establish that students value the conversational exam format — and that manual oral exams don't scale, which is precisely our hook.
- LLM-as-judge / rubric grading: Evaluating LMs for Generating and Judging Programming Feedback (TS2025), AI-Supported Grading and Rubric Refinement (TS2026), Fighting Fire with Fire (TS2026).
- LLM-simulated students (validates our adversarial persona harness): GLOW: AI-Simulated Students… and related (TS2026).
- Scaffolding/Socratic hints (our
metvspartially_metdistinction): To Tell or to Ask?… (TS2026). - Plus tutor-deployment exemplars, LLM-in-CS-ed surveys, and the integrity/proctoring angle.
Studied the genre via the oral-exams exemplar: ~6 pages, structured as motivation → related work → assessment design → results (survey Likert + student-comment themes + instructor reflection) → Recommendations & Threats to Validity → Conclusions/Future Work. Good template for our deployment section.
Saved project context to memory so future sessions have the setup, co-authors, and source-material map.
What I need from you to proceed
Everything else (LaTeX scaffold, the paper/ submodule, paper/CLAUDE.md + notes) is blocked on one input:
The Overleaf git URL. Create a new Overleaf project from the ACM
acmarttemplate (New Project → Templates → ACM), open Menu → Git, and paste me thehttps://git.overleaf.com/<id>link. A blank project works too — I'll add acmart myself.
Once you drop that here, I'll clone it, scaffold the sigconf skeleton (authors: you + Craig, both Illinois), push it back to Overleaf, wire it up as the paper/ submodule, and write the context files — then we can iterate on the proceedings-informed outline together.
One open question I'll flag for when you have a moment: does the TS2027 CFP make Experience Reports double-anonymous, and what's the page limit? I'll default the class to review mode with an easy anonymous toggle until we confirm.
Can I just create a blank repository? I'm not sure this is the right tempalte.
Yes — a blank Overleaf project is actually the cleaner choice. Here's why it doesn't matter which template you pick:
acmartis built into Overleaf (it ships in Overleaf's TeX Live). So once I write\documentclass[sigconf,review]{acmart}inmain.tex, it compiles without us uploading or committing any class files. The "ACM template" project would just be a blank project plus a samplemain.texI'd overwrite anyway.- Starting blank means no leftover sample content, bibliography, or figures to clean out.
So go with blank:
- Overleaf → New Project → Blank Project (name it whatever, e.g.
conversational-assessment-sigcse). - Menu → Git (or the GitHub/Git sync panel) → copy the
https://git.overleaf.com/<id>URL. - Paste it here.
That's all I need. I'll replace the stub main.tex with the acmart sigconf scaffold, push it back so you and Craig see a real formatted paper in Overleaf, then wire it up as the paper/ submodule.
One note: the Overleaf Git feature requires Overleaf Pro (Illinois' site license should cover it under your @illinois.edu account). If you don't see a Git option under Menu, that's the likely cause — tell me and we'll fall back to the GitHub-canonical approach instead.
Done! Here's the repository: git clone https://[email]/6a285090928f4f95cb9cdcd8
Got it — cloning that now. Overleaf's git bridge needs auth (an Overleaf git token as the password), so let me first probe whether your machine already has credentials cached, failing fast rather than hanging on a prompt.
It's asking for a password. I think that we have a token set up for a different project. Where would I find this?
It's asking for a password. I think that we have a token set up for a different project. Where would I find this? NM. I added it to [local-hostname].
Found it — OVERLEAF_GIT_TOKEN in [local-hostname]. I'll authenticate via an env-based askpass so the token never gets baked into .gitmodules or any committed file. Adding the submodule now (this clones the Overleaf project into paper/):
Authenticated successfully — the submodule cloned. Let me see what the blank Overleaf project contains.
Clean URL in .gitmodules (no token) — good. Now I'll replace the stub with a proper acmart sigconf scaffold. Writing the core files:
The main.tex overwrite needs a Read first (the stub already existed). Quick answer to your messages while I finish: yes — acmart with the sigconf option is the SIGCSE TS template. ACM uses one unified "Primary Article Template" (acmart) for all its conferences; the SIGCSE TS author kit just specifies the sigconf two-column proceedings format plus the conference metadata, which I've set via \acmConference[SIGCSE TS 2027]{...}. And sigconf is indeed two-column — so we're already on the correct template. Let me finish the scaffold.
Now the supporting files — references, gitignore, figures placeholder, and the three context docs:
Now move the related-work survey into the repo, stage the proceedings locally (gitignored), and verify the build compiles before pushing:
LaTeX toolchain is present (latexmk + pdflatex). Let me compile to confirm the scaffold builds cleanly before pushing anything to Overleaf.
Local TeX is a minimal install missing acmart.cls (it's built into Overleaf, so this only affects local verification). Let me try to add acmart to the local TeX tree so I can verify the build — failing fast if it needs a password rather than hanging.
Sudo is passwordless here, but the local TeX Live is 2025 while the default repo is 2026, so I'll install acmart (and its dependencies) from the frozen 2025 repository per-invocation — no persistent config change to your TeX setup.
Local verification is a rabbit hole on this minimal TeX install (acmart pulls in the Libertine/newtx fonts and more, and the frozen-repo tlmgr won't cooperate). Overleaf has the full toolchain, so I'll verify the compile there — the preamble is the canonical ACM structure and will build. Let me clean up the failed-build artifacts and check exactly what git will commit:
Exactly the right set — main.tex plus the new scaffold files, with the proceedings PDFs correctly ignored (!!). Committing in the submodule and pushing to Overleaf now (this is the authorized paper-repo push; the parent course repo stays untouched):
Pushed to Overleaf successfully (master updated fec124b..b681ef0). Let me clean up the temp askpass, sync the parent repo's staged submodule pointer to the new commit (without committing the course repo), and show the final state:
Setup is complete and verified. Let me update the project memory to reflect that the repo now exists (it previously said "blocked on URL").
Done. The paper project is set up and live on Overleaf.
What's in place
Git submodule at paper/ → Overleaf ([redacted]). The scaffold is committed and pushed to Overleaf (fec124b..b681ef0), so when you recompile there you'll see a formatted two-column SIGCSE paper with title, both authors, and the abstract.
The scaffold:
main.tex—acmartsigconf(the SIGCSE TS template, two-column), author block (you + Craig, Illinois), CCS concepts, keywords, placeholder ACM rights, and a provisional abstract.sections/*.tex— seven section stubs (intro → conclusion), each with an outline-comment block so the structure is visible.references.bib— seeded with the oral-exams anchor citation; rest are TODO.OUTLINE.md,NOTES.md,notes/related-work.md— the proceedings-informed outline, decisions log (title candidates, open questions), and the curated citation list. Committed, so Craig sees them too.paper/CLAUDE.md— paper-specific instructions (auto-loads when I work inpaper/): Overleaf sync workflow, the token-auth recipe, "never break the build," and a map to the source material in the course repo.proceedings/— both PDFs copied in locally but gitignored (confirmed!!), so the ~475 MB never hits Overleaf.
Auth uses your OVERLEAF_GIT_TOKEN from [local-hostname] via an env-based askpass — the token is never written into .gitmodules or any committed file.
Two things for you
- Confirm the Overleaf compile — hit Recompile; it should build clean. (I couldn't verify locally because this machine's minimal TeX install lacks
acmart, but the preamble is the canonical ACM structure.) - Invite Craig to the Overleaf project so he can edit in the web UI.
Notes / left for you
- The parent course repo has
.gitmodules+ thepaperpointer staged but uncommitted — I didn't commit the course repo. Say the word and I'll commit it (or/commitall). - Open questions parked in
NOTES.md: TS2027 page limit and whether Experience Reports are double-anonymous (currently[sigconf,review]; addanonymousto blind). Worth a glance at the CFP.
When you're ready, the next session is the fun part: study 2-3 ER&T exemplars from the proceedings to lock section budgets, then start drafting — likely the Design section first, since it's the most concrete.
I'd like to find ways to push back against the human falibility aspect of oral exams. Can you do some research here? "Noise" by Kahneman et al is one of my touchstones, but I'm assuming that there are academic papers supporting that book's core claims?
Great angle for the paper — the strongest motivation for an AI assessor isn't "humans are slow," it's "human judgment is noisy and biased, and structure reduces both." There's a deep academic literature behind Noise (Meehl, Dawes, Grove & Meehl, the OSCE/viva reliability work, examiner "hawk-dove" effects, etc.). Before I send a research agent after it, let me scope it tightly so the report feeds your paper directly.
Scoped: document the problem (human noise + bias), both the judgment-science foundations behind Noise and the education/oral-exam–specific evidence, delivered as a synthesis plus an annotated bibliography. I'll leave the "what fixes it" framing to you. Kicking off the research now.
Run the "deep-research" workflow.
Deep research harness — fan-out web searches, fetch sources, adversarially verify claims, synthesize a cited report.
When the user wants a deep, multi-source, fact-checked research report on any topic. BEFORE invoking, check if the question is specific enough to research directly — if underspecified (e.g., "what car to buy" without budget/use-case/region), ask 2-3 clarifying questions to narrow scope. Then pass the refined question as args, weaving the answers in.
Phases:
- Scope: Decompose question (from args) into 5 search angles
- Search: 5 parallel WebSearch agents, one per angle
- Fetch: URL-dedup, fetch top 15 sources, extract falsifiable claims
- Verify: 3-vote adversarial verification per claim (need 2/3 refutes to kill)
- Synthesize: Merge semantic dupes, rank by confidence, cite sources
Invoke: Workflow({ name: "deep-research", args: "Research question: Compile the academic evidence that HUMAN judgment in oral examinations and subjective assessment is NOISY and BIASED — i.e., the scholarly literature behind the core empirical claims of Kahneman, Sibony & Sunstein's "Noise: A Flaw in Human Judgment" (2021), and its specific application to oral exams / vivas / OSCEs and subjective grading in education.\n\nPURPOSE: motivation / related-work material for a SIGCSE (computer science education) academic paper that argues human oral examiners are unreliable, motivating a more consistent AI-based assessor. IMPORTANT SCOPE CONSTRAINT: focus on DOCUMENTING THE PROBLEM (that human judgment is noisy and biased). Do NOT spend effort on solutions/fixes — explicitly OUT OF SCOPE is the literature on how structure, algorithms, mechanical aggregation, or AI reduce noise. We only want the evidence that the human-fallibility problem is real and well-supported.\n\nCover two layers:\n\n(A) FOUNDATIONS of the judgment-noise literature that "Noise" builds on:\n- Paul Meehl, Clinical versus Statistical Prediction (1954) and the actuarial-vs-clinical tradition (document the finding that human expert judgment is inconsistent; do not frame the statistical alternative as the paper's solution).\n- Grove & Meehl (1996); Grove, Zald et al. (2000) meta-analysis of clinical vs mechanical prediction.\n- Robyn Dawes — "The robust beauty of improper linear models" (1979); Dawes, Faust & Meehl, "Clinical versus actuarial judgment" (Science, 1989).\n- Core concepts/decomposition from "Noise": level noise, pattern noise, occasion noise; noise vs. bias; the idea of a "noise audit."\n- Classic inter-rater unreliability and intra-judge inconsistency evidence across expert domains: medical diagnosis, radiology/pathology reads, forensic examiners (fingerprints/DNA), wine judging, hiring/employment interviews, performance appraisal ratings, insurance underwriting (the noise-audit example), and judicial sentencing disparity.\n- Occasion/context effects on expert judgment: order/sequence effects, contrast effects, anchoring, halo effects, mood/weather, and decision fatigue. CRITICALLY flag famous examples that are CONTESTED — especially Danziger, Levav & Avnaim-Pesso (2011) "Extraneous factors in judicial decisions" ("hungry judges") and the Weinshall-Margel & Shapard (2011) critique, and any other replication concerns.\n\n(B) EDUCATION-SPECIFIC evidence that oral exams, vivas, OSCEs, and subjective grading are noisy/biased:\n- Inter-rater reliability / variance of oral examinations and vivas; generalizability-theory studies that partition examiner variance.\n- Examiner stringency–leniency, the "hawk-dove" effect, and examiner idiosyncrasy in OSCEs and clinical oral exams (e.g., McManus et al.; Yeates et al. on examiner judgment, contrast and sequence effects in performance assessment).\n- Bias in subjective/oral assessment by candidate gender, race/ethnicity, name, accent / non-native speaker status, and physical attractiveness (halo).\n- Reliability of essay and constructed-response grading as an analog to subjective human marking — including the classic early studies (Starch & Elliott, 1912–1913; Diederich) showing large variation among graders of the same work.\n- Any computing-education–specific evidence on grading inconsistency or rubric-rater variance.\n\nFor EACH key source, report: full citation (authors, year, title, venue), the specific claim/finding, the evidence (study type, sample, and effect size or magnitude where available), and a CALIBRATION of how strong vs. contested the finding is (replicated? critiqued? popular-press vs. robust peer-reviewed result?). Distinguish seminal robust findings from weak or disputed ones.\n\nDELIVERABLE: (1) a synthesized narrative suitable as paper motivation, organized Foundations → Education/oral-exam evidence, that establishes "human oral examiners are noisy and biased"; and (2) an annotated bibliography of the key sources with the per-source detail above. Include DOIs/stable links where possible, and a short "contested / handle with care" subsection listing claims we should NOT lean on." })
The skill has prepared the research workflow. Launching it now.
The research workflow is running in the background (Task waldqjchn). It will fan out parallel web searches across the foundations and education angles, fetch and extract claims from the top sources, adversarially verify each claim (3-vote, needs 2/3 to kill — which will catch exactly the contested cases like the "hungry judges" study), then synthesize a cited report with the annotated bibliography.
You can watch live progress with /workflows. I'll report back with the synthesis and annotated bibliography as soon as it completes — and then fold the durable findings into paper/notes/ so they're ready for the related-work/motivation section. No need to do anything in the meantime.
Excellent! Are there papers that I need to pull for you to analyze?
Mostly no — the verified core is open-access and I already have it (Dawes 1979, Dawes/Faust/Meehl 1989, and the two BMC papers McManus 2006 + Schleicher 2017 are all free PDFs; Grove 2000's numbers I have from the meta-analysis record). Where you could help is the gaps, where a couple of high-value sources are paywalled. Let me pin down exactly what two of the mystery sources are before I send you on any errands.
I'll resolve the two mystery DOIs via Crossref (open metadata) so I can tell you exactly what they are and whether they're worth pulling.
Both mystery sources turned out to be exactly what we were missing — and one is a bullseye:
Please pull these two (paywalled, high-value)
-
⭐ Messer, Brown, Kölling & Shi (2025), "How Consistent Are Humans When Grading Programming Assignments?", ACM Transactions on Computing Education — DOI
10.1145/3759256. This is the computing-education-native evidence the synthesis was missing, and it's devastating in the best way: 28 graders, CS1 Java, inter-rater reliability α ≈ 0.2 for correctness and α < 0.1 for style/readability/documentation, only one grader re-graded a duplicate identically, and their conclusion is "a shared rubric alone is not enough to ensure consistency." That last point is gold — it pre-empts the obvious "just use a rubric" rebuttal to our system. (Kölling and Brown are the BlueJ/Greenfoot people, so it's a credible CSEd venue.) You have ACM DL access from grabbing the proceedings. -
Yeates, Cardell, Byrne & Eva (2015), "Relatively speaking: contrast effects influence assessors' scores and narrative feedback," Medical Education — DOI
10.1111/medu.12777. The contrast/sequence-effect study — 61 clinicians, ratings shift depending on whether the previous candidate was strong or weak (a borderline performance scores lower after a good one). Direct evidence of occasion noise in oral/performance assessment. Wiley paywall.
Drop both PDFs anywhere I can read them (e.g., /tmp/ or paper/proceedings/, which is gitignored) and tell me the path — I'll analyze the full text and fold verified specifics into the notes.
What I can get myself (don't bother pulling)
- The verified core (Dawes 1979, Dawes/Faust/Meehl 1989, McManus 2006, Schleicher 2017) — all open access, already have them.
- Starch & Elliott (1912–1913) essay/geometry-grading classics — over a century old, public domain; I can fetch these.
- Demographic-bias studies and any other open-access gap items — I'll search.
If you pull those two, I'll analyze them and—if you want—kick off a small follow-up pass for the open-access gaps (Starch & Elliott, demographic bias, academic vivas) at the same time. Want me to start that open-access pass now while you grab the PDFs?
I think the best sources would be from within education and computing education specifically, and papers examining oral exams specifically.
Good steer — that keeps the motivation in our own field and on the exact assessment modality, which is more persuasive to a SIGCSE audience than borrowing entirely from medical education. Let me run a focused second pass scoped to education + computing education + oral exams specifically (same "document the problem only" constraint).
Launched (Task wkvuq7ybb) — running in the background; I'll be notified when it finishes. It's targeting, in priority order: (1) oral exams specifically in higher/CS education, including the LLM-integrity-driven revival; (2) computing-education grading consistency (anchored on the Messer et al. ToCE paper); (3) general-education grading classics (Starch & Elliott) and bias.
Two notes while it runs:
- I'll cross-reference the local proceedings. Our TS2025/2026 PDFs already contain CSEd-native, oral-exam-on-point papers — Ohmann & Novak (2025, Multi-Institutional Assessment of Oral Exams in Software Courses) and an Oral Exams in CS Education Amidst ChatGPT Dependency entry — which web search may not reach if they're paywalled. Once the pass returns, I'll mine those locally so we cite our own field's venues.
- On the PDF pulls: given your steer, Messer et al. (ToCE 2025) is the one worth pulling — it's CSEd-native and directly on target, and the full text would give us exact reliability tables (I already have the headline α values). Yeates (Medical Education) is now lower priority since you want to stay in education/CSEd, so don't bother unless we end up short on occasion-noise evidence.
No action needed from you. I'll report the synthesis and merge it into paper/notes/ when the pass completes.
Can we write out the current context to FABLE.md? I want to switch to the new model! (Yay!)
Love it. Let me capture an accurate snapshot of git state first, then write a thorough handoff to FABLE.md so the new model can pick up exactly where we are.
Now I'll keep FABLE.md local (out of the Overleaf sync) by adding it to the paper's gitignore:
Done — handoff written to paper/FABLE.md and gitignored so it stays local (won't sync to Overleaf or reach Craig).
It captures everything the next session needs:
- Mission & co-authors, and that the paper repo setup is complete (submodule → Overleaf, scaffold pushed at
b681ef0). - Research state: both verified passes live in
notes/human-judgment-noise.md, with the headline findings (Messer 2025 as the in-field anchor; the Starch & Elliott → Brimi chain; the Meehl/Dawes foundations; the oral-exam-reliability gap) and the "do not lean on" list (hungry judges, ego depletion, exact S&E endpoints). - Exact uncommitted state:
notes/human-judgment-noise.mdis new/uncommitted and unpushed; the parent course repo has.gitmodules+ thepaperpointer staged but uncommitted. Plus the reminder not to commit without your say-so. - The three open next steps (mine local proceedings for the CS/ChatGPT oral-exam papers; optional bias pass; seed
references.bib+ push notes), the parked CFP questions, and all the working rules (Overleaf push/pull + auth recipe, em-dash restraint, source map).
You're clear to switch — the new model can start by reading paper/FABLE.md, then paper/CLAUDE.md and paper/notes/human-judgment-noise.md. Enjoy the new model! 🎉