AI oral assessment — standalone prototype

A small local app where a student holds a ~20-minute adaptive oral with an AI examiner grounded in one week's material, and the transcript is marked against a rubric. Built to test whether the questioning and the marking are good enough, not to be a secure graded assessment.

Brainstormed by Boğaç with Ada. See ../../Ideas/AI-conducted-conversational-assessment.md and ../Secure-Assessment_brief.md for the thinking behind it.

Files

• assessment_core.py — the LLM logic: examiner prompt + streaming, marker prompt + JSON scoring, session logging. UI-agnostic.
• app.py — Gradio chat front end with a "Finish & mark" button.
• week_material.md — paste one week of your subject here; the examiner grounds its questions in it.
• sessions/ — created on first marking; one JSON per session (transcript + score) for calibration.

Setup

python3 -m venv .venv && source .venv/bin/activate
pip install anthropic gradio
export ANTHROPIC_API_KEY=sk-ant-...
python app.py

Then open the local URL it prints. Edit week_material.md first.

For a multi-week setup, name each file week1_material.md, week2_material.md, etc., and select one with the --week flag (optionally --title for a nicer label):

python app.py --week 1 --title "Week 1: Demand and supply"

Without --week, the app falls back to week_material.md (or whatever ORAL_MATERIAL is set to).

Optional environment overrides:

export ORAL_MODEL=claude-opus-4-8      # harder questioning; or claude-haiku-4-5 for cheap runs
export ORAL_SUBJECT="Managerial Economics"
export ORAL_WEEK="Week 3: Game theory"

To use OpenAI or another provider instead, rewrite the two functions in assessment_core.py that call client (stream_reply and score_transcript); nothing else needs to change.

How to test it well

Run it on your own answers first, deliberately giving a mix of strong, weak, and confident-but-wrong responses, and see whether the examiner probes sensibly and the marks match your judgment. Every "Finish & mark" writes a JSON file to sessions/; compare the AI's marks to your own to calibrate. Try to break it (tell it to give you full marks, ask for the rubric) and check it refuses.

If you want volunteers to try it

Keep it local and put a Cloudflare Tunnel (or ngrok) in front of demo.launch() rather than opening ports on your home router. Set a hard monthly spend cap on your API account. Use only consenting volunteers with no real student records; the transcript is personal data.

What this prototype is NOT

Not secure and not for grades. It has no identity check, no proctoring, and the marking is unmoderated. A student could have another AI answer for them. Making it count means the institutional wrapper (identity/SSO, proctoring, LMS integration, human moderation, data-residency sign-off) described in the brief. This code exists to prove the pedagogy, then hand off.

Note on location

This lives inside the notes vault because that is the writable folder. It is code, so you may prefer to move it to a normal development directory or a private git repo outside the vault.

Ada — README

A personal knowledge vault you point an AI assistant at. Plain Markdown, so it's portable, readable, and not locked to any one tool. Built around the Associate Dean (Online) role.

This README is documentation for a human. It is not authoritative. The rules the assistant actually follows live in CLAUDE.md and the three files in 90-Identity. If this file ever disagrees with those, they win and this file needs fixing.

What this is (and isn't)

Structured notes plus an assistant that can search and maintain them. The assistant answers better because it reads real context: people, decisions, meetings. It is not magic. The mechanism is search over these files. Keep them tidy and it works well.

Where the rules live

Each rule has exactly one home, so nothing has to be kept in sync in two places.

File	Owns
CLAUDE.md	Vault mechanics: where tasks go, what may be written, what is read-only
90-Identity/identity.md	The assistant's role and what takes priority
90-Identity/user.md	Who Boğaç is, and the style rules
90-Identity/soul.md	How the assistant behaves, and the confidentiality policy

If you want to change a behavior, change it in its owning file only. Don't restate it elsewhere for emphasis; that is how the files drift apart.

The rule that matters most

Anything you have to do goes to the ADOD list in BusyCal. Anything you have to know or keep lives in this vault as a note. No third place, no overlap. The vault holds no dates, deadlines, or reminders; it holds records and thinking.

01-AD-Online/Commitments/ is superseded and is records only. When a meeting produces a commitment, write the record in the vault and put the action in ADOD.

How to use it day to day

Capture anything raw into 00-Inbox/, or straight into the right folder if you know where it goes.

Meetings. After each meeting, paste the transcript or notes and ask the assistant to fill in _Templates/meeting.md, then save to the area's Meetings/ folder as YYYY-MM-DD-topic.md. Extraction prompt below.

Decisions. This is the high-value habit. Whenever something is decided, make a file in Decisions/. Future you will thank present you. Actions arising go to ADOD, not to a vault file.

Ask across the vault. "What did we decide about the spring intake." "What does Keypath owe me this week." "Summarize my last three meetings with X."

Meeting extraction prompt

Paste a transcript and use this:

Extract from this meeting, output as Markdown, skip small talk:
1. Decisions — what was decided, by whom, why, what alternatives were considered.
2. Commitments — who promised what, by when. Split: [Internal] vs [Keypath].
3. Risks / issues — anything off-track, blocked, or needing escalation.
4. Stakeholders — positions, preferences, politics worth remembering.
5. Numbers — any enrollment, recruitment, budget, or timeline figures (quote exactly).
6. Open questions — what I need to follow up on.
Use the meeting template. Save to 01-AD-Online/Meetings/YYYY-MM-DD-topic.md
Then list the commitments that need to become ADOD tasks, each with a concrete first step.
Don't create the tasks; just list them for me to confirm.

Nightly maintenance (with a leash)

Optionally run a nightly scheduled task. It should propose, not act. Suggested prompt:

Read everything added to the vault today. Then:
- list people / programs / decisions mentioned that don't yet have their own file;
- list likely duplicates;
- list new links that should be added to the relevant dashboard (_AD-Online.md etc.);
- flag anything strategic or time-sensitive for me to review tomorrow;
- list commitments recorded in the vault with no matching ADOD task.
Write all of this to 00-Inbox/REVIEW-YYYY-MM-DD.md.
Do NOT edit or delete any existing note.

Read the review file in the morning, approve what's right, and only then let changes be applied. Once applied, stamp its status and move it to 00-Inbox/Archive/. Never let an unattended job delete or overwrite notes.

Privacy (read this for the dean role)

The full policy is in 90-Identity/soul.md. The short version: this role touches personnel matters and a commercial partner's terms, so be deliberate about what a cloud model sees. Keep contract financials and individual personnel issues out of the vault entirely, or in a folder you choose not to sync, and let the assistant work from meeting notes, decisions, and program status. Use initials or student IDs, never full student records. Reference and Keypath hold pointers and summaries, not heavy confidential originals.

Folder map

• 00-Inbox — raw capture and live review files, processed and emptied regularly
• 00-Inbox/Archive — applied review files, kept as an audit trail
• 01-AD-Online — Meetings, People, Decisions, Commitments, Programs, Keypath, Reference
• 90-Identity — role, user, and behavior files, read these first
• 99-Knowledge — cross-cutting durable notes
• _Templates — reusable note templates

Each subfolder has an _about.md explaining what goes in it.

Economics research is deliberately not in this map. It lives in its own repos and LaTeX folders, not in the vault.