The writer's loop
A creative writing session moves through five stages, looping until the writer chooses to publish. The writer never leaves the canvas to find guidance — the AI and the rubric ride along.
Stage-by-stage
Configure (one-shot, before the loop) — the writer picks a genre from the 117-genre catalog. The matching rubric loads as a structural prior: not "is this good?" but "here are the moves a good X makes."
Draft — the writer composes in the canvas. The AI offers prompts, completions, and revisions on request. AI offers · adapt, edit, or ignore. The byline is the writer's.
Self-review — the writer measures the draft against the rubric's criteria. Strengths land in "glow" pills; gaps land in "grow" pills. The writer can act on this without ever submitting.
Submit — the draft is handed off to the review pipeline. Process data (revision count, pause distribution, AI offers accepted) is bundled with the artifact under opt-in consent.
Score — AES (Automated Essay Scoring) runs against the rubric. An instructor or TA reviews the same submission. If the AES-vs-human delta exceeds the published threshold, a third rater is invoked automatically — see § 4.
Feedback — annotated reasons land back in the canvas, tied to specific rubric traits. Glow and grow are kept distinct; the writer is never given a single number with no story behind it.
Revise — the writer returns to Draft with the feedback in view. The loop continues until the writer chooses to publish.
Why a rubric per genre
Genre conventions are not interchangeable. An argumentative essay is judged by claim, evidence, warrant, counter, refutation. A haiku is judged by syllable structure, juxtaposition (kireji), seasonal reference (kigo), and emotional resonance. A research abstract is judged by purpose, methods, results, conclusion — and a 250-word ceiling. A single rubric for "good writing" cannot evaluate any of these well.
Scriptorium ships 117 genres organized into 16 categories — Academic Writing, Essays & Analysis, Workplace Communication, Technical & Documentation, Journalism & Media, Genre Fiction, Poetry & Verse, Scripts & Drama, Personal & Life Writing, and others. Each genre has a versioned baseline rubric in data/genre-rubrics/ (repo) and _db/genre_rubrics/ (shared GCS prefix, cross-tenant). Writers and instructors fork these into per-tenant drafts; the shared baseline is the structural authority.
Two roles, one artifact
Structural prior · for drafting
Before a single word is written, the rubric tells the writer what good X looks like — the moves a strong instance of the genre makes. The writer can reference it during composition and use it for self-review.
Scoring instrument · for review
After submission, the same rubric is used by AES and human raters to score the draft. Construct authority lives with the human who authored the rubric — not with the model's prior beliefs about quality.
Versioning & lineage
Rubrics are versioned with a lineage trail. Every published version is immutable; new versions snapshot under /v{ver}.json with the prior version's hash referenced. An "AI complete" pass can extend a rubric by adding criteria; the human author signs off before the new version is published. The history tells you why a criterion was added or retired — see docs/rubric-lifecycle.md for state transitions.
How AI is used
Scriptorium speaks to multiple LLM providers through one gateway: server/lib/llm.js. The default fallback chain is anthropic:claude-sonnet-4 → openai:gpt-4o → google:gemini-2.5-pro; a single failure flips to the next. No call site imports a provider SDK directly, so adding or swapping a provider is a config change, not a refactor.
Three things never change
- AI offers; the human decides. Every AI output ships with the same line: adapt, edit, or ignore. The writer's name is on the byline. The AI's name is not.
- Outputs are grounded in the rubric. The model scores against the published rubric, not against its prior beliefs about quality. Construct authority lives with the human who authored the rubric.
- Detection outputs are advisory. AI-text detectors run on submissions but never gate a score. False-positive rates against L2 writers and short responses are documented; students get a counter-evidence path.
Uncertainty is surfaced, not hidden
Multi-call self-consistency sampling is supported via LLM_N_SAMPLES (n > 1). When the samples disagree above the configured threshold, the panel surfaces low confidence rather than silently averaging the disagreement away. A score with low confidence routes to a third rater — see § 4.
The AI is a sparring partner, not a ghostwriter. It accelerates the writer's loop — it does not replace any step of it.
How we keep AI honest
A rubric is only useful if the score it produces is reliable. Three mechanisms keep the AES end of the loop trustworthy.
Calibration rounds
Instructors and TAs grade a shared anchor set; per-rater deviations are computed and shown in the Calibration Studio. Graders cannot release a batch until their κw (weighted Cohen's kappa) on the anchors clears the configured threshold. Auditors can participate as observers — their parallel scores flag rubric drift without influencing the human-AI kappa.
AES-vs-human reconciliation
Every submission is scored by both the AES and at least one human. When the delta exceeds the published threshold, a third rater is invoked automatically. The reconciliation rate (how often the AES and human agree closely enough to skip the third rater) is published as a condition of the AES being used — not buried in an internal dashboard.
Subgroup fairness audits
Standardized mean differences, demographic parity ratios, and per-subgroup κw are computed per release batch. The auditor role (lateral observer in the hierarchy; cannot be demoted by an instructor) approves or vetoes the release. Models cannot promote to production without clearing the eval set.
These mechanisms exist because AI scoring is fallible at the margins — particularly for L2 writers, short responses, and genres that reward unconventional choices. Naming the fallibility, measuring it, and gating release on it is more honest than pretending it isn't there.
The audit trail
Every state-changing operation in Scriptorium emits an xAPI 1.0.3 statement under our perp.scriptorium profile. Sign-ins, rubric publishes, score overrides, feedback releases, calibration anchors — all of it. The LRS is the audit trail; if Firestore is wiped, the system can be rebuilt by replaying the LRS against GCS.
The vocabulary includes verbs like initialized, loggedIn, submitted, scored, compared, released, denied, errored, calibrated, audited, configured. Every statement carries actor, verb, object, and structured extensions; the full schema is documented in sample/scriptorium-docs/03-xapi-lrs.md.
One source for failures, too
The xAPI bus also doubles as the bug tracker. Every uncaught exception, unhandled rejection, and 5xx response is logged as an xAPI errored statement with a structured payload (severity, fingerprint, route, user context). There is no Sentry, no Rollbar, no GitHub Issues mirror — one source of truth for both successful state changes and failures.
Storage & residency
GCS is the durable system of record
Per-tenant data lives under tenants/{tid}/...; cross-tenant reference data (genre rubric templates, locales, content cache, TTS cache) lives under _db/.... Firestore is used as a queryable index — never as the source of truth. The "if Firestore is wiped, replay from GCS" property is a deliberate design choice: every artifact is versioned, immutable, and signable.
Privacy is regional
Hong Kong is primary; Singapore is failover; EU multi-region for EU students; mainland China region for mainland users. Data does not cross regions without explicit consent. Residency configuration lives in the Privacy & Residency surface and is gated to the governance roles.
Process data is opt-in
Keystroke timing, revision pauses, AI-offer acceptance — all of it is captured only with explicit consent and is never used for grading or model training. Revision history is preserved verbatim. Inline annotations are tied to specific rubric traits and classified as glow (strength) or grow (improvement).
Design philosophy · the ten principles
Drawn verbatim from docs/PHILOSOPHY.md — the document that grounds the contextual help assistant.
- AI augments, never replaces, human judgment. Every score is reviewable, every override is logged, every decision can be reproduced from immutable artifacts.
- Rubric-grounded scoring. Models score against published rubrics with anchored exemplars — not against the model's prior beliefs about quality. Construct authority lives with the human who authored the rubric.
- Reconciliation is the workflow, not an afterthought. When AES and a human disagree above the published threshold, a third rater is invoked. The reconciliation rate is published as a condition of legitimacy.
- Anonymity by default during grading. Pseudonyms are bound to (user × term × course); identity is reattached only at release time, with logged reason.
- Subgroup fairness is a release gate. Standardized mean differences, demographic parity ratios, and per-subgroup κw are computed per batch. Models cannot promote to production without passing the eval set.
- Detector outputs are advisory. AI-text detection never gates a score. False-positive rates against L2 writers and short responses are documented. Students get a counter-evidence path.
- GCS is the durable system of record. Firestore is a queryable index. If Firestore is wiped, the system can be rebuilt by replaying GCS — every artifact is versioned, immutable, and signable.
- xAPI is mandatory. Every state-changing operation emits a statement. The LRS is the audit trail. No exceptions.
- Privacy is regional. HK primary, SG failover, EU multi-region for EU students, mainland China region for mainland users. Data does not cross regions without explicit consent.
- Pedagogy first. Process data is captured under opt-in consent only and is never used for grading or model training. Revision history is preserved verbatim. Inline annotations are tied to specific traits, classified as glow (strength) or grow (improvement).
References
- Human Roles, Responsibilities & Accessibility — the role hierarchy, capability matrix, and where each gate lives in the codebase.
- The Genre Manual — every supported genre with its construct, writing context, evaluation rubric, and a five-step writer guide. (Prototype: Poetry & Verse fully rendered; rest rolling out incrementally.)
docs/PHILOSOPHY.md— the document that grounds the contextual help assistant; canonical source of the ten principles.docs/rubric-lifecycle.md— rubric state transitions (draft → published → curated → retired) and the xAPI events emitted at each.docs/genre-rubric-templates.md— the shared_db/genre_rubrics/prefix, schema, and the per-tenant copy lifecycle.docs/auth-and-roles.md— the user-management contract layered on top of the role hierarchy.sample/scriptorium-docs/— the originating spec for endpoint shapes, payloads, and xAPI verbs.