Most writing tools optimize for the output. Scriptorium optimizes for the instrument that measures the output. Every assignment ships with a peer-reviewed, calibration-tested rubric; every essay is scored against the same rubric that taught the student to write it; every rubric is approved by a lateral auditor role that has no management stake in the work. The rubric is the curriculum.
Signature differentiator · The rubric pipeline
A rubric earns the right to grade a single student
Most platforms treat the rubric as a checklist the instructor types in minutes before the assignment opens. Scriptorium treats it as a psychometric instrument that must clear 12 publish gates across an 11-step authoring wizard and a 5-gate validation pipeline. End-to-end, a fresh rubric burns 200+ LLM calls of automated quality assurance — inter-rater κ, swap tests, discriminability, robustness, sensitivity, anchor compliance, subgroup parity — before it can grade a single real student. This is the most expensive part of the system on purpose.
~108
Step 08 Calibration
+
~80
Validation pipeline
+
~25
Swap tests & descriptors
+
~10
Source generate · compliance
=
200+
LLM calls / rubric
①
CREATE · 11-step authoring wizard
RubricCreateView.jsx · 2 HERO steps
STEP 00
~5 calls
Choose source
Manual · AI-generate from genre prior · CFP paste · sample essay · fork · import JSON. Five on-ramps, one schema.
STEP 01
Context & purpose
Title, framework, course, stakes, intended use. Declare what the instrument is for.
STEP 02
Construct definition
Name and define the latent traits the rubric claims to measure. Theory first, items second.
STEP 03
Criteria & sub-criteria
Hierarchical traits with weights. Sub-construct sums must equal 100 — the wizard refuses to advance until the math holds.
STEP 04
Performance levels
Define L1–L5 in the abstract before writing per-cell descriptors.
STEP 05
HERO
Descriptor authoring
L1–L5 cell for every trait. Swap test ≥ 0.85 — LLM must tell adjacent levels apart from descriptors alone. ~25 calls
STEP 06
Anchored exemplars
Bind real student work to specific (trait, level) cells. Concrete anchors for abstract descriptors.
STEP 07
~6 calls
Anchor compliance & genre
Cross-check exemplars against the genre template. Are anchors typical of the genre?
STEP 08
HERO
Reliability calibration · κ_w
18 essays (12 senior-anchored + 6 engineered) scored by LLM, instructor blind, and senior. Three κ_w thresholds. ~108 calls
STEP 09
Subgroup parity
Cohen's d, demographic parity ratio, per-subgroup κ_w. Advisory at first publish, required after 200 graded submissions.
STEP 10
Governance & publish
Tier selection, sign-off, change log. Validation report attached; cross-tenant promotion needs coordinator + auditor co-sign.
↓ validation pipeline ↓
②
VALIDATE · 5 psychometric gates
RubricValidationWizard.jsx · ~80 calls · per-item fan-out
GATE 1 · SCHEMA
Well-formed?
0 LLM calls
Trait ids, level coverage, weights sum to 100, no missing descriptors. Pure function. Unforgiving.
GATE 2 · INDEPENDENCE
Orthogonal traits?
1 LLM call
Pairwise overlap matrix. Two traits measuring one construct would distort every downstream signal.
GATE 3 · DISCRIMINABILITY
Good vs bad?
~18 LLM calls
3 anchor samples (good · ord · bad) scored per-trait. Each trait must separate quality levels on its own dimension.
GATE 4 · ROBUSTNESS
Same paper, same score?
~18 LLM calls
Same essay scored N times. Vague descriptors surface as variance. Pure-function gate on cached scores.
GATE 5 · SENSITIVITY
Edits move scores?
~42 LLM calls
Perturb samples (paraphrase · weaken evidence · shuffle structure) and re-score. Rubric must move scores the way it claims to.
↓ all 12 gates must clear ↓
③
PUBLISH · 12-gate contract
all green · or no rubric ships
Gate to publish:
Out-of-scope ≥ 1
Sub-construct sum = 100
Trait coverage
No tag redundancy
Descriptors L3/L4/L5
Swap test ≥ 0.85
Exemplars L3/L4/L5
Anchor compliance set
κ_w(LLM,senior) ≥ 0.70
Construct alignment ≥ 0.80
Subgroup audit (deferred)
Governance sign-off
Our philosophy
Most writing coaches focus on producing better essays. Scriptorium focuses on producing better assessments of writing. Six commitments shape every design decision — from the rubric wizard's "ten gates" to the way the admin's view-as-role mode behaves.
Rubric-First Pedagogy
The rubric is the curriculum — negotiate it before a word is written
If the rubric is wrong, every downstream score is noise. The rubric wizard puts ten gates between idea and ink — genre fit, dimension balance, anchor calibration, perturbation tests, peer fork, auditor blessing — because rigor at the instrument level pays for itself in every grading cycle that follows.
117 Genres · 117 Rubrics
A sonnet is not a startup pitch — genre-shaped writing needs genre-shaped assessment
Each of 117 genres ships with its own rubric template, dimension weights, AI prior, and pedagogical scaffolding. Templates live in a shared _db/genre_rubrics/ library; tenants get per-course copies they can fork, never edits to the master. The system always knows what kind of writing it is grading.
Lateral Audit
The chain that makes the work is not the chain that blesses it
Curated rubric promotion, residency, and fairness audits all flow through the auditor role — rank above instructor, but canManage:false. No one in the instructional hierarchy can approve their own rubric. External review is structurally separate, not a policy.
xAPI as System of Record
Every state-change is a learning record — including the bugs
Scores, grades, rubric edits, AI calls, console errors, and 5xx failures all emit xAPI statements under the perp.scriptorium profile. The LRS is authoritative; Firestore is a queryable index, never the source of truth. There is exactly one timeline — analytics, process data, and bug triage all read from it.
Living Schema Artifacts
AI generates the start; humans complete the lineage
"AI generate" is the seed; "AI complete" is the future. Every extension is recorded with the model, the prompt, and the human who accepted it. A rubric in week 30 is the auditable accumulation of every edit, model, and reviewer that touched it — not a snapshot, a lineage.
Faithful View-as-Role
Inspection cannot leak privilege — even for the admin
When the bootstrap admin views the app "as a student", they see exactly what a student sees. Every capability gate keys off activeRole — never sticky isAdmin. The role-switcher itself is the only legitimate isAdmin reader anywhere in the codebase.
Theory behind Scriptorium
🎯
Backward Design
Wiggins & McTighe, 1998
Start with the assessment, then design the task, then teach. The rubric wizard ships before the first prompt.
🧪
Construct Validity
Messick, 1989
A rubric measures what it claims to measure. Samples, baselines, and perturbation tests surface dimension drift before release.
📐
Inter-Rater Reliability
Cohen's κ, 1960
Two graders applying one rubric should agree. The calibration studio quantifies κ across TAs before grades release.
📊
Evidence-Centered Design
xAPI · Mislevy · ADL
Capture the process, not just the artifact. Every keystroke, AI call, and revision is a learning record.
Full theory & mechanism →
5
Validation Gates
schema · independence · discriminability · robustness · sensitivity
11
Wizard Steps
with reliability & parity built in
200+
LLM Calls
per rubric · seed to publish
117
Genres
each with its own rubric template
7
Roles
admin · coord · auditor · researcher · instructor · ta · student
Who it's for
- Draft across 117 genres — from haiku to literature review to AI video script — with per-genre scaffolding
- See the rubric you will be graded by, before you start writing
- AI assistance via the help assistant, selection menu, and block-refine — never opaque, always logged to the LRS
- Revision history captures process, not just product — every draft becomes evidence
- Peer review rounds with structured anchor prompts
- Inline annotations and itemized feedback tied to specific rubric dimensions
- Code-grading support for technical genres (with AST-aware rubrics)
- Track your own AI usage, model mix, and cost — full transparency
- UI in 11 languages including Arabic and Hebrew (full RTL)
- OCR & handwriting recognition for paper-first classrooms
- Author rubrics via the 10-gate wizard, starting from any of 117 genre templates
- Calibrate AES (auto-essay scoring) against your own anchor set before release
- Run validation: sample, baseline, perturb — surface dimension drift before grading
- Co-sign grades: TAs grade, instructors release (calibration-required by default)
- Answer-cluster view groups similar submissions for batch feedback
- Anonymous mode for blind grading; identity restored only on release
- Coord: tenant-level oversight; manages researchers, auditors, instructors, TAs, students
- Auditor: read-only lateral observer — the only role that approves curated rubrics & residency
- Bias & fairness audits across cohorts, sections, demographics
- LMS / SSO via OAuth gateway; multi-tenant GCS isolation
- Privacy & residency controls (admin-owned, auditor-approved)
- Model governance: per-tenant LLM allow-lists and fallback chains
- Raw xAPI stream — every statement under the
perp.scriptorium profile
- Process-data view: keystroke-level revision telemetry, AI-interaction sequences
- IRB-scoped access via the researcher role (lateral, sanctioned by coord)
- Cross-cohort fairness statistics with configurable demographic slices
- LLM call ledger: provider, model, tokens, latency, cost per interaction
- Genre-rubric lineage history for diachronic instrument studies
What's inside
✍️
Creative Writing Studio
Drafting surface with selection-AI, block-refine, and per-genre scaffolding tied to the active rubric
📐
Rubric Studio
10-gate wizard that authors, AI-completes, and lineage-tracks rubrics from genre templates
🎯
Calibration Studio
Multi-grader κ analysis on anchor sets before AES is trusted or grades released
⚖️
Bias & Fairness Audit
Cohort, section, and demographic slicing — flags disparate impact in scoring distributions
📥
Inbox & Grading Queue
Role-tuned queue — instructor releases, TA grades, student tracks own submissions
🕵️
Anonymous Mode
Author identities masked at grading time; restored only at release. Audit-trail preserved throughout
💬
Help Assistant Live
Floating "?" chip grounded in the DOM under your cursor + the project philosophy doc — tour guide, not coach
📡
xAPI Stream & Bug LRS
Every state change emits an xAPI statement — including uncaught errors. Single source of truth, no parallel tracker
How it works
01
Instructor authors the rubric
Pick a genre template from the 117-genre library, walk the 10-gate wizard, run AI-complete passes, validate with sample/baseline/perturb, save a versioned draft.
02
Student drafts within it
The rubric is visible in the writing studio. AI assistance is logged. Revisions snapshot to the LRS. Peer review and inline annotation anchor on rubric dimensions.
03
AES + human grader score
AES runs against the calibrated rubric. TA grades under instructor calibration. Instructor co-signs before release. Every score is dimension-itemized and explainable.
04
Auditor reviews the instrument
Fairness audit, rubric integrity, curated promotion — all reviewed by the lateral auditor role. The chain that approves the instrument is structurally separate from the chain that uses it.
Key differentiators
Rubric-First
Instrument
Before Output
Typical writing tools
Generate suggestions, autocorrect, polish output. The rubric is an afterthought — usually a flat checklist typed in minutes before the assignment opens and applied after the essay is done.
Scriptorium 11-step wizard
The rubric is built first through a structured pipeline: source → context → constructs → criteria → levels → descriptors → exemplars → genre compliance → reliability calibration → subgroup parity → publish. The author can't reach "publish" until inter-rater reliability and demographic parity both clear on the anchor set.
12 Gates · 200+ calls
Psychometric
Validation
Industry baseline
Rubrics ship un-validated. If two graders disagree, that's flagged as "rater error" — not a defect in the instrument. Discriminability, robustness, sensitivity, and inter-rater κ are concepts from the psychometrics literature, not features of the platform.
Scriptorium 12-gate contract
Before any student is graded, a rubric burns 200+ LLM calls: ~108 on Step 8 reliability calibration alone (18 essays, three rater pairs, quadratic-weighted κ ≥ 0.70), plus the 5-gate validation pipeline, swap tests, anchor compliance, and subgroup parity. When a gate fails, the system returns per-trait edit suggestions — not just a verdict. The rubric earns the right to grade.
Lateral Audit
Auditor Sits
Outside the Chain
Conventional admin sign-off
The same person who runs the institution approves the rubrics that grade its students. Quality review collapses into management authority.
Scriptorium auditor role
Rank 70, canManage:false. Sits above instructor in rank but has zero management authority. The only role that can approve curated rubrics or residency. Audit is structural, not procedural.
Genre-Native
117 Templates
Not One Rubric
Generic essay scorers
One rubric framework — usually "clarity / organization / evidence / mechanics" — applied to every genre, every level. A haiku is judged like a research paper.
Scriptorium genre library
Each genre ships with its own template — dimensions, weights, AI prior, and pedagogical scaffolding. Tenants fork per-course copies; the master library evolves through auditor-approved curation.
One Timeline
xAPI Is
The Source
Typical SaaS architecture
Analytics in Mixpanel, errors in Sentry, learning records in a separate LRS, the database is the source of truth. Reconciling four systems to answer one question.
Scriptorium LRS-of-record
Scores, grades, AI calls, console errors, and 5xx failures all emit statements under perp.scriptorium. The LRS is authoritative; Firestore is a queryable index. Analytics, process data, and bug triage all read from one timeline.
Technology
3 LLM Providers · Anthropic · OpenAI · Google
Per-provider concurrency gate + 429 retry
Async AI-review jobs · jobId + poll
LLM response cache · prompt-hash · GCS
Capability-gated authz · role matrix
Research-papers corpus · Drive import
Google Cloud Storage · per-tenant prefixes
xAPI / LRS · perp.scriptorium profile
OAuth gateway (oauth.xiangenhu.info)
i18n hash-based · 11 languages incl. RTL
pdfjs · pdf-parse · mammoth · pdfkit
OpenAI TTS · cached to GCS
Impact
Scriptorium is not measured by essays produced but by the quality of the judgments it makes about writing. Every architectural commitment above — psychometric gates, lateral audit, immutable scoring, one-timeline xAPI — exists to produce six outcomes an ordinary writing tool cannot claim.
Fairer assessment
A student is graded by an instrument that was tested for bias — and a score, once given, never silently changes
Subgroup parity (Cohen's d, demographic parity ratio, per-subgroup κ_w) is a publish gate, and scoring is immutable: an essay scored against v2 keeps that scoring forever. Revisions ship as new versions and only apply going forward. Fairness is a structural contract, not a policy statement.
Defensible grades
Reliability is quantified before grades release — disagreement is a measured number, not a shrug
Step 08 burns ~108 LLM calls on quadratic-weighted κ across three rater pairs on an 18-essay anchor set. When a program says "our grading is consistent," Scriptorium can produce the κ that proves it — per rubric, per subgroup, before a single real student is touched.
Accountable AI
Every model call is on the record; the help bot guides but never ghost-writes
Provider, model, tokens, latency, and cost land in the xAPI ledger under perp.scriptorium for every interaction. AI assistance is visible to instructor, auditor, and researcher alike — academic integrity is preserved by transparency, not by pretending AI isn't in the room.
A research instrument
Process-level telemetry and rubric lineage turn a classroom into a study
Keystroke-level revision, AI-interaction sequences, and diachronic rubric lineage give IRB-scoped researchers a raw stream for writing-analytics and psychometric work — the same data that grades the student advances the science of how writing is taught and measured.
Institutional memory
Nothing is a black box or a one-off — the record is durable and reproducible
GCS is the system of record; every per-trait review persists the moment it lands, rubrics are versioned with an auditable lineage of model, prompt, and human sign-off. A rubric in week 30 is the accountable accumulation of everyone who touched it — recoverable after any restart, tab close, or network loss.
Wider reach
Paper-first and non-English classrooms are first-class, not afterthoughts
UI in 11 languages including full RTL (Arabic, Hebrew), OCR and handwriting recognition for hand-written submissions, and cached TTS for audio access. The assessment rigor above reaches classrooms that browser-only, English-only tools leave behind.