The Rubric

The schema

The shape has three layers: the persistence envelope (id, ownership, status, validation report), the gradable core (traits, sub-criteria, levels, descriptors), and the optional metadata (framework, course, references, anchor samples). The smallest object the grader will operate on is below — everything else adds audit, governance, and validation context, but does not change the gradable core.

The minimum gradable shape

{
  "rid":          "r_a1b2c3d4e5f6",
  "title":        "Argumentative Essay — Disciplinary Reasoning",
  "totalPoints":  100,
  "weight":       100,

  "levels": {
    "count":  5,
    "labels": ["Insufficient", "Developing", "Adequate", "Proficient", "Distinctive"]
  },

  "traits": [
    {
      "id":          "thesis",
      "name":        "Thesis & Position",
      "description": "Clarity, specificity, and arguability of the central claim.",
      "weight":      20,
      "subCriteria": [
        {
          "question":    "Is the central claim explicit, specific, and arguable?",
          "weight":      0.45,
          "description": "An evaluator looks for one sentence the reader could disagree with…",
          "descriptorPriors": {
            "L1": "No identifiable thesis…",
            "L2": "A claim is present but generic…",
            "L3": "In addition to L2's claim, the claim is locatable in one sentence, names a specific aspect, and is arguable.",
            "L4": "In addition to L3's specific claim, the claim distinguishes itself from rivals.",
            "L5": "In addition to L4's distinction, the claim identifies a non-obvious tension whose resolution drives the essay."
          }
        }
      ]
    }
  ],

  "descriptorPriors": {
    "thesis": { "L1": "…", "L2": "…", "L3": "…", "L4": "…", "L5": "…" }
  }
}

Sub-criterion question is always a yes-question — never a noun phrase, never a statement. This forces the grader (or AI) to commit to an answer. description names the test the evaluator applies. descriptorPriors is the per-level anchor; the cumulative rule lives here.

Two layers of descriptors

Both traits and sub-criteria carry their own per-level descriptors. They serve different scopes.

Distributive evaluation

The pipeline fires one LLM call per trait, in parallel. The holistic score is computed after per-trait scores by weighted average — it does not feed back. So a trait descriptor may reference the paper, the genre, the assignment context, or what other writers in this form do; it may not reference another trait's score. This is a hard requirement, not a preference.

Field reference

Top-level fields

Trait fields · rubric.traits[i]

Sub-criterion fields · trait.subCriteria[i]

Lifecycle & transitions

A rubric moves through a small set of states, each with different visibility and editability. The only path from draft to a gradable status runs through the validation wizard.

State definitions

READABLE_STATUSES in server/lib/rubrics.js is the allow-list: validated · production-validated · hs-validated · published. A rubric outside this set is owner-private regardless of visibility.

Storage

Per CLAUDE.md, durable rubric state lives in GCS keyed by tenant. Per-tenant rubrics under tenants/{tid}/rubrics/; cross-tenant reference data (genre templates, the curated library) under the _db/ prefix.

gs://{bucket}/
├─ tenants/{tid}/rubrics/
│  ├─ {rid}.json                                      // canonical rubric
│  ├─ {rid}/draft-validation.json                     // in-flight validation work
│  └─ {rid}/exemplars/{trait}/L{level}/{sub_id}.json  // anchored exemplars
│
├─ _db/genre_rubrics/                                 // 117-genre template library
│  ├─ {id}.json                                       // HEAD
│  └─ {id}/v{templateVersion}.json                    // immutable snapshots
│
└─ _db/rubric_library/                                // curated cross-tenant rubrics
   └─ r-{n}.json

Validation tiers

The validation wizard walks a rubric through a tiered set of quality gates and promotes it to one of three validated statuses. Tiering is FDA-analogous — regulatory burden scales with consequence. Forcing every author through high-stakes rigor would deter use; allowing every author to ship at Core would harm students.

The gates

Each gate answers one psychometric question. Pure-function gates run on cached scores (no LLM cost, idempotent); LLM-gates fire calls. Filled dots show which tiers run each gate.

The Anchor Provenance gate requires LLM-generated anchors at Production tier (with cross-model separation to avoid circularity), but requires human-rated anchors with κ ≥ 0.61 at High-stakes. The CI gate is a fixture that runs against tests/fixtures/broken-rubric.json and asserts at least 3 of 5 Core gates fail — it catches gate-leniency regressions over time.

What the headline gates actually do

Structural Integrity

The completeness contract from §2 — missing trait id, no levels, sub weights don't sum to 1.0, leaked-descriptor subs. Zero LLM calls.

Disjointness

One LLM call returns a pairwise overlap matrix across trait descriptors; pairs above independenceThreshold are flagged. Measures overlap among the rubric text, not among scored data.

Discrimination

Score three anchors (good · ord · bad) × N traits, then check both gap (adjacent levels separate) and range coverage (good ≥ 4.0, bad ≤ 2.0 on a 1–5 scale). Reports point estimate ± SE and 95% CI per trait.

Test-Retest

Score one sample R times with the same rater; compute per-trait SD; gate on threshold. Reports per-trait ICC(2,1) with 95% CI — informational at Core, gating at High-stakes (robustnessICCMin = 0.75).

Sensitivity

For each (sample, trait), generate a perturbation that damages that trait specifically; rescore; verify the targeted drop, that other traits don't drift, and that neutral edits don't move scores.

Cross-Rater

Re-score with a second rater (default Anthropic + OpenAI pairing). Compute per-trait ICC and Krippendorff's α with ordinal weights. Default pass thresholds: ICC ≥ 0.60, α ≥ 0.667.

Fairness / DIF

Generate matched-content variants differing only in surface markers (L2 phrasing, register shifts, dialectal markers); rescore; check mean drift per (variant family × trait) is below threshold. A lite DIF check — real DIF on cohort data is a separate workstream.

Wizard & architecture

The validation wizard is a tier-dependent flow (7 steps at Core, 9 at Production, 11 at High-stakes) opened from Rubric Studio. The architecture is forced by one binding constraint: codespace and reverse-proxy gateways enforce a 30–60 second per-request timeout, and Claude Opus on a long writing routinely takes 15–45 s per scoring call. Three crosscutting design decisions follow.

Wizard step rail

                              Core   Production   High-stakes
1 · Configure                  ✓        ✓             ✓     // tier, model, thresholds
2 · Independence               ✓        ✓             ✓     // pairwise overlap matrix
3 · Anchor Provenance          —        ✓             ✓     // LLM gen / upload / hybrid
4 · Samples                    ✓        ✓             ✓     // generate or load anchors
5 · Discrimination             ✓        ✓             ✓     // score baseline + range coverage
6 · Test-Retest Reliability    ✓        ✓             ✓     // R-repeat scoring + ICC
7 · Cross-Rater Agreement      —        ✓             ✓     // re-score with alt rater
8 · Causal Sensitivity         ✓        ✓             ✓     // perturb + score
9 · Perturbation Fidelity      —        ✓             ✓     // audit subsample
10 · Fairness / DIF            —        —             ✓     // matched-content variants
11 · Save                      ✓        ✓             ✓     // persist report; promote status

Past steps are revisitable (rail shows ✓); forward navigation without a gate pass is blocked. Tier downgrade at save is silent (a Production-attempt that only completed Core gates saves at Core); tier upgrade reopens the wizard and runs only the new gates against the previously-saved data.

Reporting & freshness

Every saved validation produces a Report Card surfaced in Rubric Studio and the Library. It is the artifact downstream consumers (Calibration Studio, the Bias & Fairness Audit, Cohort Analytics) actually read. The Report Card always carries:

• The tier badge — Core, Production-validated, or HS-validated.
• The model + version stamp — e.g. scoring=anthropic:claude-opus-4-7@2026-04-15.
• The prompt hash — sha256:9c3f… — captured at run time.
• A gate-by-gate verdict table with ✓ / ✗ / N/A by tier.
• Per-trait statistics: mean, SE, 95% CI, ICC, Krippendorff's α.
• The anchor provenance summary, with rater metadata if human.
• Failure diagnoses + suggested edits (if any gate fails).
• A disclosures block — always shown — listing what this validation does not cover (content validity, criterion validity, real-cohort DIF, calibration to absolute scale, longitudinal model drift).

The freshness indicator

A validated rubric is not fresh forever. Anthropic, OpenAI, and Google all rev models; yesterday's validation does not hold for free. The Library shows a freshness state on every validated rubric: