LLM Annotation and Review Platform

A multi-user workbench for human annotation of LLM output. A batch of prompts, their candidate responses, and the rubric judgements an automated judge produced for those responses are loaded into PostgreSQL. Annotators are then handed one prompt at a time, score each response, and correct each rubric judgement. A second pass by a reviewer covers anything the first pass skipped or flagged. The corrected batch is exported back to JSONL.

The interface is Gradio; all coordination between concurrent annotators is done in the database, so the application itself holds no work queue in memory.

Architecture

flowchart TD
    U[JSONL batch upload] --> ING[insert_file_data]
    ING --> DB[(PostgreSQL)]
    A[Annotator] --> UI[Gradio app]
    R[Reviewer] --> UI
    ADM[Administrator] --> UI
    UI -->|initialize_response_scores<br/>FOR UPDATE SKIP LOCKED| DB
    UI -->|update_response_scores<br/>update_judgements| DB
    UI -->|update_prompt_status| DB
    DB --> REP[Dashboard, productivity report,<br/>project summary, release WIP]
    DB --> EXP[export_to_jsonl]

Loading

Most of the concurrency-sensitive logic lives in PL/pgSQL rather than in Python. The function bodies are in annotation_platform/procedures.py; the Python that calls them is in annotation_platform/workflow.py.

Data model

Created by annotation_platform/schema.py, applied on start-up by annotation_platform.db.setup_database and after a reset by scripts/init_db.py.

Table	Purpose
users	username, password_hash, user_role (admin, creator, reviewer)
files	one row per uploaded JSONL batch
prompts	prompt text, domain, task, free-form meta_data, plus the whole workflow state: phase, status, create_user, review_user, start and end timestamps, skip reason and skip category for each phase
responses	candidate responses, many per prompt
judgements	the judge's original rubric, score and reason, many per response
labelled_responses	the annotator's overall score for a response
labelled_judgements	the annotator's corrected score and reason for a judgement
reviewed_responses, reviewed_judgements	created and joined by the export query, reserved for a separate review-pass record; the current UI does not write to them
archived_files	batches hidden from the file picker

A prompt moves through two axes. phase is create or review; status is yts (yet to start), wip, done or skip. The pair decides who may claim the row next.

Roles and workflow

Three roles exist in users.user_role, and a person picks a task (Create or Review) at login.

• Creator — claims a prompt in phase create, scores each of its three responses 1-5, then corrects the rubric judgements attached to each response. Finishing marks the prompt done. Skipping records a category and a reason and pushes the prompt into phase review with status yts.
• Reviewer — claims a prompt in phase review, i.e. one a creator skipped or flagged, sees the earlier skip category and reason, and either completes it (done) or skips it (skip).
• Administrator — uploads JSONL batches, exports completed ones, archives batches, adds users and changes credentials, and reads the reporting tabs: per-file dashboard, per-user productivity report, project summary by skip category, and Release WIP, which returns prompts stuck in wip to yts so another worker can pick them up.

The administrative tabs are shown when the logged-in username equals config.ADMIN_USERNAME; user_role is stored but is not what gates the UI. See Security notes below.

Work handout under concurrency

Handing the same prompt to two annotators would waste the second one's time and produce conflicting labels. The initialize_response_scores stored procedure avoids this without an application-level queue:

UPDATE prompts
SET create_user = p_create_user, status = 'wip', create_start_time = NOW()
WHERE id = (
    SELECT id
    FROM prompts
    WHERE status = 'yts' AND phase = 'create' AND file_id = v_file_id
    ORDER BY id ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED
)
RETURNING id INTO v_prompt_id;

FOR UPDATE SKIP LOCKED makes each concurrent session step over rows another session has already locked, so N annotators hitting Next at the same instant receive N different prompts with no retry loop and no serialisation failure. The same procedure first looks for a wip row already owned by that user, so a browser refresh resumes the prompt in hand rather than abandoning it. Its single returned row also carries the prompt, its three responses and two judgements per response, so claiming the work and loading the screen is one round trip.

Rubric scoring

Each response arrives with judgements of the form (rubric, score, reason) — for example Logical Reasoning, 4, and a sentence justifying it. Ingestion keeps at most two judgements per response and only those sharing a single rubric, picked from a random rotation of the list, so every annotator sees a comparable two-judgement screen.

The annotator supplies two things:

1. an overall 1-5 score per response, written to labelled_responses;
2. a corrected score and reason for each judgement, written to labelled_judgements.

Originals are never overwritten. When a prompt is served again the query COALESCEs the corrected value over the original, and the export emits both, as judger_N_score and judger_N_updated_score.

Skips are categorised — the categories are the single list config.SKIP_CATEGORIES — and the project summary counts prompts by category.

Stack

• Python 3.10+
• Gradio for the interface
• PostgreSQL, driven through psycopg2, with PL/pgSQL stored procedures
• pandas for the reporting tables, tabulate for the local table dump in annotation_platform.db.show_tables

Repository layout

app.py                              entry point: schema, procedures, launch
annotation_platform/
    config.py                       environment-driven settings and vocabulary
    schema.py                       table definitions and the reset statements
    db.py                           connections, schema bootstrap, table dump
    procedures.py                   the PL/pgSQL function bodies
    workflow.py                     claim a prompt, save scores, skip, advance
    reports.py                      dashboard, productivity, summary, WIP release
    ingest.py                       JSONL upload into the relational store
    export.py                       completed batches back out to JSONL
    auth.py                         credential hashing, login, user administration
    ui.py                           the Gradio Blocks
scripts/init_db.py                  destructive reset plus the first administrator
standalone/score_responses.py       single-user prototype: response scoring
standalone/correct_judgements.py    single-user prototype: judgement correction
examples/sample_batch.jsonl         three synthetic records in the input format

Running it

PostgreSQL must be reachable and the target database must already exist.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

createdb llm_annotation                 # or any name you put in DATABASE_URL
cp .env.example .env                    # then fill in DATABASE_URL and the admin secret

python scripts/init_db.py               # drops and recreates everything, seeds one admin
python app.py                           # serves on APP_HOST:APP_PORT

scripts/init_db.py is destructive: it drops every table and function in the current schema. app.py is not — it applies CREATE TABLE IF NOT EXISTS and CREATE OR REPLACE FUNCTION only, so it is safe to restart against live data.

Log in as the administrator, upload examples/sample_batch.jsonl from the Admin Panel, create a creator account in the same panel, then log in as that account to annotate the batch.

Configuration

Every value is read from the environment by annotation_platform/config.py.

Variable	Meaning	Default
DATABASE_URL	libpq connection URI for PostgreSQL	postgresql://localhost:5432/llm_annotation
BOOTSTRAP_ADMIN_USERNAME	administrator created by scripts/init_db.py	admin
BOOTSTRAP_ADMIN_PASSWORD	that account's initial secret; the script refuses to run without it	none
APP_HOST	interface Gradio binds to	127.0.0.1
APP_PORT	port Gradio serves on	7860
APP_SHARE	create a public Gradio tunnel	false
APP_DEBUG	Gradio debug mode	false
SAMPLE_DATA_PATH	JSONL batch the standalone prototypes read	examples/sample_batch.jsonl
SCORED_OUTPUT_PATH	where standalone/score_responses.py appends	scored_responses.jsonl
CORRECTED_OUTPUT_PATH	where standalone/correct_judgements.py appends	corrected_judgements.jsonl

JSONL format

One JSON object per line. examples/sample_batch.jsonl holds three records in this shape:

{
  "prompt": "Which planet in the Solar System has the shortest day?",
  "responses": ["...", "...", "..."],
  "per_response_judgements": [
    [{"rubric": "Logical Reasoning", "score": 5, "reason": "..."}, "..."],
    ["..."],
    ["..."]
  ],
  "meta": {"prompt_domain": "general knowledge", "prompt_task": "factual recall"}
}

per_response_judgements[i] holds the judgements for responses[i]. meta.prompt_domain and meta.prompt_task land in the domain and task columns; the whole meta object is kept in meta_data.

The export writes one object per prompt, carrying line_id, question, responses, the annotator scores, the skip flag and reason, and a judger_responses list holding each rubric with both the original and the corrected score and reason.

Standalone prototypes

standalone/ holds the two single-user precursors to the platform: one for response scoring, one for judgement correction. They need no database and no login, read the same JSONL format, and append their output to a second file. They are kept because they are the clearest small statement of what the two annotation screens have to do.

python standalone/score_responses.py
python standalone/correct_judgements.py

Security notes

This was built for a trusted internal annotation team behind a private network, and two properties of that design are worth stating plainly before anyone exposes it more widely.

• Authorisation is by username, not by role. users.user_role is recorded but the administrative tabs are unlocked by comparing the logged-in name with config.ADMIN_USERNAME. Any account created with that name is an administrator. Gate on user_role before putting this on a shared network.
• Some reporting queries interpolate their filters. The date-range reports build their SQL with f-strings over values that come from fixed dropdowns, and the skip handlers interpolate the task name into a column name. The values cannot currently be chosen freely from the UI, but the queries should be parameterised before any of those inputs becomes free text.
• Credentials are stored as salted scrypt digests (annotation_platform/auth.py) and DATABASE_URL never leaves the environment. There is no session expiry, no rate limiting on login, and no audit log of authentication events (the prompts table does record who did what, and when).
• APP_SHARE defaults to false. Turning it on publishes a tunnel from the public internet to the machine running the app.

Status

Built in 2024 as client work and published in de-identified form. Names, credentials and sample data from the original engagement have been removed; the sample batch in examples/ was written for this repository.

Licence

PolyForm Noncommercial License 1.0.0. See LICENSE.