How AI-Assisted Judging Works in Our Contests — and Where Humans Step In

When we tell people that AI helps judge our contests, we get two reactions. Some assume the AI does everything and the results are arbitrary. Others assume "AI" is a marketing word and a person is reading every entry. Neither is right, and the actual process is worth explaining because it affects whether you trust the outcome.

The problem judging has to solve

A popular contest on Sanora One might receive 800 entries. If each takes a human judge fifteen minutes to evaluate properly, that's 200 hours of judging — five full working weeks for one person. Nobody does that, which is why most large contests quietly rely on a first-pass filter that nobody talks about.

We'd rather talk about it.

Stage one: mechanical checks (fully automated)

Before anything is scored, an automated pass checks whether the entry is eligible at all:

• Was it submitted before the deadline?
• Is it the right format and size?
• Does it meet the stated minimums — word count, resolution, number of slides?
• Is it a duplicate of another entry, or a near-copy of something already public?
• Does it contain content that breaks our community rules?

Entries that fail here are rejected with a specific reason, and you're told which check failed. This stage is not judgement; it's a bouncer checking IDs.

Stage two: AI scoring (automated, but not final)

Eligible entries are scored by a language or vision model against a rubric written by the contest organiser. The rubric is published on the contest page, so you can see exactly what's being scored.

A typical rubric for a writing contest might weight:

• Relevance to the brief — 30%
• Structure and clarity — 25%
• Originality — 25%
• Technical quality (grammar, formatting) — 20%

The model reads each entry, scores each rubric line, and writes a two-to-three sentence justification for each score. Every entry gets this treatment — not just a sample.

Here's the important part: the AI score is used to rank, not to decide. Its job is to sort 800 entries into a rough order so human judges can spend their time where it matters.

Stage three: human review (the part that decides)

Human judges then review, at minimum:

• Every entry in the top 20% by AI score
• A random 10% sample from the remaining 80%
• Every entry that the AI flagged as "uncertain" — where its own confidence in the score was low
• Every entry a participant has asked us to review (see below)

The random sample is a safeguard. If human judges find that the AI ranked several strong entries in the bottom 80%, that's evidence the model is misreading the rubric, and we re-run the scoring with adjusted instructions or widen the human review band.

Human judges can move any entry up or down. Their decision overrides the AI score. Final rankings are set by humans.

Where AI fails, and how we compensate

We've seen models make predictable mistakes:

• Rewarding length. Longer entries often score higher on "thoroughness" even when they ramble. We now cap the benefit of length in the rubric.
• Missing cultural context. A joke or reference that lands perfectly for an Indian audience may read as a non-sequitur to a model trained mostly on Western text. Human judges from the relevant community catch this.
• Over-trusting confident writing. Assertive prose scores well even when it's wrong. Human subject-matter judges check facts.
• Penalising unconventional structure. A brilliant entry that ignores the expected format may be scored low on "structure". This is why the top-20% band is a floor, not a ceiling, for human review.

What you can do if you disagree

Every participant can request a human review of their entry, once per contest, within seven days of results being published. You'll see:

• Your AI score on each rubric line, with the model's justification
• Whether a human already reviewed your entry, and if so, their comments
• The option to write a short note explaining why you think the score is wrong

A judge who hasn't seen your entry before will review it fresh. If they change the outcome, the results are updated and everyone affected is notified. This happens more often than you might think — in our first year, roughly one in fifteen review requests led to a change.

Why we do it this way

It would be cheaper to let the AI decide. It would be more prestigious to claim every entry is read by an expert panel. We do neither, because the first is unfair and the second is usually a lie.

Splitting the work — machines for scale, people for judgement, and a transparent path to appeal — is slower and more expensive than either extreme. We think it's the only honest way to run a contest at scale, and we'd rather explain it than hide it.