Research reading
From guidelines to good annotations.
SOURCE AUTHOR / NIKOLAI LIUBIMOV
Notes on “Rubric Design: The Missing Layer Between Guidelines and Good Annotations.” These are report summaries, not the complete original articles.
Two annotators can understand “high quality” and still judge the same output differently. They may attend to different evidence or apply different thresholds. Clear guidelines alone do not resolve that problem.
The source article describes a method for translating evaluation requirements into a sequence of observations and judgments. A decision graph makes that reasoning inspectable, and synthetic preflight testing helps identify ambiguity before the human pilot.
Observable evidence should come before an overall preference. What did the output do? Which requirements did it meet? Where did it fail? These questions make a later judgment easier to interpret.
When measuring taste, disagreement is valid data. This report therefore reserves fields for preference distributions, agreement, and rubric dimensions. A single winner cannot capture all of those signals.
Explore the study results →