Write and revise the rating guidelines that human raters use to score LLM outputs, turning vague program specs into instructions that hold up under edge cases. Work spans finance, retail, insurance, legal, and sports domains, as a W-2 employee of Cincinnatus LLC placed with a leading AI lab.

Summary, eligibility guidance and screening preparation are prepared by Sidequest from the platform's posting. Check the source listing before applying.

What the work involves

You sit between research teams who know what they want measured and raters who need to apply that intent consistently across thousands of examples. Day to day, that means interviewing program owners to extract the actual decision criteria behind an ambiguous spec, drafting rubrics and rating scales, building out edge-case tables and worked examples, and then stress-testing your own drafts for internal contradiction, undefined terms, and coverage gaps. When rater escalations or low inter-annotator agreement come back, you diagnose whether the problem is the guideline, the examples, or the task design — and revise accordingly.

Domains rotate. One sprint you may be defining what counts as a materially misleading financial claim; the next, what distinguishes acceptable from unacceptable legal-adjacent hedging, or how to score sports statistics accuracy. You are not expected to be a subject-matter expert in all of these, but you are expected to interrogate SMEs efficiently and convert their nuance into instructions a non-expert rater can apply.

What the platform screens for

Mercor's screen is AI-led and evidence-driven. Expect follow-up questions that push past general claims: reviewers want concrete before/after examples of specifications you disambiguated, the specific rubric structures you have used, and how you measured whether a guideline worked. Prior hands-on experience with GenAI/RLHF rating programs is weighted heavily — writing style guides or internal SOPs alone tends not to clear the bar. Career progression and writing samples matter; so does a plausible account of your availability.

Logistics

W-2 employment with Cincinnatus LLC (employer of record), placed with the client AI lab's extended workforce
Remote, with an expectation of at least 35 hours/week during weekday business hours — this is not an async side engagement
Observed pay band of $45–65/hr, varying with experience and program; not guaranteed
Collaborative rather than solo: regular working sessions with program leads and other SMEs

AI Rater Guidelines Writer (Linguist / Instructional Designer)

What the work involves

What the platform screens for

Logistics

Interview prep — 10 questions