What the work involves

You receive benchmark tasks built in the SWE-Bench mould: a real repository at a pinned commit, an issue statement, a reference ("gold") patch, and FAIL_TO_PASS / PASS_TO_FAIL test sets executed inside a container. Your job is to decide whether that task actually measures what it claims to. That means reading the issue for underspecification, checking the gold patch against the tests rather than against your own taste, running the harness to confirm the failing tests fail before and pass after, and hunting for the ways a task quietly breaks: tests that assert on the exact diff, hints or file paths leaked into the problem statement, network-dependent installs, flaky or time-sensitive assertions, and grading that a model can satisfy without solving the issue.

Output is written, not just a score. Each audit typically carries a per-criterion rubric judgment, a reproduction log, and a concrete disposition — accept, accept with a specified edit to the test set or problem statement, or reject with the failure mode named. Volume varies by task complexity; a straightforward single-file fix may take under an hour, a multi-package task with a fragile install can take a full session.

What the platform screens for

  • Provable engineering history. Mercor's screen leans on verifiable artifacts: merged PRs, commit history, maintainer or triage roles. Have URLs ready.
  • Harness literacy. Expect probes on Docker isolation, dependency pinning, pytest collection behaviour, and how you'd tell a genuinely flaky test from an environment problem.
  • Adversarial instinct. Scenarios where a task "passes" but is exploitable. The screen is looking for whether you reach for leakage and reward hacking unprompted.
  • Calibration in writing. Whether you can defend a reject decision in a few sentences without hedging or moralising about code style.

Logistics

Fully remote and asynchronous, contract engagement, work claimed from a queue rather than assigned on a schedule. Observed pay for this band is $70–90/hr, rated by task or by hour depending on the project — not guaranteed, and subject to the lab's own tiering. Most contributors run 10–25 hours a week; sustained throughput matters more than any particular time zone, though a few calibration syncs may land in US hours. You will need a machine that can build and run Docker images comfortably.