What the work involves

You write and review evaluation tasks that probe how well AI systems handle real engineering work — not toy exercises. In practice that means constructing problems drawn from the kinds of issues you already triage in open source: a failing build under a specific toolchain, a subtle concurrency bug, a refactor that has to preserve public API behavior. Each task ships with test cases that actually discriminate — passing for a correct solution, failing for the plausible-but-wrong one — plus a rubric or reference solution that another engineer can audit.

A second strand of the work is analysis. You run models against real-world use cases in your language, catalogue where they break down (hallucinated APIs, mishandled build systems, plausible code that fails under load), and write that up for the research team in terms specific enough to act on. Clear written communication matters more here than it does in most contract engineering work.

What the platform screens for

  • Verifiable contribution history. Mercor's screen leans heavily on a public GitHub or equivalent profile with frequent, substantive commits to well-used projects in the last 12 months. Drive-by typo fixes and self-authored repos with no users won't clear the bar.
  • Depth in at least one language. Expect follow-ups on the internals of your ecosystem — GIL semantics, JVM memory model, undefined behavior in C, the TypeScript type system, whatever matches your claim.
  • Evaluation judgment. Can you tell a hard task from a merely long one, and write tests that fail for the right reasons?
  • Architecture and performance reasoning, since many tasks turn on scalability and code quality rather than correctness alone.

Logistics

Fully remote and asynchronous, structured as a talent network rather than a single fixed engagement — you're matched to projects as they come up. Commitments are typically part-time and flexible, often 10–20 hours per week, though some projects request more. Pay observed in the $100–150/hr range; actual offers vary by language, project, and screening outcome, and nothing here is a guarantee. Git, CI/CD fluency, and comfort operating with minimal oversight are assumed throughout.