What the work involves

You receive a set of prompts drawn from genuine litigation and disputes questions — procedural timing under the Code of Civil Procedure, evidence and document production, provisional remedies, appellate posture, settlement mechanics, enforcement. You run the same prompt through two LLM platforms, read both answers closely, and score each against a standardized rubric the client provides. Every evaluation ends with concise written feedback: what the model got right, where it drifted, and what a practising bengoshi would have flagged as wrong or dangerous to rely on.

The judgment being paid for is not general legal knowledge. It is the ability to spot the specific failure modes of a model writing about Japanese law: citing a repealed provision, importing US-style discovery into a Japanese procedural context, confusing 仮処分 with 仮差押え, stating a filing deadline confidently but incorrectly, or producing an answer that is textually accurate yet useless as practical advice. Scores must be defensible and consistent across batches, which is why the rubric matters more than personal style.

What the platform screens for

  • Verifiable practice history — where you were qualified, which firm, which contentious practice, what kinds of matters. Vague answers stall the screen.
  • Law-firm weighting — at least 2 years (3+ preferred) at a firm in litigation, disputes or contentious work. In-house is acceptable only if you moved within the last two years after firm experience.
  • Procedural depth under follow-up — the screener will push past your first answer on a Japanese procedural point.
  • Rubric discipline — evidence you can apply someone else's scoring criteria rather than substituting your own instincts.
  • Written Japanese precision — feedback is read by people calibrating the benchmark, so it must be tight and unambiguous.

Logistics

Fully remote and asynchronous. The initial commitment is roughly 10 hours, with strong potential for continued batches if your scoring holds up against calibration. Work is done on your own schedule against deadlines, and everything is covered by NDA. Observed pay for this listing is $140–150/hr; rates on Mercor vary by project and are not guaranteed.