Audit applied machine-learning tasks — experiment designs, model-selection arguments, evaluation setups — that a frontier lab uses to train and test its models, and write rubric-anchored feedback on where the methodology holds and where it breaks. Observed pay runs $70–90/hr on Mercor, contract and remote.

Summary, eligibility guidance and screening preparation are prepared by Sidequest from the platform's posting. Check the source listing before applying.

What the work involves

You receive applied ML tasks and their associated solutions or claims, and your job is to decide whether the methodology actually supports the conclusion. In practice that means reading a training and evaluation pipeline closely enough to spot the things that quietly invalidate results: a target-derived feature, a scaler fit before the split, group leakage across folds, a validation set that was consulted fifty times, a metric chosen because it flatters the model. You then write feedback against a rubric — specific, evidence-linked, and readable by someone who did not sit with the task for an hour.

Some items ask you to reproduce a result rather than reason about it. Others ask you to judge whether the task itself is well-posed: whether the dataset supports the question, whether the baseline is honest, whether the metric matches the stated objective. Expect to move between tabular gradient boosting, standard vision and sequence setups, and classical statistical evaluation. The listing is explicit that this is not LLM app-building or MLOps work; infrastructure fluency is not what is being measured.

What the screening looks for

Verifiable hands-on depth. Three or more years where you personally designed experiments and made model-selection calls, not where you consumed someone else's pipeline. Expect follow-ups that push on one specific project until the details either hold or don't.
Leakage and metric-gaming instinct. The strongest signal is naming failure modes you have actually caught, with the mechanism and the size of the effect on the reported number.
Calibration in critique. Whether you can separate "this is fatal" from "this is sloppy but the conclusion survives," and say so in writing without hedging into uselessness.
Framework fluency. PyTorch, TensorFlow, scikit-learn, XGBoost at the level where you know where each one makes leakage easy to commit.

Logistics

Remote and asynchronous, paid hourly against logged work. Throughput expectations tend to be stated in tasks or hours per week rather than fixed shifts, so evenings and weekends are workable. Most contributors treat it as part-time alongside a primary role; commitment often starts around 10–20 hours per week and adjusts with queue volume. Rates in this band are what has been observed for the posting, not a guarantee.

ML Challenge Task Auditor

What the work involves

What the screening looks for

Logistics

Interview prep — 10 questions