Software Engineer — Agentic Search Systems

$80–150/taskrate as listed

Mercor is screening engineers who have owned retrieval or relevance on production search systems, including agentic search built on LLMs. The entry point is a 25-minute conversational interview about how you evaluate search quality — no coding and no take-home.

Summary, eligibility guidance and screening preparation are prepared by Sidequest from the platform's posting. Check the source listing before applying.

What the work involves

This is an expert-sourcing pipeline for search engineering judgment, not a generic coding gig. Tasks in this category typically center on evaluating and stress-testing agentic retrieval behavior: reading a query and an agent's tool-call trace, judging whether the retrieval step was the right call, deciding whether a result set actually answers the user's intent, writing rubrics for relevance, and constructing hard cases that expose failure modes like query rewriting drift, over-fetching, stale indexes, or reranker overconfidence. Observed pay is $80–150 per task, which varies with task length and complexity and is not guaranteed.

The screening interview itself is 25 minutes and conversational. Mercor is listening for specifics: what system you owned, what metric you moved, how you knew it moved, and what you gave up to move it. Vague ownership claims fall apart under follow-up, so the interviewer will press on details — offline versus online metric disagreements, how you built eval sets, what your ground truth was, why NDCG or recall@k was or wasn't the right lens.

What the platform screens for

Direct ownership of relevance, retrieval, or ranking on a system with real users
A working theory of measurement: how you quantified quality when labels were scarce or noisy
Hands-on exposure to agentic search — embeddings plus tool use, multi-hop retrieval, query planning, RAG pipelines in production
Data infrastructure fluency: indexing, freshness, throughput, cost tradeoffs at scale
Ability to explain a messy tradeoff clearly and without polish

Logistics

Fully remote and asynchronous. The interview is self-scheduled and takes 25 minutes with no coding component and no take-home. Standout candidates are invited to a paid 30-minute live conversation with the team, observed at $200 paid on completion of the call. Ongoing task work is contract-based with self-selected volume; most contributors treat it as part-time alongside a primary role.

What they look for

Production experience owning search relevance, retrieval, or ranking
Search evaluation methodology (offline eval sets, A/B testing, NDCG/recall/precision tradeoffs)
Agentic search or RAG systems built with LLMs and tool use
Data infrastructure for search: indexing, embeddings, freshness, and scaling
Clear technical communication about tradeoffs and failure modes
Software engineering background (Python or similar) with distributed systems exposure

Before you apply

Would you pass this screen?

Drawn from what this listing actually requires. Answered in your browser — nothing is sent to us, and we do not store your answers.

Have you shipped a search or retrieval system to production that real users or customers depended on?
Production search system · required
Have you directly owned or led work on the ranking, relevance, or retrieval strategy of a production search system?
Ownership of relevance or ranking · required
Have you built or shipped a system that uses LLMs, agents, or retrieval-augmented generation (RAG) in production?
Experience with agentic or LLM-based systems · preferred
Have you designed metrics or evaluation frameworks to measure search quality or system success?
Evaluation and metrics design · preferred

0 of 4 answered

Included with this listing

Interview prep — 12 questions

Matched to what this platform's screening actually measures. Answer each out loud in 60–90 seconds: claim → concrete example → implication.

What it tests: Whether the candidate has genuine ownership experience with production search, not theoretical knowledge; whether they can articulate the actual user need and constraints that shaped the system design.
A strong answer: A specific system (e.g., search ranking at Uber, retrieval pipeline at Notion, agent search at a startup) with concrete user base size or criticality, the problem it solved (latency, relevance, freshness, scale), and how they knew it mattered—user volume, query volume, or business metric.
Where people lose it: Describing a demo, a course project, or a feature touching search without owning the system's quality or its real-world performance—or staying so high-level ('I worked on our search team') that you can't be pinned down on what you actually shipped.