This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.
Apply Now
Role Description
Mercor is partnering with an AI research organization to engage independent evaluation contractors who can assess agentic tool-use quality—specifically whether a model calls search appropriately and rewrites user prompts into effective queries. This short term engagement focuses on high-accuracy judgments, clear rationales, and consistency across a large volume of model–rater traces. The work is well-suited for experts in information retrieval, prompt engineering, and product QA who prefer remote, asynchronous projects.
Key Responsibilities
- Review model interaction logs and decide if invoking the search tool was appropriate given the initial prompt and context.
- Evaluate the rewritten search query for clarity, specificity, and fidelity to the user’s intent.
- Provide concise, evidence-based rationales tied to rubric criteria; label edge cases and ambiguities.
- Score query quality (e.g., intent capture, keyword selection, operator use) and overall tool-use timing.
- Calibrate against gold examples; surface rubric gaps and propose improvements.
- Track decisions in a task portal; maintain high inter-rater agreement and throughput targets.
- Flag potentially sensitive content according to provided safety guidelines.
Qualifications
- Excellent written communication; able to justify decisions succinctly with references to instructions/rubrics.
- Meticulous attention to detail; comfort working independently with minimal oversight.
- Nice to have: familiarity with annotation tools, basic scripting (Python/SQL), and multilingual proficiency.
Requirements
- Remote and asynchronous—contractors set their own hours.
- Expected commitment: ~10–20 hours/week; flexible, project-based workload.
- Duration: initial 6–10 weeks with potential for additional task batches.
- Resource sharing and best-practice guides provided; support team available for inquiries.
Compensation & Contract Terms
- Compensation for completed work: estimated $45/hour equivalent or calibrated per-task rates based on complexity and geography (final rates confirmed before work begins).
- Payments for services rendered via platform (e.g., weekly through Stripe Connect, where available).
- Independent contractor engagement; project-based statement of work; no employment relationship or benefits implied.
Application Process
- Submit a brief profile (CV or LinkedIn) and note relevant evaluation/search experience.
- Complete a short skills check and sample grading exercise to demonstrate rubric alignment.
- If matched, you’ll sign a simple contract/NDA and receive task access details.
- Typical follow-up within a few days after the sample review.
Company Description
- Mercor is a talent marketplace connecting experts with leading AI labs and research groups.
- Backed by Benchmark, General Catalyst, Adam D’Angelo, Larry Summers, and Jack Dorsey.
- Thousands of professionals across domains—research, engineering, law, and creative—partner with Mercor on frontier AI projects.
Apply Now