AI Evaluation Rubric Reviewer
Mercor • Remote
Education
Not stated
Type
Hourly
Pay
$40-75/hr
Your rate depends on your experience and the task.
Listed
1d ago
Checked
Only apply if this role fits your skills and interests.
About this role
From the Mercor listing
The work
We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:
- Was the information available?
- Does the rubric accurately grade what the prompt asked for?
- Would a valid answer have been marked wrong?
- Is the rubric structure correct per the guidelines?
Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.
Must-haves
- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
- Can read a long agent transcript and find where it went wrong.
- Precise writing, and you check the data instead of trusting the reference answer.
How long hiring takes
On the platforms we refer people to, half of hires came within about 25 days of applying. Time from applying to hire:
Talent pool members
When your profile is a strong match, we can put you forward to Mercor. Set up your profile first if you haven't.
Set up your profile →Typical contract length
Across 10 Mercor Generalist listings we've tracked, contracts typically run about 6.8 weeks. Length varies by project.
Why this role
This AI Evaluation Rubric Reviewer role pays $40-75/hr and is fully remote, so you can do Generalist work for an AI lab from home.
Skills and categories
More jobs in these fields:
Related jobs
Enterprise Sales Expert (AE / Sales Leader)
Mercor • Generalist
$70 /hr
Creative & Media Expert
Mercor • Generalist
$90 /hr
Network Engineering & Simulation Expert — Hephaestus
Mercor • Generalist
$90 /hr
Loan Underwriter / Loan Officer — Paid Research Survey (US)
Mercor • Generalist
$50 flat, whole project
Verified reviews
Community reviews
Leave your review
Common questions
Does it cost money to apply to Mercor?
No, applying to and joining Mercor is free. Mercor charges the client a fee on top of your rate. If anyone asks you to pay to join, treat it as a scam.
Is Mercor for freelancers or full-time contractors?
Mercor places you with one client for a set contract, such as 'Python Tutor for 3 months'. Most roles are steady contract work rather than one-off tasks.
What does task-based AI training work look like?
Short, hands-on tasks such as recording videos, sorting images, rating AI answers or checking data. Most take 5 to 60 minutes each.
Can I choose my own hours?
Usually, yes. You log in when you want, pick up a task, finish it and submit it, with no meetings. Other people work from the same pool of tasks, so some days there may be no work until more is added.
What does Generalist work look like for an AI Evaluation Rubric Reviewer?
You check AI answers to Generalist questions, correct their errors and write expert reasoning. As an AI Evaluation Rubric Reviewer, you're hired for that judgment, so hands-on Generalist experience matters.
What specific skills does this listing call for?
Review & QA is named directly in the listing. If you don't have hands-on experience with this, expect the screening to test for it.
How much does this specific role pay?
This listing is posted at $40-75/hr, an hourly rate. Where you land in the range depends on your background and the assessment. Pay can change after we check the listing, so confirm the rate on the platform's application page.
What happens when I click Apply on this listing?
Apply opens Mercor's site, where you create an account and follow its sign-up steps.
How soon will I start working after applying to Mercor?
Usually after a few weeks. Applying puts you in Mercor's pool of candidates, and you start once a client, such as an AI lab, picks your profile.