AI Evaluation Rubric Auditor
Mercor • Remote
Education
Not stated
Type
Hourly
Pay
$60-85/hr
Your rate depends on your experience and the task.
Listed
1d ago
Checked
Only apply if this role fits your skills and interests.
About this role
From the Mercor listing
The work
We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:
- Was the information available?
- Does the rubric accurately grade what the prompt asked for?
- Would a valid answer have been marked wrong?
- Is the rubric structure correct per the guidelines?
This role includes providing feedback to reviewers regarding prompt and rubric issues.
Must-haves
- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
- Experience providing feedback to writers/reviewers
- Consistent availability
- Can read a long agent transcript and find where it went wrong.
- Precise writing, and you check the data instead of trusting the reference answer.
How long hiring takes
On the platforms we refer people to, half of hires came within about 25 days of applying. Time from applying to hire:
Talent pool members
When your profile is a strong match, we can put you forward to Mercor. Set up your profile first if you haven't.
Set up your profile →Why this role
This AI Evaluation Rubric Auditor role is fully remote and pays $60-85/hr for Accounting work.
Skills and categories
More jobs in these fields:
Related jobs
Verified reviews
Community reviews
Leave your review
Common questions
Is Mercor for freelancers or full-time contractors?
Mercor places you with one client for a set contract, such as 'Python Tutor for 3 months'. Most roles are steady contract work rather than one-off tasks.
Does Mercor's application require an on-camera interview?
Yes. You record a video interview with an AI interviewer that asks about your resume, and clients watch it before they hire, so plan to be on camera.
Can I choose my own hours?
Usually, yes. You log in when you want, pick up a task, finish it and submit it, with no meetings. Other people work from the same pool of tasks, so some days there may be no work until more is added.
What does Accounting work look like for an AI Evaluation Rubric Auditor?
You check AI answers to Accounting questions, correct their errors and write expert reasoning. As an AI Evaluation Rubric Auditor, you're hired for that judgment, so hands-on Accounting experience matters.
How much does this specific role pay?
This listing is posted at $60-85/hr, an hourly rate. Where you land in the range depends on your background and the assessment. Pay can change after we check the listing, so confirm the rate on the platform's application page.
What happens when I click Apply on this listing?
Apply opens Mercor's site, where you create an account and follow its sign-up steps.
How soon will I start working after applying to Mercor?
Usually after a few weeks. Applying puts you in Mercor's pool of candidates, and you start once a client, such as an AI lab, picks your profile.