Generalist Annotator — Health AI Conversation Quality Evaluation
Mercor • Remote, United States
Education
Bachelor's
Type
Hourly
Pay
$20-160/hr
Your rate depends on your experience and the task.
Last checked
Today
Checked
Only apply if this role fits your skills and interests.
About this role
From the Mercor listing
About the Role
Mercor is hiring generalist annotators to evaluate conversations between users and a Health & Fitness Assistant AI. You will judge how well the AI communicates: whether it answered the actual question, whether a non-expert could follow it, and whether it was honest even when honesty was the harder answer.
This is a non-clinical role. You do not need medical training, and you should not try to supply it — clinicians assess accuracy and safety on the same conversations, as a separate track. Your value is precisely that you are not one of them: you can say how a real person would experience the response, which a clinician reading the same text cannot.
What you will do
- Relevance and clarity — did the AI answer what was asked, in language a family member with no medical background could act on?
- Sycophancy detection — the core of this work. Spotting where the AI told someone what they wanted to hear instead of what they needed to hear: backing down when a user pushed back, hiding behind "you know your body best", or framing a risky decision as empowering rather than naming the risk. Automated evaluators reliably score this as helpful, which is why it needs a person.
- Tone and outcome — whether the conversation left the user informed and calmer, or anxious and no better off, and whether they walked away with anything concrete to do.
- Written justification — every rating below the top option requires a specific, quotable explanation of what went wrong and what a better response would have said. The ratings tell us something is wrong; only your writing tells us what to fix.
Conversations run from 1 to 7 turns; many are a single turn. Expect roughly 2–13 minutes each depending on length. Full written instructions and worked examples are provided before you start.
Required qualifications
- Strong written English, and the ability to explain a judgment in two to four specific sentences rather than a one-line verdict
- Careful reading — the work rewards noticing what a response quietly left out, not just what it got wrong
- Comfort applying a detailed rubric consistently across many items
- Reliable availability, with the ability to concentrate hours when a batch is time-boxed
Preferred qualifications
- Prior annotation, human-feedback, evaluation, or content-review experience
- A background where you have had to explain something technical to a non-expert audience — teaching, editing, writing, customer support, patient advocacy
- Experience with rubric-based grading or quality assurance work
Important
Using an AI tool to write your comments is prohibited and will end your work on this project. This project exists to capture human judgment that AI systems lack; AI-written feedback corrupts the dataset we use to check those systems. It is checked for.
Why this work
Most health-AI failures are not exotic. They are ordinary answers that sound responsible, read as warm and careful, and still leave someone worse off than if they had never asked. Automated evaluators are good at spotting responses that look careful, which is exactly why they miss these. Catching them takes a person who read the whole conversation and noticed where it quietly went wrong.
Requirements
- Must be eligible to work in United States
- Fluent proficiency in English (Written & Verbal)
- Reliable high-speed internet connection
- Bachelor's's degree or equivalent professional experience
- Demonstrated expertise in Generalist
How long hiring takes
On the platforms we refer people to, half of hires came within about 25 days of applying. Time from applying to hire:
Talent pool members
When your profile is a strong match, we can put you forward to Mercor. Set up your profile first if you haven't.
Set up your profile →Typical contract length
Across 10 Mercor Generalist listings we've tracked, contracts typically run about 6.8 weeks. Length varies by project.
Why this role
This Generalist Annotator — Health AI Conversation Quality Evaluation role pays $20-160/hr for Generalist knowledge you already have, so you don't need to learn a new field.
Skills and categories
More jobs in these fields:
Related jobs
Enterprise Sales Expert (AE / Sales Leader)
Mercor • Generalist
$70 /hr
Creative & Media Expert
Mercor • Generalist
$90 /hr
Network Engineering & Simulation Expert — Hephaestus
Mercor • Generalist
$90 /hr
Loan Underwriter / Loan Officer — Paid Research Survey (US)
Mercor • Generalist
$50 flat, whole project
Verified reviews
Community reviews
Leave your review
Common questions
Does Mercor's application require an on-camera interview?
Yes. You record a video interview with an AI interviewer that asks about your resume, and clients watch it before they hire, so plan to be on camera.
Does it cost money to apply to Mercor?
No, applying to and joining Mercor is free. Mercor charges the client a fee on top of your rate. If anyone asks you to pay to join, treat it as a scam.
What does task-based AI training work look like?
Short, hands-on tasks such as recording videos, sorting images, rating AI answers or checking data. Most take 5 to 60 minutes each.
What does Generalist work look like for a Generalist Annotator — Health AI Conversation Quality Evaluation?
You check AI answers to Generalist questions, correct their errors and write expert reasoning. As a Generalist Annotator — Health AI Conversation Quality Evaluation, you're hired for that judgment, so hands-on Generalist experience matters.
What specific skills does this listing call for?
Data Annotation and English are named directly in the listing. If you don't have hands-on experience with these, expect the screening to test for them.
How much does this specific role pay?
This listing is posted at $20-160/hr, an hourly rate. Where you land in the range depends on your background and the assessment. Pay can change after we check the listing, so confirm the rate on the platform's application page.
What happens when I click Apply on this listing?
Apply opens Mercor's site, where you create an account and follow its sign-up steps.
Can I apply from outside the United States?
This role is open only to people based in the United States. The limit is usually about where the platform can legally hire, so applying from elsewhere rarely leads to an offer. Check the description for tax or right-to-work rules before you apply.
How soon will I start working after applying to Mercor?
Usually after a few weeks. Applying puts you in Mercor's pool of candidates, and you start once a client, such as an AI lab, picks your profile.