Skip to content
aitrainer.work - AI Training Jobs Platform

RLHF and AI Evaluator Jobs: Get Paid to Rate Model Responses

Paid work to judge AI answers: read a prompt and two or more model responses, pick the better one, score it against a rubric, and explain why. Listings call it AI rater, evaluator, reviewer, task auditor or RLHF annotator; the session is the same. It is remote contract work you do from home, usually part-time and in sessions you schedule. Every field and most languages are represented, from generalist rating to physicians grading clinical answers. Most roles here pay $30 to $100/hr.

485 open jobs
Typical pay
$57/hr
Most roles pay
$30 to $100/hr
Based on
258 posted rates
Updated
November 16, 2026

About RLHF & AI Evaluation work

RLHF evaluation is paid judgment: you decide which answer is better and say why

A session opens with a prompt someone sent to a model, followed by two to four responses to it. You read all of them, pick the best one or rank them, score each against a short rubric (did it follow the instruction, is it correct, is it safe, how is the tone), and write a rationale in your own words. Your choices become the preference data that reinforcement learning from human feedback (RLHF) trains the next model version on, which is why the listing title can say annotator, evaluator, AI rater, reviewer or task auditor and describe the same afternoon of work.

The skill being bought is calibration, not machine learning. Nobody in this queue needs to know what a reward model is. What matters is whether your 5 out of 7 means the same thing as the next rater's 5 out of 7, whether you spot that a confident answer invented a citation, and whether your rationale lets a reviewer check your reasoning in thirty seconds. The word rater comes from search-engine quality rating, and the habits transfer: read the guidelines, apply them the same way every time, flag what the guidelines do not cover.

Three tiers show up in the listings here. Generalist rating takes any careful reader with strong written English or another required language, and most language-paired roles sit here. Expert rating asks for a credential or a career (clinicians grading medical advice, lawyers grading legal reasoning, engineers grading code) and pays accordingly. Benchmark authoring is the hardest version: writing questions a frontier model cannot yet answer, with a reference solution and a grading key, so a lab can measure progress.

One example task, start to finish

This is our own example, not a question from any platform. The prompt: a user asks for a three-day packing list for a November work trip to Oslo, carry-on only, with one client dinner.

Response A runs to forty items, includes a beach towel and sunscreen, and never mentions the carry-on limit. Response B is twelve items, grouped by day, reminds the user that liquids over 100 ml will not clear security, and adds a wool layer for the dinner. It also states that Oslo averages 15°C in November, which is wrong by about ten degrees.

The task asks you to pick the better response, score each from 1 to 7 on helpfulness, accuracy and instruction following, and write a rationale. B is the pick: it followed the carry-on constraint and the dinner, and it is usable as written. The 15°C claim costs it on accuracy, and your rationale has to say so, because an unflagged factual error is the thing a reviewer will catch. A good rationale names the specific failure and the specific win; a weak one restates the scores.

Weak rationaleStrong rationale
B is better because it is more helpful and better organized. A is too long.B is the pick: it respects carry-on only (liquids note) and covers the dinner. A ignores both constraints and lists beach items for Oslo in November. B loses an accuracy point for "15°C average", which is far too warm for the month and should not have been stated without a hedge.

Most queues run tasks like this at ten to thirty minutes each, with a quality reviewer re-checking a sample of your work. The free Academy module Your First Annotation Task walks through a full pairwise comparison with the rationale written out.

It suits careful readers, bilinguals and credentialed experts, in that order of volume

Three groups get hired for these roles. The largest is people who read closely and write clearly in the language the listing names: many roles here are language-paired, which means a native Korean, Danish or Brazilian Portuguese reader with good English is more employable than a monolingual English speaker with the same judgment. The language pages list those roles by language.

The second group is domain experts. A hospitalist grading a model's discharge summary, a tax preparer scoring a workflow, a security engineer auditing a code change: these listings name the credential in the title and pay at the top of the range. The third group is engineers and researchers writing benchmarks and evaluation harnesses, where the work is closer to test design than to rating.

It does not suit people who want to skim. Rating queues track agreement with gold-standard answers and with other raters, and accounts that drift below the threshold stop receiving work without much explanation. If you tend to finish tasks fast and move on, transcription or agentic task work rewards speed more than this does.

Screening tests whether you grade the way the guidelines do

The usual sequence is a profile, a qualification test, and a paid trial batch. The qualification test gives you sample prompts and responses with the platform's guidelines and compares your scores with a gold standard, so the question it answers is how calibrated you are, not how much you know. Some platforms add an AI-led video interview before the test, and expert tiers add a credential check or a domain task. Where a listing on this page states its screening step, it is almost always an interview; where it says nothing, assume a test.

Preparation that works: read the guidelines twice before the first sample, score slowly, and write every rationale as if a stranger will audit it. The Academy modules Passing the Screening Process and Reading the Rubric Like a Grader cover the stages and the anatomy of a rubric, and the guide on what rubrics are explains the vocabulary. Interview questions on this site are never quoted from a platform, because candidates sign NDAs.

Pay comes from the listings above, and it splits by tier more than by platform

These figures come from the rates the open listings state, and they update as listings change. How we work out pay.

Most roles here pay $30 to $100/hr.

Within that spread, generalist rating sits at the bottom, language-paired rating in the middle, and credentialed expert rating and benchmark authoring at the top. A few listings pay per task rather than per hour; those are excluded from the hourly figures. For the ranking across every category, see the highest-paying fields and platforms.

How to start this week

  1. Apply to two or three platforms at once. Rating queues fill and empty by project, and one platform rarely gives steady hours. Sort the list below by newest and pick listings whose language and expertise match yours exactly.
  2. Do the free walkthrough before the test. Foundations of RLHF (50 minutes) explains where your ratings go, and Your First Annotation Task has you make a real pairwise decision and write the rationale. Both are readable without an account.
  3. Write your profile in the listing's words. If the title says "AI Response Evaluator (Native Speaker)", your profile should say native speaker, evaluation, and the language, with any credential spelled out.
  4. Take the qualification test rested and unhurried. Agreement with the gold standard decides it, and most people fail by rushing the last third.
  5. Treat the first paid batch as the second test. Quality review on the first fifty tasks decides whether you see more work. Advanced Instruction Following covers the failure reviewers look for most: a response that looks compliant and is not.

Which companies are hiring raters right now, with live counts, is kept up to date in this guide. If you would rather have the board filtered to your background and minimum rate, the matcher does it in three questions and no account.

Adjacent kinds of work: red teaming and safety review grades the model on what it should have refused, agentic tasks grade what a model did rather than what it said, and transcription produces the speech data that voice models train on.

Type:
Popular:

485 jobs

Page 6 of 17

Alignerr AI training platform

Dutch Voice AI Evaluator – Healthcare Conversations

$12-40

/hr

Alignerr•18d ago
Alignerr AI training platform

Korean Voice AI Evaluator (AI Training, Remote)

$12-40

/hr

Alignerr•18d ago
LanguagesKorean
Alignerr AI training platform

Korean Voice AI Evaluator – Native Korean Speakers

$12-40

/hr

Alignerr•18d ago
Alignerr AI training platform

Korean Voice AI Evaluator (Remote, Flexible)

$12-40

/hr

Alignerr•18d ago
Newsletter subscription
AI Training Jobs

New AI training jobs every Tuesday

Trending jobs, new guides and pay data from listings, in one weekly email.

500+ subscribers

Unsubscribe anytime.

Alignerr AI training platform

Korean Voice AI Evaluator – Native Speakers Wanted

$12-40

/hr

Alignerr•18d ago
Alignerr AI training platform

Korean Voice AI Evaluator (Healthcare Conversations)

$12-40

/hr

Alignerr•18d ago
Alignerr AI training platform

German Voice AI Evaluator (Healthcare Conversations)

$12-40

/hr

Alignerr•18d ago
Alignerr AI training platform

German Voice AI Evaluator – Zurich (Healthcare Conversations)

$12-40

/hr

Alignerr•18d ago
Alignerr AI training platform

German Voice AI Evaluator – Vienna (Healthcare Conversations)

$12-40

/hr

Alignerr•18d ago
Terac paid research studies platform

Software Engineers & Data Scientists: AI Slide Deck Evaluation

$110

/hr

Terac•21d ago
Terac paid research studies platform

Business Professionals: Presentation Slide Deck Evaluation for AI Training

$110

/hr

Terac•21d ago
AI TrainingReview & QA
Terac paid research studies platform

Radiology Residents: Medical Image Evaluation for AI Benchmarks

$105

per task

Terac•21d ago
Terac paid research studies platform

Radiology Residents: Medical Image Evaluation for AI Benchmarks

$90

per task

Terac•21d ago
Meridial expert contractor network, by Invisible Technologies

Multilingual AI Evaluation — Independent Contractor Opportunity

$9

/hr

Meridial•24d ago
LanguagesMultilingual
AnyoneAI expert human data platform

Machine Learning Engineer – ML Evaluation & Experiment Design

$65

/hr equivalent

AnyoneAI•34d ago
AnyoneAI expert human data platform

Senior Software Engineer – Open Source & SWE-Bench Evaluation

$65

/hr equivalent

AnyoneAI•34d ago
micro1 AI training platform

AI Agent Evaluator

$25-45

/hr

micro1•Bachelor's•37d ago
Mercor AI hiring platform

CAD / PCB / 3D Design Engineer — AI Evaluation

$60-100

/hr

Mercor•PhD•33d ago
Mercor AI hiring platform

Primary Care Physician (MD) - Clinical Documentation & AI Evaluation

$170

/hr

Mercor•36d ago
Innodata AI data engineering company

AI Voice Evaluation Specialist - 3 month contract

$20-25

/hr

Innodata•33d ago
ExpertAI Training
Mercor AI hiring platform

Football (Soccer) Fan – Live Match AI Evaluator (UK & Spain)

$105-140

/hr

Mercor•40d ago
LanguagesEnglishSpanish
Meridial expert contractor network, by Invisible Technologies

Technical Task Auditor

$60

/hr

Meridial•40d ago
AccountingAI Training
Mercor AI hiring platform

Cricket Fan – Live T20I Match AI Evaluator (India)

$20

/hr

Mercor•40d ago
EnglishExpert
RWS TrainAI data services community

AI Linguistic Evaluator - Telugu

$18

/hr

RWS (TrainAI)•41d ago
LanguagesEnglishTelugu
RWS TrainAI data services community

AI Linguistic Evaluator - Marathi

$16

/hr

RWS (TrainAI)•41d ago
LanguagesEnglishMarathi
Meridial expert contractor network, by Invisible Technologies

Image Data Contributor and Evaluator

$9

/hr

Meridial•40d ago
Data AnnotationAI Training
Remo Experts AI training platform

Software Engineer – LLM Evaluation & Repository Validation

$50

/hr

Remo Experts•Master's•42d ago
← Previous Page 6 of 17 Next →

Get new RLHF & AI Evaluation jobs as they open

One browser alert a day with new roles, RLHF & AI Evaluation first. You don't need an email.

Or follow new roles in Telegram: @aitrainerwork

Browse by category or skill

Similar to: Offensive Cybersecurity Engineers: Build RL Environments for AI Evaluation

Similar to: Subject Matter Experts: Machine Learning Model Evaluation

Similar to: AI Conversation Evaluator

Common questions

What is an AI rater job?

An AI rater job pays you to read a model's responses and score them against written guidelines, usually picking the better of two answers and explaining why. Platforms also call it evaluator, reviewer or RLHF annotator work, and most roles are remote hourly contracts with no fixed schedule.

Do I need to know machine learning to do RLHF work?

No machine learning knowledge is needed. Your scores are the training signal, and the learning algorithm is the lab's problem. Platforms screen for whether you grade the way the guidelines do, for clear writing, and, for expert tiers, for a credential in the field being graded.

How much do AI raters and evaluators earn per hour?

Generalist rating pays the least and credentialed expert rating the most. A few roles pay per task instead of per hour. Most AI rater and evaluator roles pay $30 to $100/hr.

Is RLHF evaluation the same as data annotation?

RLHF evaluation is one kind of data annotation. Classic annotation labels inputs, such as boxes on images or tags on text, while RLHF evaluation judges outputs, such as which of two answers is better. Listings use the words interchangeably, so search both when you look for work.

What is the qualification test for AI rater jobs like?

It is an open-book exam on the platform's guidelines. You get sample prompts, responses and the guidelines, then score the samples. Your scores are compared with a gold standard and with other raters, and agreement above a threshold passes. Platforms do not publish the threshold.

Can I do AI evaluation work part-time or alongside a job?

Yes, most AI evaluation roles are hourly contracts with hours you choose. Many state a minimum of 10 to 20 hours a week during a project. Hours are not steady, because projects end and queues empty, so most raters hold accounts on two or three platforms.

Why do AI rater listings disappear and come back?

Labs commission rating in projects of a few weeks, and platforms open and close recruiting as each project starts and fills. The same title can reappear a month later, so a job alert for the title is the easiest way to catch it.

How many RLHF & AI Evaluation AI training jobs are open right now?

There are 485 open listings, from 17 platforms: Mindrift (140), Mercor (138), RWS (TrainAI) (53), Turing (44) and Terac (22). The newest one was added 2 days ago. The count changes as platforms post and close roles.

Other kinds of AI training work

See all categories