AI Evaluation & Rating (RLHF)

If you want to get paid as an AI evaluator or AI rater (the RLHF and model evaluation work behind every chatbot), here is the honest headline: the open AI evaluation roles on this site typically pay Unavailable to Unavailable an hour, with a median of Unavailable. Generalist rating sits at the low end of that band, and credentialed specialists (coders, PhDs, MDs, lawyers) reach the top. For comparison, ZipRecruiter put the US average for reinforcement learning from human feedback (RLHF) work at $40.70/hr in its 16 Oct 2026 update. The work is real, it is remote, and the demand is large ( 0 AI evaluation roles are open on this site today). But it is freelance, feast-or-famine, and full of low-effort scams. This guide tells you exactly what the work is, what each platform actually pays, how to get in, and where the traps are.

RLHF and model evaluation is the judgment corner of the broader "get paid to train AI" world. It is not image tagging or transcription. It is judgment work: you read what an AI model produced, decide whether it is good, and tell the lab why. Because that judgment is what actually improves frontier models, the labs pay for it as judgment, and they pay a large premium when you bring real expertise to the table: the median AI evaluation role here advertises Unavailable, the median law, medicine and finance role Unavailable.

What RLHF and Model Evaluation Work Actually Is

Reinforcement Learning from Human Feedback is the technique that turned raw language models into useful assistants. A base model predicts text; RLHF teaches it which responses humans actually prefer. As Palo Alto Networks explains, humans rate or rank model outputs, those ratings train a separate "reward model," and the reward model then steers the main model toward human-preferred behavior. You are the human in "human feedback." Your ratings are the raw material.

In practice, the day-to-day breaks into a handful of recurring task types.

Preference ranking (the core task)

You are shown a prompt and two or more model responses (often labeled A and B). You pick the better one and usually explain why: which is more accurate, more helpful, better formatted, safer, or more complete. This is the single most common RLHF task. Rise Data Labs, which sells this work to AI companies, describes the step as: ' Generate and rank high-density comparison pairs at scale, validated through structured quality control' (its RLHF page, read 5 Nov 2026). Most tasks take minutes, not hours.

Rating and scoring on rubrics

Instead of comparing two responses, you score a single response against defined criteria: helpfulness (1 to 7), factual accuracy, instruction-following, tone, safety. The lab supplies a rubric and you apply it consistently across hundreds of examples. Quality is tracked through "gold standard accuracy, inter-annotator agreement, and per-worker trust scores," and low-quality labels get auto-reassigned ( TaskMonk).

Writing demonstrations and corrections

Sometimes the model's answer is wrong and you write the correct one (a "demonstration"), or you edit the model's output into a gold-standard version. This is where strong writers and domain experts earn more: you are not just judging, you are producing the reference answer the model should learn from.

Prompt engineering and adversarial prompting

You write prompts designed to probe the model, elicit specific behaviors, or expose weaknesses, then evaluate how it responds across multi-turn conversations.

Red teaming and safety evaluation

The highest-skill category. Red teaming is "systematic stress-testing for safety, bias, and jailbreak vulnerabilities before public release" ( APxML). You try to make the model behave badly (produce disallowed content, contradict itself, leak information) and document exactly how you did it. These structured findings feed straight into the safety training pipeline. Red teaming, multi-turn conversations, and agentic or RL-environment data are the harder task types, and they command premium rates: the median red-teaming role here advertises Unavailable, the median agentic task role Unavailable.

Two delivery modes exist: live chat evaluation (you interact with the model in real time and rate as you go) and async transcript rating (you review batches of saved conversations). Most generalist work is async; live evaluation is more common on specialized projects.

How Much You Can Actually Make

Pay is driven by four levers, in order of impact: (1) your credentials and domain, (2) task complexity, (3) which platform you are on, and (4) your geography. The single biggest lever is expertise. A generalist and a board-certified physician can be on the same platform earning very different rates: on Mercor's researched rate card (checked 8 Nov 2026), its generalist role reads $15–$150/hr and its physician role $45–$450/hr.

Here is the overall picture. On this site's open roles, AI evaluation work typically pays Unavailable to Unavailable an hour with a median of Unavailable, against Unavailable to Unavailable and a median of Unavailable across all AI training work; the AI evaluation market report follows those figures day by day. Mercor's rate data (updated 23 Jun 2026) puts the national average AI-trainer rate at about $31/hr in early 2026, with a full range of $12 to $200+, in three tiers:

Pay by seniority and tier

The percentile breakdown from the same Mercor data (updated 23 Jun 2026) is a useful reality check: bottom quartile $18.50 to $21, median $25 to $33, upper quartile $53 to $58, and the top 10% at $100+. So the top of the band is real, but it is a top-decile outcome that generally requires credentials.

Pay by platform

This is the table most people are looking for. Each rate below is the platform's researched rate card on this site: its rating or evaluation role where the card has one, and its whole card beside it. Each opens the platform's pay page with the sources, and the date is the day that card was last checked against them.

A few clarifications. Mercor, Surge AI, and Handshake AI sit at the top because they are built around credentialed experts. DataAnnotation and Outlier are the most common entry points for generalists. Appen/CrowdGen and TELUS are the legacy "search rater" world (Appen reported revenue of US$273.0M for 2023 in its 26 Feb 2025 results and US$230.8M for 2025 in its 25 Feb 2026 results, which guided to US$270M to US$300M for 2026, and reviewers say task volume has thinned ( review)). Prolific is a slightly different animal: it is a research-participant panel where the "AI" work is often academic behavioral studies rather than pure RLHF, with a required floor of £6/$8 per hour and a recommended £9/$12 in Prolific's payment principles (updated 17 Jul 2026).

Pay by geography

Geography matters enormously, both because some platforms take only a few countries, and because rates differ by region. These are the ranges The AI Rankings gave in its guide as archived in June 2026 (its current version, updated 5 Nov 2026, no longer carries a regional table):

DataAnnotation publishes no country list (its FAQ asks only for reliable internet and fluent English), but it is reported to take workers from six countries only: the US, Canada, the UK, Ireland, Australia and New Zealand ( The AI Rankings, archived June 2026; its page here). That is worth knowing, because it is one of the best-paying generalist entry points and it turns away most of the world.

What a realistic month looks like

Advertised rates are ceilings. The binding constraint is almost never the rate, it is the number of paid hours available. Multiple sources describe the same pattern: workers often start strong and see it decline over months as platforms surge-hire and then run out of tasks. For a part-time contributor, Outlier reviews describe solid months during active project periods, and nothing when a project pauses. TELUS-style rater work is more stable but part-time and capped lower ( review). Plan for lumpiness, and diversify across two or three platforms so a project drought on one does not zero out your income.

Skills and Qualifications You Actually Need

The barrier to entry is lower than most people assume for generalist work and higher than most people assume for the money tiers.

For generalist RLHF (the low end of the band): No degree is strictly required on DataAnnotation, Outlier, or Alignerr for generalist tracks. What you genuinely need is:

• Native-level (or very strong) reading and writing in the target language, usually English.
• The patience to read long, multi-page instructions and follow them exactly. This is the number-one differentiator. The whole game is consistency against a rubric.
• Basic critical-thinking and fact-checking instincts: spotting when a confident answer is subtly wrong.
• Reliable internet and a computer (most tasks are desktop-only).

For the expert tiers (the top of the band): A credential and provable domain depth. Platforms explicitly target PhD students, post-docs, practicing physicians, attorneys, and senior software engineers ( Mercor). Coding is the most accessible premium skill: DataAnnotation's coding work is carded at $40–$150+/hr, and you do not need a CS degree, just the ability to evaluate and write correct code in a real language.

The tools are simple: a browser, the platform's web-based labeling interface, and often a rubric document open in another tab. You do not need to know how to code a neural network. You need to be a sharp, consistent judge of quality.

How to Get Started and Pass the Assessments

The legitimate path has a fixed shape: find a platform, apply, complete a sample or qualification task, and if your work is good, you start receiving paid tasks. Money flows one direction only, from the company to you, and there is never a fee to join ( HireFeed).

Step 1: Pick platforms that match your profile

• Strong writer, no special credential, in one of the six countries: Start with DataAnnotation. It is writer-friendly and among the best-paying generalist entry points.
• Generalist, broader country list: Outlier and Alignerr accept a wide range and have high volume.
• You have a PhD, MD, JD, or senior engineering background: Go straight to Mercor, Surge AI, and (if you are based in the US) Handshake AI, where credentials unlock the expert end of their rate cards.
• You want stable, lower-stakes work: TELUS International AI and Appen/CrowdGen search/rater projects.

You can and should apply to several. A profile aggregator like OpenTrain lets you build one expert profile and reuse it, but applying directly on each platform is fine.

Step 2: Pass the qualification test

Most platforms gate paid work behind an unpaid assessment. On DataAnnotation, the Starter Assessment takes about an hour and covers writing, grammar, reading comprehension, and simple logical reasoning, and critically, you can only take it once, with no retakes ( DataAnnotation's FAQ, read 29 Oct 2026). The proven approach:

1. Read the instructions completely before you touch anything. The test is designed to confirm you can follow complex, multi-page instructions accurately. Skimming is the most common failure.
2. Prioritize accuracy over speed. Successful candidates routinely take longer to be thorough. The platforms track quality, not clock speed, on the assessment ( OperationsArmy).
3. Be internally consistent. Apply the same criteria to every example, especially ambiguous ones. Inconsistency is what fails people.
4. Sweat the small details. Grammar, spelling, and formatting in your written justifications are part of the score, not a side note.
5. Explain your reasoning. When you rank A over B, the "why" is often what is actually being graded.

Step 3: Pass the follow-on qualifications

Passing the starter test does not mean paid work yet. On DataAnnotation you then take multiple short, unpaid qualification tests that match you to specific projects ( OperationsArmy). Treat each one like the first: read fully, be consistent, explain your reasoning.

Step 4: Protect your quality score

Once you are working, your per-worker trust/quality score determines whether you keep getting tasks and get promoted to higher-paying projects. Slow down on the first tasks of any new project, re-read the guidelines when a project updates them, and never guess when you can verify.

Payment Logistics and Taxes

You are an independent contractor, not an employee. That has consequences you should plan for from day one.

How you get paid: Most platforms pay weekly or bi-weekly via PayPal, and some use services like Hyperwallet. Handshake AI, for example, pays weekly through Stripe, and DataAnnotation pays to PayPal when you request a payout (each platform's page here says how it pays).

Taxes (US): All of it is taxable, whether or not you receive a tax form. You will typically get a 1099-NEC (non-employee compensation) or 1099-K (via PayPal). The One Big Beautiful Bill Act restored the 1099-K threshold to more than $20,000 AND more than 200 transactions ( IRS, 23 Oct 2025), so many contractors will not receive a 1099-K, but the income is still reportable. The IRS is explicit: all gig income is taxable and self-reported on Schedule C.

Key tax facts for US contractors:

• You owe self-employment tax of 15.3% on net earnings (up to the Social Security wage base), on top of ordinary income tax ( IRS, read 29 Oct 2026).
• No tax is withheld. You are expected to make quarterly estimated payments (roughly April 15, June 15, September 15, January 15, per IRS Publication 505, read 29 Oct 2026).
• You can deduct legitimate business expenses on Schedule C: a portion of internet, a home-office deduction if eligible, and equipment.
• Set aside a share of every payment for taxes so quarterly deadlines do not hurt.

Pros and Cons: An Honest Ledger

The upside

• Genuinely flexible and remote. You choose your hours. Glassdoor contributors consistently cite work-from-home flexibility as the top pro ( Glassdoor).
• Low barrier for generalists, high ceiling for experts. You can start with no credential, and if you have a PhD/MD/JD you can earn professional-consultant rates from your couch.
• The work is interesting. You are looking at frontier models before the public does, and the variety of projects keeps it from getting stale.
• You get paid to build AI literacy that is increasingly valuable in any career.

The downside

• Feast or famine. The recurring, near-universal complaint. Task availability is unpredictable; income swings from a strong quarter to nearly nothing ( Mercor review context). Platforms over-onboard: an Alignerr contributor describes new freelancers joining its Slack every day, many with no work ( Glassdoor).
• Unpaid onboarding. Long tutorials, mandatory webinars, and assessment tasks are often uncompensated or paid at a fraction of the task rate. One Outlier reviewer on Glassdoor worked out that, counting unpaid training, they were paid about $120 for 15 hours, below their state's minimum wage ( summary).
• No benefits, no stability. No health insurance, no PTO, no guaranteed hours. This is supplemental or bridge income for most, not a salary. If you are bridging after a job whose plan paid for a GLP-1 prescription, GLPcompare's page on lost coverage, published by the owners of this site, lists what each medicine costs without insurance.
• Legal and payment friction on some platforms. Outlier (operated by Scale AI) has faced multiple lawsuits over wage and misclassification claims ( review), and Handshake's Project HH contributors reported being paid for only part of their tracked hours ( review).
• Occasional exposure to disturbing content, especially in red-teaming and safety work.

Red Flags and Scams to Avoid

The RLHF gold rush attracts fraud. The rule is simple: money only ever flows from the company to you. Watch for these ( FTC, HireFeed):

• Any request for payment. "Training materials," "certification," "platform access," or "starter kit" fees are always a scam. Legitimate platforms are free to join.
• The hiring process runs on Telegram or WhatsApp. Almost certainly fraudulent. Real platforms use their own web portals and company email.
• You are asked to download an unknown "annotation tool" or register on a strange, off-brand website to take a "test." Real assessments happen inside the platform's own environment.
• Requests for banking passwords, or "verification" via a check they send you. Never share banking credentials; check-cashing schemes are classic fraud.
• An "assessment" that is suspiciously large and looks exactly like real production work. Some scams extract free labeling under the guise of an unpaid test. A legitimate qualification is a bounded sample, not an open-ended project.
• Impersonation. Scammers use the names of real companies (a review of Appen notes that it charges no fees and that impostors use its name). Always apply through the official domain you verified independently, never a link from an unsolicited message.

The 2026 Outlook

The demand side is strong and structurally shifting toward exactly the skilled evaluation work this guide covers. The AI data labeling market is estimated at about $2.32 billion in 2026, projected to reach $6.53 billion by 2031 at a ~23% CAGR ( Mordor Intelligence, updated 11 Sep 2026); a broader estimate reaches about $18.2 billion by 2035 ( Precedence Research, updated 15 Sep 2026). Meta's $14.3 billion investment for a 49% stake in Scale AI, reported by CNBC on 12 Jun 2025, signals how strategic this supply chain has become.

The important trend for a worker: the market is moving from volume to depth. Simple annotation is being automated and commoditized (and its rates are falling), while RLHF, expert feedback, red teaming, and domain evaluation command premium rates (the AI evaluation market report follows what this work pays, day by day). New credential-first platforms keep launching ( AfterQuery, a Y Combinator lab, announced a $30M Series A on 9 Apr 2026, and its researched rate card reads $15–$175/hr; middle half $50–$104/hr). The takeaway: generalist rates will stay under pressure, but anyone who can bring verifiable expertise (coding, science, medicine, law, advanced math, or genuinely rigorous reasoning) is entering a growing, well-paid market.

Bottom Line

RLHF and model evaluation is legitimate, remote, and its open roles here typically pay Unavailable to Unavailable an hour, depending mostly on what you bring to it. If you are a careful reader and writer, start with DataAnnotation, Outlier, or Alignerr, pass the assessment by reading every instruction and prioritizing accuracy, and diversify across platforms to smooth out the feast-or-famine cycle. If you have a real credential, skip straight to Mercor, Surge AI, or Handshake AI, where your expertise is worth the most (the law, medicine and finance roles here typically pay Unavailable to Unavailable an hour). Treat it as flexible supplemental income, set aside money for self-employment taxes, and never pay a cent to join. Do that, and this is one of the better ways to earn from a laptop in 2026.