Coding & Software Evaluation

If you can read code, spot a bug, and explain why an answer is wrong, you can get paid to evaluate AI-generated software: the coding roles open on this site typically pay Unavailable to Unavailable an hour today, with a median of Unavailable. "Coding and software evaluation" work in AI training means reviewing the code that large language models write, judging whether it is correct, secure, efficient, and production-ready, ranking competing answers, and writing the corrected reference solution when every model gets it wrong. Frontier AI labs (OpenAI, Anthropic, Google DeepMind, Meta) pay real engineers to do this because a model cannot reliably grade its own code at the level of nuance required, and the budgets are large: Anthropic alone discussed spending more than $1 billion on reinforcement-learning environments over the following year, TechCrunch reported on 21 Sep 2025, citing The Information. This guide covers exactly what the work is day to day, what the legitimate platforms actually pay (by platform, seniority, and country), how to pass the assessments, the tax and payment logistics, and the red flags that separate real work from the scams flooding the space.

The short version: this is one of the highest-paying remote gig categories that exists, it is genuinely legitimate at the platforms named below, and it rewards actual engineering skill rather than credentials or resume polish. It is also inconsistent, project-based, and not a replacement for a salary. Both of those things are true at once, and this guide will be honest about both.

What "Coding & Software Evaluation" Work Actually Is

The core loop is the same across almost every platform. You are handed a coding prompt (sometimes a self-contained problem, sometimes a realistic scenario like "debug this production issue" or "implement this feature across these files") together with one or more candidate solutions written by an AI model. Your job is to judge those solutions. Day to day, you read the prompt and the candidate solutions, run them mentally or in a sandbox, find bugs, evaluate code quality, and write a corrected reference solution if all candidates fail.

That single loop shows up in several named task types, and most contracts mix them:

Pairwise comparison and ranking

You see two (or more) model answers side by side and decide which is better, then explain why in structured feedback. This is the backbone of RLHF (reinforcement learning from human feedback). You are not just picking a winner: you are producing the reasoning that teaches the model what "better" means. As one practitioner guide puts it, the work means reviewing AI responses against instructions, sometimes comparing two answers, and sometimes rewriting an answer so it becomes the example the model should have produced.

Code review and bug hunting

You review a single AI-written solution for correctness and quality, catching real defects. On DataAnnotation and its peers that means catching bugs like race conditions and inefficient algorithms, across systems programming, algorithms, security, backend design, and frontend. The subtle bugs matter most: an answer that looks right and passes a naive test but has an off-by-one error, a race condition, an O(n squared) approach where O(n) exists, or a security hole.

Reference solution authoring

When every candidate is wrong, or when a project needs a gold-standard answer, you write a clean, correct, production-quality implementation yourself. You are authoring production-quality implementations that serve as training data. This is the highest-value work because your code becomes the target the model is trained toward.

Prompt writing and adversarial testing

You craft precise engineering prompts designed to probe the limits of a model's reasoning. Mercor frames the "test" side of the job as assessing how models handle ambiguity, incomplete requirements, architectural tradeoffs, conflicting constraints, and multi-step engineering workflows. You are effectively trying to break the model in instructive ways.

Rubric design and RL environments

At the senior end, you stop grading individual tasks and start designing the evaluation itself: building rubrics, test harnesses, containerized environments, and reinforcement-learning environments where an agent's code is automatically scored. Surge AI's senior coding role involves owning "end-to-end coding data projects" and setting up technical environments like containers, repos and test harnesses. This is where the money is highest.

The common thread: human judgment is the product. You are being paid for the specific engineering discernment a model cannot yet reliably reproduce.

The Platforms That Hire for Coding Evaluation (and What They Really Pay)

There is a large gap between the headline rate a platform advertises and what a typical contributor nets. The rates below are each platform's researched rate card on this site, read from its own pages and listings; each opens the platform's pay page, and the Checked column is the day the card was last checked against its sources. Treat the top of each range as the specialist ceiling and the bottom as generalist or onboarding pay.

Pay by platform

A few honest caveats on these numbers. First, advertised rates are not effective rates. Time spent reading instructions, waiting for tasks and doing calibration counts against your real hourly rate, and contributors report queues that go dry for days or weeks between projects. Second, the highest tiers are real but scarce, gated behind specialist domains or senior credentials: coding roles' typical pay on this site tops out at Unavailable an hour, and only a specialist tail advertises more. Third, several platforms ( Handshake, DataAnnotation) can and do ghost long-term contractors without warning when a project ends, so treat any single platform as impermanent.

Pay by seniority

The single biggest lever on your rate is not which platform you pick, it is how senior and specialized your engineering skill reads in the assessment. Mercor's advertised tiers show this cleanly.

Source: Mercor's software-engineer page, which listed these as recent opportunities on 29 Oct 2026. They are advertised tiers; what its listings pay is lower, $50–$120/hr for software and ML engineers on its rate card (checked 8 Nov 2026), and coding roles across this whole board typically pay Unavailable to Unavailable an hour.

Pay by geography

This work is global, but the pay is not evenly distributed, and this is one of the least-discussed realities of the field. Country of residence affects both your rate and which platforms will even accept you. The table counts the coding roles on this site that are open to applicants in each country today; each count opens its list.

The practical takeaway from the geographic data is blunt: the best-paying platforms restrict by country, and on several that accept workers globally the same work pays less in some regions (Meridial's identical work is reported at about $6 an hour in some regions against $20 in others, RemoWork, updated 17 Oct 2026). The two escape hatches from geographic pay tiers are the same two that raise anyone's rate: genuine specialization and speed. A distributed-systems specialist in any country out-earns a generalist anywhere.

Skills and Qualifications You Actually Need

The good news for engineers: this work is merit-based to an unusual degree. DataAnnotation is explicit that no degree is required ("We care about what you can do, not where you learned it"), and it assesses you through a skills exercise rather than credentials. You do not strictly need a computer science degree. What you need is the ability to demonstrate real engineering judgment on a timed test.

The hard requirements

• Working professional fluency in at least one mainstream language. Python is the most in-demand, followed by TypeScript/JavaScript, then Rust, C++, Go, and Java. If you know two or three of these well, you qualify for far more projects.
• The ability to find subtle bugs. Not just "does it compile," but "does this pass the naive test and still fail on an edge case, a concurrency scenario, or a security vector."
• Complexity reasoning. Recognizing when an O(n squared) solution should be O(n log n), or when an approach will not scale.
• Clear written explanation. You must articulate why one answer beats another. Feedback that just says "wrong" is useless; the reasoning is the training signal.

What raises your tier

• 2 to 6+ years of professional software engineering experience ( Surge's coding-evaluation posting asks for this range explicitly, read 10 Nov 2026; Mercor wants professionals who can independently reason through production systems).
• Depth in a niche: distributed systems, infrastructure, security, systems/low-level programming, ML systems, or a specific mainstream framework. Niche depth is what moves you from review projects into the specialized tiers.
• Open-source contribution history or a demonstrable public track record ( DataAnnotation asks for professional or serious open-source experience shipping production code).
• Domain crossover: coding plus law, medicine, or finance credentials pushes you toward the professional tier, where law, medicine and finance roles on this site typically pay Unavailable to Unavailable an hour.

Students are welcome too. The assessments test what you can do, so a strong CS undergrad who can crush a bug-hunting exercise can start earning coding rates.

Tools You Will Use

The tooling is deliberately lightweight. Most coding evaluation happens inside a web-based platform interface, not your local IDE. Expect:

• A browser-based task interface presenting the prompt, the model outputs, and the rating rubric or ranking controls.
• A code sandbox or execution environment for actually running candidate code. On senior projects you may set up your own containers and test harnesses.
• Markdown/rich-text feedback fields for your structured reasoning.
• Your own local environment for authoring reference solutions when the task allows it, then pasting in the verified result.
• ID verification tooling. Most legitimate platforms verify identity, and some (like Outlier) require ID verification and sometimes a video selfie to prevent fraud and duplicate accounts.

You do not need special software, a paid license, or a "starter kit." If a platform asks you to buy tools, that is a scam signal (covered below).

How to Get Started and Pass the Assessment

The application funnel is nearly identical across platforms: apply, complete a qualification exercise, get matched to projects, start earning. Money only ever flows from the platform to you. Here is the realistic path.

Step 1: Pick two or three legitimate platforms and apply

Do not apply to just one. Work arrives in waves and any single platform can go quiet or drop you. A sensible starter portfolio for a strong engineer: DataAnnotation (no degree required, merit-based assessment, high coding pay), Mercor (clear expert tiers, weekly pay), and one of Braintrust / Handshake / micro1 depending on your region and credentials. Apply directly on each platform's own domain, never through a third-party link.

Step 2: Take the assessment seriously

This is the gate, and it is a real test of engineering skill, not a formality.

• DataAnnotation: an engineering assessment testing whether you can identify correctness bugs, reason about complexity, and write clean code. Its FAQ says most starter assessments take about an hour and specialized ones, coding included, one to two hours (read 10 Nov 2026). No degree required, purely merit.
• Outlier: a timed test matching your selected domain, with LeetCode-style problems if you pick coding (RemoStack, 28 Jun 2026). Treat it as a real exam: failing it ends the application regardless of coding experience.
• Surge / senior roles: a short exercise that you email back with your resume, designed to show you can scope a project and communicate technically.

Practical tips that move the needle:

• Read the rubric before you start and grade against it, not against your own taste. The number one reason strong engineers fail is answering the question they wish was asked instead of the one on the screen.
• Justify every judgment in writing. Even when only a rating is requested, clear reasoning is what the platform is actually screening for.
• Do not rush the bug hunt. The tests are built so a fast skim misses the planted subtle defect. Run the code (mentally or in a sandbox), test the edges.
• Write clean reference code. If asked to author a solution, treat it as production code: correct, readable, reasonably efficient, no debug cruft.

Step 3: Get matched and calibrate

Once you pass, you are matched to projects by preferred stack and assessment result. Early tasks are often "calibration" work where your ratings are compared against a gold standard. Your quality score here determines your ongoing task flow and sometimes your rate, so treat calibration as more of the assessment.

Step 4: Manage the reality of project waves

Expect income to be lumpy. AfterQuery-style work typically arrives as 2 to 3 week assignments at 10 to 20 hrs/week, RemoWork reports (updated 17 Oct 2026), and queues can empty without warning. The contributors who earn consistently are the ones active on several platforms so that when one queue dries up another is filling.

Realistic Earnings: What You Can Actually Make

Here is the honest arithmetic. The advertised rate, the effective rate, and the annual total are three different numbers.

At the typical rate: coding roles on this site advertise a median of Unavailable an hour today. Multiply the rate by the hours the queue actually gives you, not the hours you hope for, then take off the unpaid time (calibration, instructions, waiting) and the dry spells between projects. What is left is the honest side-income figure, and it is usually well below the headline arithmetic.

At the expert level: the typical band tops out at Unavailable an hour, and Mercor advertises specialized engineering tiers to $200 an hour ( Mercor, read 10 Nov 2026). Steady work at that level, full-time, is a senior salary, and Mercor's own guide says the very top full-time AI training roles for senior engineers and researchers can pay upwards of $200,000 annually (updated 23 Jun 2026).

Proof it adds up: one DataAnnotation contributor documented earning $37,000 over two years on the platform (The Work at Home Woman, 5 Jul 2026), and DataAnnotation's FAQ says it has paid hundreds of millions of dollars to contractors since 2020 (read 10 Nov 2026).

The realistic mental model: coding evaluation is an excellent high-hourly side income and a plausible full-time income for a senior specialist who treats it like a business (multiple platforms, high quality scores, specialist positioning). It is not a stable salary with benefits, and anyone promising that is selling something.

Payment and Tax Logistics

You are an independent contractor (1099), not an employee. This has concrete consequences you must plan for.

How you get paid

Most platforms pay weekly via PayPal or direct deposit. DataAnnotation, for example, pays weekly and processes coding work task-based. DataAnnotation's FAQ says deposits arrive within a few days of a payout request, through PayPal (read 10 Nov 2026). Mercor and several others also pay weekly. This fast, frequent payout is one of the genuine perks of the category.

The tax reality (US)

• You owe self-employment tax. On top of income tax, contractor income is subject to self-employment tax of 15.3% (12.4% Social Security + 2.9% Medicare) (IRS, read 10 Nov 2026), because no employer is paying the other half for you.
• 1099-NEC thresholds. A platform must send you a 1099-NEC if it pays you $2,000 or more for payments made in 2026, up from $600 before (IRS, read 10 Nov 2026). But you must report all income even below that. Payment apps send a 1099-K only above $20,000 and 200 transactions a year, the threshold the One Big Beautiful Bill Act restored (IRS, 23 Oct 2025), so do not rely on receiving a form. Keep your own records of every payment from every platform.
• Quarterly estimated taxes. If you expect to owe $1,000 or more when you file, the IRS expects quarterly estimated payments (read 10 Nov 2026). Missing them can trigger penalties.
• Set aside a fixed share of every payout, sized to your own bracket, for federal, state, and self-employment taxes. This is the single most common mistake new contractors make: spending the gross and getting a nasty bill in April.
• Deduct legitimate business expenses. Your computer, home-office portion, internet, and relevant software can offset income. Track them.

Non-US contributors face their own local tax obligations and often currency-conversion haircuts on payout, so check your country's self-employment rules.

Pros and Cons: An Honest Ledger

Pros

• Exceptional hourly pay for remote work. Coding roles on this site advertise a median of Unavailable an hour, against Unavailable for AI training roles as a whole.
• Merit-based entry. DataAnnotation and others require no degree; your assessment is your application. Skill beats pedigree.
• Genuine flexibility. Asynchronous, work-from-anywhere, set your own hours around a primary job.
• Fast, frequent payment. Weekly payouts, often via PayPal, with short holds.
• You stay sharp. Reading and critiquing frontier-model code keeps your engineering judgment current, and you see how the best models actually reason.
• Real demand. Every major lab needs this continuously; the AI training dataset market is projected to grow from $3.9B in 2026 to $16.3B by 2033 (Grand View Research, June 2026).

Cons

• Income is lumpy and impermanent. Projects arrive in waves, queues empty without notice, and platforms can ghost even long-term contractors.
• Effective rate is below the headline. Unpaid calibration, instruction-reading, and downtime pull your effective rate below the dashboard rate.
• No benefits, and you owe the full self-employment tax.
• Quality pressure. Low quality scores cut your task flow or rate; the calibration bar is real.
• The work can be repetitive once the novelty of grading model code wears off.
• The field is crowded with scams (next section), so vetting every opportunity is on you.

Red Flags and Scams to Avoid

AI training is now a magnet for job scams. Reports to the FTC topped 105,000 in 2024, nearly three times the 2020 total, with reported losses over $513M (FTC, 24 Sep 2025). Learn these signals cold.

• Any upfront payment is a scam. Full stop. No legitimate platform charges for "training materials," "certification," "platform access," equipment, or a background check. Money flows one direction only: from the company to you.
• Hiring conducted entirely over WhatsApp/Telegram/iMessage with no verifiable company presence and no real interview is a serious red flag.
• Vague task descriptions. If a posting says "train AI models" but cannot tell you whether you will rate code, rank text, or label images, be skeptical. Legit platforms are transparent about tasks.
• Fake domains. Watch for lookalikes like scaleai-jobs.com instead of the real scale.com. Always apply on the platform's official domain, cross-checked from its main website.
• "Assessment" that is actually free labor. Some scams disguise unpaid production work as an endless "test." A legitimate qualification exercise is short and clearly bounded.
• Too-good-to-be-true rates with no skill gate. "$90/hr, no experience, start today" with an instant offer is the classic bait.

How to verify a real opportunity: look up the role on the company's official website; if it is not listed there, walk away. Legitimate platforms also have active subreddits, forums, or Discord communities where real contributors discuss payment and tasks. Every platform named in this guide passes these checks.

The 2026 Outlook: Is This Work Going Away?

The obvious worry: if AI is getting better at coding, will it stop needing humans to grade its code? The evidence points the other way for expert-level evaluation, even as it shifts the work upward.

Demand for high-quality human coding feedback is rising, not falling. Every frontier lab needs a continuous stream of expert human judgment to train, refine, and safety-test models, and human judgment is precisely the thing a model cannot yet reliably supply about its own outputs. As models get stronger, the bar for useful feedback rises: catching a model's remaining mistakes now requires more senior engineers, not fewer, which is exactly why the specialist tiers pay the most.

At the same time, the low end is under pressure. Routine, generalist annotation is being automated and commoditized, and the LSE analysis of the AI-jobs hype cycle (12 Mar 2026), which expects AI to keep disrupting coders' jobs while arguing that the real question is the quality of the work that remains, applies to this work too. The trajectory of the job is moving from "grade this snippet" toward "design the rubric, build the RL environment, adjudicate the hardest cases." That is a more senior, better-paid role, and it favors engineers who invest in specialization now.

The market backdrop supports staying in: Grand View Research projects the AI training dataset market to grow about 22.6% a year to 2033 (June 2026), and Mordor Intelligence puts the data labeling market at $2.61B in 2026, heading to $7.02B by 2031 (updated 11 Sep 2026). The honest forecast for a skilled coder: the floor of this work will keep eroding, the ceiling will keep rising, and your job is to position toward the ceiling by getting more specialized and maintaining high quality scores.

Bottom Line

Coding and software evaluation in AI training is a rare thing in remote work: genuinely legitimate, genuinely high-paying, and genuinely merit-based. Coding roles on this site typically pay Unavailable to Unavailable an hour today, with a median of Unavailable; senior specialists reach Mercor's advertised $120 to $200 tier ( Mercor, read 10 Nov 2026), and Mercor's guide says the very top full-time roles can pay upwards of $200,000 a year (updated 23 Jun 2026). The price of admission is passing a real skills assessment, positioning yourself as specialized rather than generalist, working across multiple platforms to smooth out lumpy queues, and handling your own taxes like the small business you now are. Avoid anything that asks you to pay, apply only on official domains, and treat the top of every advertised range as the specialist ceiling, not the default. Do that, and this is one of the best-paid uses of an engineer's spare hours available in 2026.