Computer-Use Agent Evaluator

Computer-Use Agent Evaluator is a remote evaluation track for reviewing computer use agent evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain.

Why this role matters

AI data reviewers help turn computer use agent evaluation outputs into auditable labels, rationales, and regression cases for AuraOne Human Data.

Responsibilities

  • Evaluate computer use agent evaluation model outputs against a versioned rubric and assign severity tags for Computer-Use Agent Evaluator assignments.
  • Compare paired responses and pick the stronger answer with a written rationale.
  • Label hallucinations, instruction-following failures, and unsafe content with structured tags.
  • Capture ambiguous prompts and route them back to the program team for rubric updates.
  • Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
  • Document recurring failure modes so the modeling team can target them in the next training run.

Qualifications

  • Prior evaluation, annotation, or human-rater experience on computer use agent evaluation or adjacent content for Computer-Use Agent Evaluator work.
  • Comfort applying multi-page rubrics consistently across long batches.
  • Clear written reasoning that names the issue and the rubric clause being applied.
  • Strong attention to detail and the ability to flag when a prompt itself is the problem.
  • Reliable async availability for at least 10 hours per week.

Example tasks

  • Compare two computer use agent evaluation model responses to the same prompt and pick the stronger one with rationale.
  • Tag an unsafe response with the correct policy category and severity.
  • Audit a 50-row batch for rubric consistency and report drift to the program lead.
  • Propose a rubric clarification after spotting a recurring failure mode.

Nice to have

  • Background in linguistics, content moderation, or trust & safety review.
  • Experience with inter-rater agreement metrics and calibration cycles.
  • Domain expertise that lets you spot subject-matter errors automated checks miss.

Skills

  • Model output evaluation
  • Rubric-based annotation
  • Severity tagging
  • Inter-rater calibration
  • Computer Use Agent evaluation
  • Software engineering and computer use
  • AI evaluation
  • Rubric writing
  • Expert review

Work model

Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.

Compensation

Hourly rate confirmed after the interview process.

Application process

Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.

Back to blog

Other Jobs To Apply

Open Roles for "Darts" Audition Movie

Remote Editor, Children's & YA - Build Publishing Job at Sourcebooks in Napervil

Warehouse Associate (PT or FT flex schedules)

Night Auditor- Hilton Ocala

Warehouse Associate (1st Shift)

Help Desk Analyst - Trenton, NJ(Remote)

Computer-Use Agent Evaluator

Help Desk Analyst - Trenton, NJ(Remote)

Area Manager, Early Career - (2027) - Illinois

Area Manager Intern – Summer 2027 (MO, OH, KY)

HV Administrative Assistant (Remote)

Medical Scribe and Medical Assistant

Remote Video Editor: Social-first Storytelling Pro

Seasonal Store Sales Associate, Levi's®, Citadel Outlets, Commerce, CA

Warehouse Cargo Agents - JFK Airport

Security Operations Center (SOC) Analyst - Remote

Specialty Warehouse Associate

Executive Assistant to the CEO | Bookyourdata

Electrical Apprentice (Comm/Resi)

Area Manager, Early Career (2027) – Ohio

Healthcare Provider- fully remote, flexible schedule, cannabis certifications

CDL A Owner Operator Amazon

Telehealth Follow-Up Nurse (LPN) - Remote! (10:30am - 7:00pm CST Shift)

Data Scientist /data analyst entry level /AI /software programmer(Remote)

Support Analyst - Remote / Telecommute

Remote Weekend Admission Liaison

Warehouse Associate – Cold Storage / Forklift Operator

Remote Client Agent (Work From Home) | Flexible Schedule

Manager - P&C Systems | Remote- United States

HR Business Partner , NASC PXT

Data Entry Clerk Work From Home - Part-Time Focus Group Participants (Up To $750/Week)

Remote 1099 Independent Contractor - Call Center Brand Ambassador - Pandora Jewelry (#OPA)

Night Owl LPN OR RN Needed – Weekly Pay + Flexible Schedule

Entry Level Creative Writer (Home Based) (Creative)

Entry level Job Delta Data Entry Jobs – Remote High Paying Opportunities Job ID- 680 Job ID- 2883

Remote Copywriter for Impactful Campaigns | Flexible Hours

Solutions Architect

Project Manager - Remote

Bilingual Medical Scribe

Bilingual Medical Scribe

Travel Radiology Tech - $1,728 to $1,986 per week in McAllen, TX - Now Hiring

Remote Social Media Manager — AI Gaming Brand Growth

Fully Remote Entry Level

Area Manager II

Verizon FIOS Sales Agent

Robotics Warehouse Associate

Contract Remote English Teacher — U.S. Elementary Schools

Overnight Remote Customer Experience Representative, Robinhood Concierge

Airport Cabin Agent F/T - LGA