Computer-Use Agent Evaluator

Computer-Use Agent Evaluator is a remote evaluation track for reviewing computer use agent evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and write the kind of structured feedback the modeling team can use to retrain.

Why this role matters

AI data reviewers help turn computer use agent evaluation outputs into auditable labels, rationales, and regression cases for AuraOne Human Data.

Responsibilities

  • Evaluate computer use agent evaluation model outputs against a versioned rubric and assign severity tags for Computer-Use Agent Evaluator assignments.
  • Compare paired responses and pick the stronger answer with a written rationale.
  • Label hallucinations, instruction-following failures, and unsafe content with structured tags.
  • Capture ambiguous prompts and route them back to the program team for rubric updates.
  • Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
  • Document recurring failure modes so the modeling team can target them in the next training run.

Qualifications

  • Prior evaluation, annotation, or human-rater experience on computer use agent evaluation or adjacent content for Computer-Use Agent Evaluator work.
  • Comfort applying multi-page rubrics consistently across long batches.
  • Clear written reasoning that names the issue and the rubric clause being applied.
  • Strong attention to detail and the ability to flag when a prompt itself is the problem.
  • Reliable async availability for at least 10 hours per week.

Example tasks

  • Compare two computer use agent evaluation model responses to the same prompt and pick the stronger one with rationale.
  • Tag an unsafe response with the correct policy category and severity.
  • Audit a 50-row batch for rubric consistency and report drift to the program lead.
  • Propose a rubric clarification after spotting a recurring failure mode.

Nice to have

  • Background in linguistics, content moderation, or trust & safety review.
  • Experience with inter-rater agreement metrics and calibration cycles.
  • Domain expertise that lets you spot subject-matter errors automated checks miss.

Skills

  • Model output evaluation
  • Rubric-based annotation
  • Severity tagging
  • Inter-rater calibration
  • Computer Use Agent evaluation
  • Software engineering and computer use
  • AI evaluation
  • Rubric writing
  • Expert review

Work model

Remote — US-eligible. Remote · Independent specialist contractor. Employment type: CONTRACTOR. Applicants must be authorized to work from US.

Compensation

Hourly rate confirmed after the interview process.

Application process

Apply through AuraOne's specialist intake for role-specific routing and review. Final project scope, schedule, and contractor terms are confirmed before placement.

Back to blog

Other Jobs To Apply

Truck Dispatcher – Fleet Operations (Remote)

Part-Time Medical Biller | Behavioral Health Practice | Remote

IT Field Support Technician, IT Operations

Remote Customer Service Contractor for Fortune 500 Companies

Remote Data Entry and Typing Clerk

Amazon Package Delivery Driver - Earn $21.50 - $29.00/hr - No Experience Required

12th Pass Freshers For Customer Care Executive

Coordinator, Creative Marketing (Disney Entertainment Television)

Media Archive Coordinator (National Geographic)

Video Game Analyst | Remote | $20/hr

Part-Time Medical Transcription Jobs from Home for Healthcare Experience Holders

Remote Part-Time Game Platform QA Tester

Games Tester

Data Entry Specialist - Work From Home

Radiation Therapist

Amazon Fulfillment Center Associate – Dothan, AL

Delivery Driver - Las Vegas - Mondays & Thursdays, 4am-9am (63185)

Dental Assistant - Training Available, No Degree Required (75454)

amazon work from home $15+/ Hour (Sign on Bonus!)!

Mobile Typing Job – Entry-Level Work with Weekly Payment

[Hiring] Advocacy Clinical Review Coordinator I/II/III @Excellus BCBS

Remote Overnight (3rd Shift) Customer Service Representative – Loan Approvals, Payment Arrangements & Client Support (11 pm – 7 am)

SATs Tutor

Inside Sales Representative

Dynamic reputed company Representative (CSR) – Evening Shift (1:00 PM – 9:00 PM) – Immediate Start at arenaflex

Remote Work From Home Call Center Representative Agent - Part Time Panelists Needed

Copy Editor and Proofreader-Remote Part Time and Full Time

United Airlines Reservations Representative (Remote)

Online Game Tester

Package Handler at UPS: Join Our Dynamic Team!

Hiring Now - Work from Home - No Experience

Social Media Moderator at At Your School (AYS, Inc.) United States

Director, Sales - Walmart at Novolex: 100% Remote (Bentonville, AR)

Manager Payroll

(Work From Home Jobs) Fedex Data Entry Jobs $25/Hour

Delta Airlines Remote Customer Support - Work From Your Own Home

Flight Attendant(PT)-Los Angeles,CA

IT Help Desk 1 Intern Job at AUDI JLR LOTUS BMW MOTO in Wilmington

OnlyFans Chatter Entry Level (No Experience)

RN Team Director (Hospice)

Virtual Data Entry Assistant - Entry Level Part-Time Opportunity with Competitive Hourly Rate

Remote Part‑Time Customer Success Associate – Evening Shift (Fri‑Sun, 2 pm‑10 pm) – arenaflex

Remote Senior Partnership Manager

Cruise Ship Staff Member

Remote Licensed Insurance Rep - $19/hr + Full Benefits

Manufacturing Quality Technician, Optimus

Remote Customer Service Contractor for Fortune 500 Companies

Internship, Service Technician Trainee (Fall 2025)

కాల్ సెంటర్ బిపిఓ ఎగ్జిక్యూటివ్

Online Chat Support & Sales Representative – Work from Anywhere