Skip to content
aitrainer.work - AI Training Jobs Platform
Software Engineering Innodata

QA / Evaluation Lead

Innodata • Hybrid - Washington D.C

Education

Bachelor

Type

Hourly

Pay

$45-50/hr

Your rate depends on your experience and the task.

Last checked

3d ago

Checked

Send this job to a friend

New roles in Telegram: @aitrainerwork

About this role

From the Innodata listing

Innodata (Nasdaq: INOD) is a global data engineering company. We believe that data and Artificial Intelligence (AI) are inextricably linked. Our mission is to enable the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise required to build AI systems that can be trusted at scale. We provide a range of transferable solutions, platforms, and services for Generative AI / AI builders and adopters. In every relationship, we honor our 36+ year legacy delivering the highest quality data and outstanding outcomes for our customers. About the Program: Innodata's Federal Practice builds the trusted data layer for critical infrastructure Trust & Safety work. Partnering with a leading systems integrator, we're delivering a modern, governed data services platform in a secure federal (IL4) environment. Over an intensive 20-week phase, you'll help stand up a data services storefront, a DataCard governance framework, synthetic data integration, and Databricks write-back capabilities. About the Role: As the QA/Evaluation Lead, you'll own quality and evaluation across the platform. You'll design the evaluation framework that measures whether our data services and outputs meet the bar, build repeatable test and validation processes, and give the team an objective read on readiness at each milestone. Partnering with the Delivery Owner and engineering leads, you'll turn quality from an afterthought into a measurable, demonstrable strength. It's a role for someone who thinks rigorously about evaluation and takes pride in evidence-backed quality. Key Responsibilities: Design and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus — metric selection (Cohen's kappa, Fleiss, Krippendorff's alpha), sampling design, adjudication workflow, and agreement thresholds Define evaluation framework architecture: test and evaluation plans, IAA targets, drift detection gates, and model performance metrics per SOW Section 2.9 Configure and operate sampling-based quality control across the self-service and white-glove annotation paths during Phase D corpus production Design and implement confidence-threshold escalation routing from automated annotation to senior-annotator adjudication Validate quality scoring and IAA computation within the Innodata data layer Support AI Solutions Engineer on evaluation design for SAM 2 and Frontier model API validation — define what 'good enough' looks like quantitatively Produce evaluation framework documentation for the Phase 1 NPP closeout package, including per-DataCard documentation with the SA Must-Have Qualifications: Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis. 6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs IAA methodology expertise — Cohen's kappa, Fleiss' kappa, Krippendorff's alpha: hands-on, not theoretical Evaluation framework design for AI/ML training data programs QC process design: sampling methodology, escalation workflows, adjudication protocols Python for QC tooling, metric computation, and statistical analysis Active Secret clearance with TS/SCI eligibility Nice-to-Have Qualifications: Prior DoD or IC data quality program experience CVAT or equivalent annotation platform QC workflow configuration Drift detection and model monitoring methodology Experience with FMV / video annotation quality standards The expected hourly salary range for this position is $45 to $50 p/hour, based on experience, skills, and qualifications. Note to Candidates: This role is not a project manager with QC responsibilities — it is a methodology expert who owns the intellectual framework behind data quality on a federal AI program. The right candidate can walk into a meeting with Government evaluators and explain exactly

Requirements

  • Must be eligible to work in Hybrid - Washington D.C
  • Fluent proficiency in English (Written & Verbal)
  • Reliable high-speed internet connection
  • Bachelor's degree or equivalent professional experience
  • Demonstrated expertise in Software Engineering

Interview prep

This listing mentions these tools. Brush up before a technical screen:

Why this role

This QA / Evaluation Lead position pays $45-50/hr. You need solid Software Engineering knowledge, and you learn the AI training steps on the job.

Skills and categories

More jobs in these fields:

Related jobs

Verified reviews

Loading reviews…

Community reviews

Loading reviews…
💬

Share your experience with Innodata

Sign in to leave a review

Common questions

Can I choose my own hours?

Usually, yes. You log in when you want, pick up a task, finish it and submit it, with no meetings. Other people work from the same pool of tasks, so some days there may be no work until more is added.

Do task-based AI training roles require an interview?

Most use an online assessment or a test task instead of a live interview. When the listing says which, it's in the role details on this listing.

What does Software Engineering work look like for a QA / Evaluation Lead?

You check AI answers to Software Engineering questions, correct their errors and write expert reasoning. As a QA / Evaluation Lead, you're hired for that judgment, so hands-on Software Engineering experience matters.

What specific skills does this listing call for?

Coding, Data Annotation, TypeScript, and Python are named directly in the listing. If you don't have hands-on experience with these, expect the screening to test for them.

How much does this specific role pay?

This listing is posted at $45-50/hr, an hourly rate. Where you land in the range depends on your background and the assessment. Pay can change after we check the listing, so confirm the rate on the platform's application page.

Can I apply from outside Hybrid - Washington D.C?

This role is open only to people based in Hybrid - Washington D.C. The limit is usually about where the platform can legally hire, so applying from elsewhere rarely leads to an offer. Check the description for tax or right-to-work rules before you apply.