Skip to content
aitrainer.work - AI Training Jobs Platform
STEM Turing

AI Evaluation Engineer (Python)

Turing • Remote

Education

Bachelor's

Type

Hourly

Pay

$25-55/hr est.

Turing does not post a rate for this role. This is our estimate for similar roles. How we estimate

Last checked

10d ago

Checked

Only apply if this role fits your skills and interests.

Send this job to a friend

New roles in Telegram: @aitrainerwork

What we know about this role

Interview
Required i As stated in the listing: “Duration of contract : 2 months”
Weekly hours
40 hrs/week
Timezone overlap
Partial overlap with US Pacific time required
Contract length
8 weeks

About this role

From the Turing listing

About Turing

Turing is one of the world's fastest-growing AI companies accelerating the advancement and deployment of powerful AI systems. Turing helps customers in two ways: working with the world's leading AI labs to advance frontier model capabilities and leveraging that work to build real-world AI systems that help businesses solve complex problems and unlock new opportunities. We are seeking detail-oriented Python developers to join Turing's AI Evaluation team as AI Evaluation Specialists. In this role, you will design and create evaluation tasks that measure the capabilities of advanced AI systems on real-world software engineering challenges. You will author structured tasks, reference solutions, verification criteria, and domain-specific guidance used to evaluate how effectively AI models perform Python-based development work. Your contributions will directly influence the benchmarking and improvement of next-generation AI systems used by leading AI organizations.

What does day-to-day look like?

  • Design realistic Python-focused software engineering evaluation tasks for AI agents
  • Write clear, precise instructions that define expected outputs, constraints, and success criteria
  • Create reference solutions that successfully solve evaluation tasks and satisfy validation requirements
  • Develop human-readable verifier descriptions that document expected behaviors and evaluation checks
  • Author domain-specific knowledge files that teach AI systems relevant workflows, conventions, and best practices without revealing solutions
  • Review tasks for clarity, consistency, edge cases, and evaluation quality
  • Analyze AI-generated outputs and identify common failure patterns
  • Collaborate with researchers and evaluation teams to improve benchmark quality and coverage
  • Follow established task-authoring standards and quality guidelines

Requirements

  • Bachelor's degree or higher in Computer Science, Software Engineering, Information Technology, or a related field
  • 3–5 years of hands-on Python development experience
  • Strong proficiency in Python and core software engineering concepts
  • Experience developing applications, automation scripts, APIs, data-processing workflows, or backend systems using Python
  • Strong understanding of data structures, algorithms, debugging, testing, and software development best practices
  • Excellent written English with the ability to write clear, precise, and unambiguous technical instructions
  • Ability to think critically about how AI systems interpret instructions and solve technical problems
  • Comfortable working with structured formats such as JSON, Markdown, YAML, DOCX, and XLSX

Nice to have

  • Experience with LLM evaluation, prompt engineering, or AI benchmarking
  • Experience creating technical assessments, coding challenges, or educational content
  • Experience with Docker, containers, or cloud-based development environments

Perks of Freelancing With Turing

  • Work on cutting-edge AI projects with leading AI research organizations
  • Flexible remote work opportunities
  • Opportunity to influence the evaluation of next-generation AI systems
  • Collaborate with a global network of highly skilled professionals

Offer details:

  • Commitments Required : 40 hours per week with overlap of 4 hours with PST
  • Engagement type : Contractor assignment/freelancer (no medical/paid leave)
  • Duration of contract : 2 months; [expected start date is next week]

Evaluation Process

  • One round of technical interview (or) Automated Live coding challenge

How long hiring takes

On the platforms we refer people to, half of hires came within about 25 days of applying. Time from applying to hire:

Under 2 weeks
~30%
2 weeks to 6 weeks
~40%
6 weeks to 3 months
~15%
Over 3 months
~15%

AI Evaluation Engineer interview prep

Questions AI Evaluation Engineer interviewers often ask. They aren't taken from this listing.

How do you design an evaluation set that catches the failure modes you're worried about, instead of mostly easy cases?

I start from known or suspected failure modes and write examples that specifically target them, rather than sampling broadly and hoping coverage happens naturally. A random sample tends to be dominated by easy cases the model already handles well, which tells you little about where it breaks.

What's your process for deciding whether a model regression is a real problem or noise in the eval?

I check whether the drop is consistent across multiple runs and whether it's concentrated in a specific category rather than spread thinly across everything. A real regression usually has a pattern behind it; pure noise tends to be scattered and doesn't reproduce reliably.

How do you evaluate a model's output when there's no single correct answer, only better and worse ones?

I define what specifically makes one response better, like factual grounding, completeness, or reasoning quality, and evaluate against those explicit criteria rather than an overall gut impression, since a vague preference judgment doesn't generalize well across raters or over time.

How do you keep an evaluation benchmark from becoming stale as the model improves and starts passing it trivially?

I track pass rates over time and treat a benchmark approaching ceiling as a signal to add harder cases, rather than treating consistently high scores as a finished success. A benchmark that stops discriminating between good and bad outputs has stopped being useful, even if the score still looks good.

See all 10 AI Evaluation Engineer interview questions →

This listing mentions this tool. Brush up before a technical screen:

Typical contract length

Across 11 Turing STEM listings we've tracked, contracts typically run about 8.5 weeks. Length varies by project. This listing's 8-week contract is shorter than the typical 8.5-week Turing STEM engagement.

Why this role

This AI Evaluation Engineer role is for Python specialists to build and grade coding evaluation tasks that test how well AI models handle real Python problems. It's a narrower, Python-only counterpart to Turing's multi-language evaluation engineer role.

Talent pool

We're light on STEM candidates

We've tracked 1,042 STEM listings and have few candidates to put forward for them. Set up a profile and we'll consider you for roles like this one.

Set up your profile

Skills and categories

More jobs in these fields:

Related jobs

Verified reviews

Loading reviews…

Community reviews

Loading reviews…
💬

Share your experience with Turing

Sign in to leave a review

Common questions

Do I need to be a software engineer to work for Turing?

No, not anymore. Turing built its name matching senior engineers with Silicon Valley companies, but it has since expanded into AGI infrastructure work and now hires non-engineering domain experts, technical writers, and researchers for post-training data annotation and RLHF. A strong analytical background and excellent English matter more than coding ability.

How does Turing's talent matching work?

Turing calls it the Intelligent Talent Cloud. You build a profile and go through vetting (automated tests, an AI-powered interview, practical skill assessments), and once vetted, Turing's algorithm surfaces your profile directly to partner companies like Fortune 500s and top AI labs. You don't browse listings or bid on work; matches come to you.

Can I choose my own hours?

Usually, yes. You log in when you want, pick up a task, finish it and submit it, with no meetings. Other people work from the same pool of tasks, so some days there may be no work until more is added.

What does STEM work look like for an AI Evaluation Engineer (Python)?

You check AI answers to STEM questions, correct their errors and write expert reasoning. As an AI Evaluation Engineer (Python), you're hired for that judgment, so hands-on STEM experience matters.

What specific skills does this listing call for?

Domain-Specific Languages, Coding, Python, and English are named directly in the listing. If you don't have hands-on experience with these, expect the screening to test for them.

How many hours per week does this role require?

Based on the listing, this role is scoped at about 40 hours per week, over roughly a 8-week contract. Plan your week around it.

How much does this specific role pay?

The listing doesn't state a rate. The $25-55/hr shown here is our estimate from the role type and location (see /pay-methodology), so treat it as a rough guide and confirm the actual rate with the platform before committing time.

What happens when I click Apply on this listing?

Apply opens Turing's site, where you create an account and follow its sign-up steps.

Is there an interview?

Yes. One round of technical interview (or) Automated Live coding challenge