AI Evaluation Engineer (Python)
Turing • Remote
Education
Bachelor's
Type
Hourly
Pay
$25-55/hr est.
Turing does not post a rate for this role. This is our estimate for similar roles. How we estimate
Last checked
10d ago
Checked
Only apply if this role fits your skills and interests.
What we know about this role
- Interview
- Required i As stated in the listing: “Duration of contract : 2 months”
- Weekly hours
- 40 hrs/week
- Timezone overlap
- Partial overlap with US Pacific time required
- Contract length
- 8 weeks
About this role
From the Turing listing
About Turing
Turing is one of the world's fastest-growing AI companies accelerating the advancement and deployment of powerful AI systems. Turing helps customers in two ways: working with the world's leading AI labs to advance frontier model capabilities and leveraging that work to build real-world AI systems that help businesses solve complex problems and unlock new opportunities. We are seeking detail-oriented Python developers to join Turing's AI Evaluation team as AI Evaluation Specialists. In this role, you will design and create evaluation tasks that measure the capabilities of advanced AI systems on real-world software engineering challenges. You will author structured tasks, reference solutions, verification criteria, and domain-specific guidance used to evaluate how effectively AI models perform Python-based development work. Your contributions will directly influence the benchmarking and improvement of next-generation AI systems used by leading AI organizations.
What does day-to-day look like?
- Design realistic Python-focused software engineering evaluation tasks for AI agents
- Write clear, precise instructions that define expected outputs, constraints, and success criteria
- Create reference solutions that successfully solve evaluation tasks and satisfy validation requirements
- Develop human-readable verifier descriptions that document expected behaviors and evaluation checks
- Author domain-specific knowledge files that teach AI systems relevant workflows, conventions, and best practices without revealing solutions
- Review tasks for clarity, consistency, edge cases, and evaluation quality
- Analyze AI-generated outputs and identify common failure patterns
- Collaborate with researchers and evaluation teams to improve benchmark quality and coverage
- Follow established task-authoring standards and quality guidelines
Requirements
- Bachelor's degree or higher in Computer Science, Software Engineering, Information Technology, or a related field
- 3–5 years of hands-on Python development experience
- Strong proficiency in Python and core software engineering concepts
- Experience developing applications, automation scripts, APIs, data-processing workflows, or backend systems using Python
- Strong understanding of data structures, algorithms, debugging, testing, and software development best practices
- Excellent written English with the ability to write clear, precise, and unambiguous technical instructions
- Ability to think critically about how AI systems interpret instructions and solve technical problems
- Comfortable working with structured formats such as JSON, Markdown, YAML, DOCX, and XLSX
Nice to have
- Experience with LLM evaluation, prompt engineering, or AI benchmarking
- Experience creating technical assessments, coding challenges, or educational content
- Experience with Docker, containers, or cloud-based development environments
Perks of Freelancing With Turing
- Work on cutting-edge AI projects with leading AI research organizations
- Flexible remote work opportunities
- Opportunity to influence the evaluation of next-generation AI systems
- Collaborate with a global network of highly skilled professionals
Offer details:
- Commitments Required : 40 hours per week with overlap of 4 hours with PST
- Engagement type : Contractor assignment/freelancer (no medical/paid leave)
- Duration of contract : 2 months; [expected start date is next week]
Evaluation Process
- One round of technical interview (or) Automated Live coding challenge
How long hiring takes
On the platforms we refer people to, half of hires came within about 25 days of applying. Time from applying to hire:
AI Evaluation Engineer interview prep
Questions AI Evaluation Engineer interviewers often ask. They aren't taken from this listing.
How do you design an evaluation set that catches the failure modes you're worried about, instead of mostly easy cases?
I start from known or suspected failure modes and write examples that specifically target them, rather than sampling broadly and hoping coverage happens naturally. A random sample tends to be dominated by easy cases the model already handles well, which tells you little about where it breaks.
What's your process for deciding whether a model regression is a real problem or noise in the eval?
I check whether the drop is consistent across multiple runs and whether it's concentrated in a specific category rather than spread thinly across everything. A real regression usually has a pattern behind it; pure noise tends to be scattered and doesn't reproduce reliably.
How do you evaluate a model's output when there's no single correct answer, only better and worse ones?
I define what specifically makes one response better, like factual grounding, completeness, or reasoning quality, and evaluate against those explicit criteria rather than an overall gut impression, since a vague preference judgment doesn't generalize well across raters or over time.
How do you keep an evaluation benchmark from becoming stale as the model improves and starts passing it trivially?
I track pass rates over time and treat a benchmark approaching ceiling as a signal to add harder cases, rather than treating consistently high scores as a finished success. A benchmark that stops discriminating between good and bad outputs has stopped being useful, even if the score still looks good.
This listing mentions this tool. Brush up before a technical screen:
Typical contract length
Across 11 Turing STEM listings we've tracked, contracts typically run about 8.5 weeks. Length varies by project. This listing's 8-week contract is shorter than the typical 8.5-week Turing STEM engagement.
Why this role
This AI Evaluation Engineer role is for Python specialists to build and grade coding evaluation tasks that test how well AI models handle real Python problems. It's a narrower, Python-only counterpart to Turing's multi-language evaluation engineer role.
Talent pool
We're light on STEM candidates
We've tracked 1,042 STEM listings and have few candidates to put forward for them. Set up a profile and we'll consider you for roles like this one.
Set up your profileSkills and categories
More jobs in these fields:
Related jobs
Verified reviews
Community reviews
Leave your review
Common questions
Do I need to be a software engineer to work for Turing?
No, not anymore. Turing built its name matching senior engineers with Silicon Valley companies, but it has since expanded into AGI infrastructure work and now hires non-engineering domain experts, technical writers, and researchers for post-training data annotation and RLHF. A strong analytical background and excellent English matter more than coding ability.
How does Turing's talent matching work?
Turing calls it the Intelligent Talent Cloud. You build a profile and go through vetting (automated tests, an AI-powered interview, practical skill assessments), and once vetted, Turing's algorithm surfaces your profile directly to partner companies like Fortune 500s and top AI labs. You don't browse listings or bid on work; matches come to you.
Can I choose my own hours?
Usually, yes. You log in when you want, pick up a task, finish it and submit it, with no meetings. Other people work from the same pool of tasks, so some days there may be no work until more is added.
What does STEM work look like for an AI Evaluation Engineer (Python)?
You check AI answers to STEM questions, correct their errors and write expert reasoning. As an AI Evaluation Engineer (Python), you're hired for that judgment, so hands-on STEM experience matters.
What specific skills does this listing call for?
Domain-Specific Languages, Coding, Python, and English are named directly in the listing. If you don't have hands-on experience with these, expect the screening to test for them.
How many hours per week does this role require?
Based on the listing, this role is scoped at about 40 hours per week, over roughly a 8-week contract. Plan your week around it.
How much does this specific role pay?
The listing doesn't state a rate. The $25-55/hr shown here is our estimate from the role type and location (see /pay-methodology), so treat it as a rough guide and confirm the actual rate with the platform before committing time.
What happens when I click Apply on this listing?
Apply opens Turing's site, where you create an account and follow its sign-up steps.
Is there an interview?
Yes. One round of technical interview (or) Automated Live coding challenge