Senior Software Engineer — AI Evaluation & Benchmarks
Alignerr • Remote
Education
Master's
Type
Hourly
Pay
$80-100/hr
Your rate depends on your experience and the task.
Last checked
12d ago
Checked
What we know about this role
- Contract length
- 13 weeks
About this role
From the Alignerr listing
What You'll Do
- Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks
- Build and maintain scalable data pipelines for AI evaluation workflows
- Analyze AI-generated code for correctness, reliability, and edge-case failures
- Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality
- Work with large code repositories and multi-language environments
- Collaborate on systems that improve how AI models understand and generate software
- Provide detailed technical feedback on model performance and failure patterns
- Contribute to the design of evaluation frameworks that set industry standards
About the Role
What if the code you write could determine how smart the next generation of AI truly is? We're looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models — the systems that decide whether an AI can actually reason, debug, and write production-quality software. This is high-impact, technically demanding work at the intersection of software engineering and AI research. You'll work with large codebases, multiple programming languages, and scalable infrastructure to create evaluation systems that push the boundaries of what AI can do. This is a fully remote contract role. If you thrive in fast-paced engineering environments and want your work to directly shape the trajectory of AI — this is the role.
- Organization: Alignerr
- Type: Hourly Contract
- Location: Remote
- Contract Length: 3 Months
- Commitment: Full-time availability preferred
Who You Are
- 4+ years of professional software engineering experience — this is non-negotiable
- Experience working at a high-growth tech company or top-tier software organization
- Expert proficiency in Python — you write clean, performant, well-tested Python code
- Hands-on experience with code repositories and working in large, complex codebases
- Proven experience designing and implementing LLM coding benchmarks and data pipelines
- Track record of working in high-performance engineering environments with large-scale products or platforms
- Strong command of version control systems (Git) and modern development workflows
- Bilingual or native English speaker with strong written communication skills
- Self-directed, technically rigorous, and comfortable operating with autonomy
What Makes a Perfect Match
Candidates with these additional qualifications have the highest chance of success:
- Senior or Lead-level engineering profiles with a history of technical ownership
- Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field — or equivalent professional experience
- Proficiency in one or more additional languages: JavaScript, Go, C++, or other relevant languages
- Experience with CI/CD pipelines and writing robust unit tests (pytest, Mocha, JUnit)
- Background in security engineering or significant open-source contributions
- Familiarity with AI/ML evaluation methodologies or model benchmarking
Why Join Us
- Work on cutting-edge AI evaluation projects alongside world-class research teams
- Fully remote — work from anywhere with a reliable internet connection
- Your benchmarks directly influence how the most advanced AI systems in the world are measured and improved
- Freelance autonomy with meaningful, high-stakes engineering work
- Collaborate with a global community of elite engineers and researchers
- Potential for contract extension and ongoing engagement as new evaluation challenges emerge
AI Evaluation Engineer interview prep
Questions AI Evaluation Engineer interviewers often ask. They aren't taken from this listing.
How do you design an evaluation set that catches the failure modes you're worried about, instead of mostly easy cases?
I start from known or suspected failure modes and write examples that specifically target them, rather than sampling broadly and hoping coverage happens naturally. A random sample tends to be dominated by easy cases the model already handles well, which tells you little about where it breaks.
What's your process for deciding whether a model regression is a real problem or noise in the eval?
I check whether the drop is consistent across multiple runs and whether it's concentrated in a specific category rather than spread thinly across everything. A real regression usually has a pattern behind it; pure noise tends to be scattered and doesn't reproduce reliably.
How do you evaluate a model's output when there's no single correct answer, only better and worse ones?
I define what specifically makes one response better, like factual grounding, completeness, or reasoning quality, and evaluate against those explicit criteria rather than an overall gut impression, since a vague preference judgment doesn't generalize well across raters or over time.
How do you keep an evaluation benchmark from becoming stale as the model improves and starts passing it trivially?
I track pass rates over time and treat a benchmark approaching ceiling as a signal to add harder cases, rather than treating consistently high scores as a finished success. A benchmark that stops discriminating between good and bad outputs has stopped being useful, even if the score still looks good.
This listing mentions these tools. Brush up before a technical screen:
Why this role
- Work on cutting-edge AI evaluation projects alongside world-class research teams - Fully remote — work from anywhere with a reliable internet connection
Skills and categories
More jobs in these fields:
Related jobs
Software Developer — Get Paid to Build Your Own Project with an AI Coding Assistant
Alignerr • Software Engineering
$100 /hr
Software Engineer Task Author (AI Training)
Alignerr • Software Engineering
$120 /hr
Senior Software Engineer — Agentic Coding
Alignerr • Software Engineering
$120 /hr
Senior Software Engineer (AI Infrastructure)
Alignerr • Software Engineering
$100 /hr
Verified reviews
Community reviews
Leave your review
Common questions
How soon can I start earning on Alignerr after passing the assessment?
After you pass, you verify your identity through Persona and set up billing through Deel. Then you wait in a pool, sometimes for weeks or months, until a project that fits your skills starts and assigns you.
Does Alignerr have a trainer community?
Yes. When you join a project, you get Slack channels where admins answer rubric questions and trainers help each other, which helps most when guidelines change mid-project.
What does task-based AI training work look like?
Short, hands-on tasks such as recording videos, sorting images, rating AI answers or checking data. Most take 5 to 60 minutes each.
What does Software Engineering work look like for a Senior Software Engineer — AI Evaluation & Benchmarks?
You check AI answers to Software Engineering questions, correct their errors and write expert reasoning. As a Senior Software Engineer — AI Evaluation & Benchmarks, you're hired for that judgment, so hands-on Software Engineering experience matters.
What specific skills does this listing call for?
Coding, Python, JavaScript, and Bilingual are named directly in the listing. If you don't have hands-on experience with these, expect the screening to test for them.
How much does this specific role pay?
This listing is posted at $80-100/hr, an hourly rate. Where you land in the range depends on your background and the assessment. Pay can change after we check the listing, so confirm the rate on the platform's application page.
What happens when I click Apply on this listing?
Apply opens Alignerr's site, where you create an account and follow its sign-up steps.
Do I need a Master's to qualify?
This role lists a master's degree. The domain assessment usually matters more, and if you pass it the degree is often secondary. Some platforms check credentials, so list your qualifications accurately.
How do I get approved on Alignerr?
You pass a timed assessment in your field, such as Python, physics or a language. After that you can join paid projects.