Skip to content
aitrainer.work - AI Training Jobs Platform
Engineering Full-Time
P-1 AI

Member of Technical Staff, AI Evaluation (US)

P-1 AI • United States

Company

P-1 AI

Annual salary

$200k – $300k/yr

Location

United States

Listed

39d ago

Experience:
5 to 15 years
Workplace:
US or Canada-based (remote), with on-site work one week every 6 weeks in San Mateo, CA. Relocation decided case by case.
Equity:
Competitive equity

Remote (listed in United States).

Send us your LinkedIn and CV. If your experience fits, we'll introduce you to the recruiter filling this role.

Apply for a referral →

The recruiter emails you before anything happens and may suggest other jobs that suit you better.

About this role

We are looking for a Member of Technical Staff, AI Evaluation with 5+ years of experience, blending physical engineering (mechanical, electrical, aerospace, robotics) and AI evals, to own how we measure Archie, our AI engineer for the physical world. You'll build the evaluation framework that shows how Archie compares to human engineers and what it takes to grow him from junior to senior engineer, and eventually to engineering superintelligence.

What you need

  • 5+ years of experience across physical engineering and/or AI work. The ideal candidate is strong in both, but deep strength in one with working knowledge of the other is fine
  • Hands-on AI evaluation experience: building eval tasks, designing judges, and running human evaluation for agentic systems
  • Physical engineering background (mechanical, electrical, aerospace, robotics, power systems), enough to recognize strong engineering design
  • The presence and confidence to lead work across SMEs, customers and cross-functional teams in a high-ownership role
  • Experience in scrappy, early-stage environments building new things from scratch (physical AI or robotics companies a plus)

What will you be doing?

  • Own Archie's evaluation framework from scratch, defining what "good" engineering design looks like when there's no single right answer
  • Work with engineering SMEs, customers, partners and internal teams to learn how top engineers work and where Archie succeeds or fails
  • Recreate real-world engineering tasks and failure modes in evaluation environments that the dev teams can rerun
  • Design and refine automated judges and human evaluation processes, and check that judges match expert human judgment
  • Turn eval results into clear priorities for improving Archie's engineering reasoning

Common questions

How do I apply for the Member of Technical Staff, AI Evaluation (US) role at P-1 AI? +

Use the Apply for a referral button on this page to send us your LinkedIn and CV. If your experience fits, we'll introduce you to the recruiter filling this role. They'll email you to check you're interested, then put you forward for this job or others that suit you better. It's free.

What does this P-1 AI role pay? +

The listing gives $200k – $300k/yr.

Is this role remote? +

Yes, the listing is remote. US or Canada-based (remote), with on-site work one week every 6 weeks in San Mateo, CA. Relocation decided case by case.

AI Evaluation Engineer interview prep

Questions AI Evaluation Engineer interviewers often ask. They aren't taken from this listing.

How do you design an evaluation set that catches the failure modes you're worried about, instead of mostly easy cases?

I start from known or suspected failure modes and write examples that specifically target them, rather than sampling broadly and hoping coverage happens naturally. A random sample tends to be dominated by easy cases the model already handles well, which tells you little about where it breaks.

What's your process for deciding whether a model regression is a real problem or noise in the eval?

I check whether the drop is consistent across multiple runs and whether it's concentrated in a specific category rather than spread thinly across everything. A real regression usually has a pattern behind it; pure noise tends to be scattered and doesn't reproduce reliably.

How do you evaluate a model's output when there's no single correct answer, only better and worse ones?

I define what specifically makes one response better, like factual grounding, completeness, or reasoning quality, and evaluate against those explicit criteria rather than an overall gut impression, since a vague preference judgment doesn't generalize well across raters or over time.

How do you keep an evaluation benchmark from becoming stale as the model improves and starts passing it trivially?

I track pass rates over time and treat a benchmark approaching ceiling as a signal to add harder cases, rather than treating consistently high scores as a finished success. A benchmark that stops discriminating between good and bad outputs has stopped being useful, even if the score still looks good.

See all 10 AI Evaluation Engineer interview questions →

Related roles

P-1 AI

Browse all referral roles

Salaried roles at startups, AI companies and established businesses, all filled by referral. One application covers every role.

See all referral roles →