In this role, you will evaluate AI-generated coding interactions and provide expert feedback to help improve the quality, reliability, and usefulness of AI-powered development tools.

• Evaluate AI-generated coding interactions from beginning to end, assessing their overall quality and effectiveness.

• Determine whether AI outputs are useful, technically sound at a high level, and aligned with the reasoning and practices of an experienced software engineer.

• Assess the quality of explanations, reasoning, and recommendations rather than focusing exclusively on code correctness.

• Differentiate between varying levels of response quality and apply consistent evaluation standards.

• Provide clear, detailed, and opinionated written feedback on what works, what does not, and what may feel inaccurate, misleading, or ineffective.

• Assess whether AI models guide developers effectively, provide meaningful context, and build trust through their interactions.

• Help define and refine quality benchmarks for AI coding agents, including tools such as OpenAI Codex, Claude Code, and Cursor.

• Record short video explanations to communicate evaluation findings and provide additional context.

Requirements

The ideal candidate is a highly experienced software engineer with strong Python expertise, a deep understanding of modern AI-assisted development workflows, and the ability to make rigorous technical assessments.

• Senior, Staff, or Principal-level software engineering experience, or equivalent professional expertise.

• Strong professional background in Python, with additional experience in TypeScript or JavaScript also relevant.

• Hands-on experience with at least one of the following AI coding tools: OpenAI Codex, Claude Code, or Cursor.

• Deep familiarity with modern AI-assisted software development workflows.

• Ability to evaluate code and technical responses without necessarily executing the code or reviewing every line.

• Strong written and spoken English proficiency (B2 level or above), as the role involves detailed written feedback and recorded video explanations.

• Confidence in providing direct, constructive, and well-supported opinions.

• Excellent critical thinking and communication skills, with a high standard for engineering quality.

• Ability to distinguish between technical correctness and genuinely useful engineering guidance.

Nice to have:

• Previous exposure to prompt design, AI evaluation, or model assessment workflows.

• Experience mentoring senior engineers or defining engineering standards and best practices.

Benefits

• Competitive hourly compensation ranging from $100 to $200, depending on experience and location.

• Fully remote contract opportunity, with worldwide availability subject to regional eligibility.

• Flexible workload of approximately 10–20 hours per week.

• Flexible working hours.

• Ongoing engagement, with individual projects typically lasting from two weeks to several months.

• Opportunities to participate in additional projects based on evaluation performance.

• Potential to start as soon as the take-home assessment is completed successfully and a project position becomes available.

• Streamlined selection process consisting of one take-home evaluation exercise and a recorded video walkthrough, with no interview required.

• Opportunity to contribute to the development and evaluation of advanced AI coding technologies.