In this role, you will assess AI-generated coding interactions from the perspective of an experienced software engineer, focusing on technical judgment, clarity, usefulness, and the quality of the overall developer experience.

• Evaluate AI-generated coding interactions from beginning to end, including responses, explanations, and recommendations.

• Assess whether outputs are useful, broadly correct, and consistent with the judgment and practices of a strong software engineer.

• Review the quality of explanations and reasoning rather than focusing exclusively on code syntax or implementation details.

• Distinguish between different levels of response quality using consistent evaluation criteria.

• Provide clear, detailed, and constructive written feedback explaining what worked, what did not, and what felt inaccurate, misleading, or ineffective.

• Evaluate whether AI agents guide developers effectively, build trust, and provide meaningful assistance rather than simply generating output.

• Help define and refine quality standards for interactions with AI coding tools such as OpenAI Codex, Claude Code, and Cursor.

• Record short video explanations to communicate evaluation findings and provide additional context.

Requirements

The ideal candidate is a senior software engineering professional with extensive experience, strong technical judgment, and a deep understanding of modern AI-assisted development workflows. You should be comfortable evaluating technical work critically, making subjective but well-reasoned assessments, and communicating your conclusions clearly.

• Senior, Staff, or Principal-level software engineer, or equivalent professional experience.

• Strong background in TypeScript, JavaScript, or Python.

• Hands-on experience using at least one of the following tools: OpenAI Codex, Claude Code, or Cursor.

• Deep familiarity with modern AI-assisted software development workflows.

• Ability to evaluate code and technical responses without necessarily executing the code or reviewing every line.

• Strong written and spoken English proficiency (B2 level or above), as detailed written feedback and short recorded video explanations are required.

• Confidence in providing direct, thoughtful, and opinionated feedback.

• High standards for software engineering quality and a strong understanding of what makes technical guidance genuinely useful.

• Ability to assess engineering judgment, communication quality, and developer experience beyond basic technical correctness.

Nice to have:

• Previous experience with prompt design, AI evaluation, or model assessment workflows.

• Experience mentoring senior engineers or establishing engineering standards and best practices.

Benefits

• Competitive hourly compensation ranging from $100 to $200, depending on experience and location.

• Fully remote engagement, with worldwide opportunities subject to regional eligibility.

• Flexible workload of approximately 10–20 hours per week.

• Ongoing collaboration, with individual projects typically lasting from two weeks to several months.

• Opportunities to participate in additional projects based on evaluation performance.

• Flexible working hours.

• Streamlined selection process consisting of one take-home evaluation exercise and a recorded video walkthrough, with no interview required.

• Opportunity to contribute to the development and evaluation of cutting-edge AI coding tools.

• Contract-based engagement, with projects offered on an ongoing basis.

• Potential to start as soon as the evaluation exercise is successfully completed and a project position becomes available.