Software Engineers

Software Engineers: Coding Tasks for AI Evaluation [$40 per hour]

A paid study on the effectiveness of coding tasks used to evaluate AI agents

How It Works

• The candidate will review a series of proposed coding tasks and assess their suitability for testing AI agents and verify the logic of the evaluation harnesses and provide feedback on their realistic application.

• The candidate will walk through the environments and point out potential flaws or improvements. The session involves a mix of code review, technical discussion, and direct feedback on the task structures.

Who This Is For

• Software engineers based in Brazil and Argentina with strong technical backgrounds

• Candidates having direct experience building, reviewing, or testing evaluation harnesses and programming tasks

• Backend developers, full-stack engineers, and quality assurance automation specialists

Role Responsibilities

• Review realistic programming tasks designed for AI evaluation

• Assess the accuracy and structure of various evaluation harnesses

• Walk us through your thought process while checking code validity

• Provide actionable feedback on how to improve task complexity and realism

Good Candidature

• Professional experience as a software engineer or developer

• Located in Brazil or Argentina

• Familiarity with building or verifying coding tasks and test environments

• Comfortable discussing technical architectures and evaluation frameworks