About This Opportunity

The field of agentic AI — where systems autonomously plan, reason across multiple steps, delegate sub-tasks, and coordinate resources to achieve complex goals — is one of the most rapidly advancing areas in global technology. Building robust evaluation frameworks and benchmark tasks for these systems is foundational work that will shape how the entire industry assesses AI capability, safety, and reliability for years to come.

This Operations Engineer role sits at the frontier of applied AI research. You will design and build the operational scenarios — logistics problems, incident response simulations, capacity planning challenges, project management tasks — that multi-agent AI systems are evaluated against. This requires both strong Python engineering skills and genuine domain expertise in operational problem-solving. You need to understand how real operational problems are structured, what makes them hard, and how to encode that difficulty into a form that AI evaluation can measure precisely.

Opportunities to work directly on multi-agent AI evaluation at this level are rare and represent significant career capital in one of technology's most consequential emerging fields.

Role Responsibilities

Design and develop multi-agent benchmark tasks involving planning, scheduling, and resource allocationCreate real-world operational scenarios (logistics, project management, incident response, capacity planning)

Build constraint-rich problem statements with multiple dependencies and variables

Develop Python-based scripts to evaluate feasibility, completeness, and optimality

Break down complex problems into structured sub-tasks for multi-agent systems

Model scenarios with timelines, dependencies, and resource constraints

Collaborate with teams to improve task quality, coverage, and evaluation rigor

Applying for This Role

• Portfolio over credentials: A GitHub repository with well-designed benchmark tasks, constraint modelling scripts, or AI evaluation frameworks will carry more weight than a CV alone. Build it before you apply.
• Domain complexity is what matters: The role requires operational domain knowledge — demonstrate understanding of scheduling algorithms, resource allocation problems, dependency networks, and constraint satisfaction. Reference specific operational domains where you have experience.
• Python engineering depth is required: This is not a research role with occasional scripting. You need to build robust, well-documented evaluation infrastructure. Demonstrate software engineering discipline in your application.
• Collaborative mindset: The role explicitly involves working with teams to improve task quality and evaluation rigour — show that you engage constructively with technical peers and incorporate feedback.

Requirements

5+ years of experience in operations, project management, logistics, or supply chain

Strong understanding of constraints, dependencies, and scheduling logic

Proficiency in Python for validation and verification scripting

Strong structured problem-solving and decomposition skills

Ability to model real-world operational scenarios

Clear technical communication and documentation skills

Ability to work in a fast-paced environment and meet deadlines

Preferred Qualifications

Experience with optimization techniques (linear programming, constraint satisfaction, scheduling algorithms)

Background in operations research

Experience with simulation or modeling tools

Familiarity with AI planning systems or automated reasoning

Experience with AI benchmarks (e.g., SWE-bench, Terminal-bench)

Hands-on experience with Docker

Application Process

Apply via Easy Apply / shared link and complete the Interest Check Form (ICF)

Complete the take-home assessment (post-shortlisting)

Shortlisted candidates will be reviewed further

The team will connect with next steps

Compensation: $15/hour