ML Inference Task Designer - Remote Worldwide Contract — Columbus

Overview

We are building a benchmark that tests how well AI agents handle real expert work in a terminal. We are hiring ML inference and serving experts to turn real, paid tasks from their jobs into expert-level Terminal-Bench challenges. Each task includes the environment, a working solution, and an automated check that proves the result is correct.

⚠️ THIS IS A TASK-BASED COMPENSATION MODEL. PLEASE CHECK THE APPLICATION QUESTIONS FOR MORE DETAILS.

Responsibilities

• Design expert-level Terminal-Bench tasks that turn real, paid ML inference work from your own experience into challenges an AI agent must solve end to end.
• Define the files, systems, and environments each task runs on — such as model serving with streaming — so the challenge is realistic, reproducible, and command-line solvable.
• Write a working reference solution for every task and verify it runs end to end, sized for about 60–75 minutes of expert effort.
• Build automated checks that objectively prove a result is correct, including cases where ML outputs go wrong silently, such as numerical drift or metric bugs.
• Package tasks with Docker and Git so they are reproducible, and document the systems, tools, data, and rules involved so reviewers can validate difficulty.
• Iterate on tasks based on reviewer feedback and agent test runs, adjusting difficulty and fixing ambiguities so each task stays expert-level, fair, and objectively gradable.

Required Skills & Experience

• 3+ years doing ML inference or serving work hands-on (not only studying, selling, or managing it), or a PhD plus active research in the subfield.
• Did this work in the last 3 years and can name the systems, tools, data, and rules used.
• Trained, served, or evaluated models at real scale (multi-GPU or production serving), not only notebooks or API calls.
• Hands-on with PyTorch or JAX internals and serving stacks such as vLLM, SGLang, or TensorRT-LLM.
• Can explain how ML results go wrong without errors (numerical drift, metric bugs) and how to catch them.
• Technical fluency in Python and the command line, and with files, CSV/JSON, APIs, or databases.
• Availability to start by early November 2026 and complete tasks of about 60–75 minutes each.

Nice-to-Have

• AI lab or LLM infrastructure experience.
• Contributions to PyTorch, vLLM, or SGLang.
• GPU profiling experience.
• Docker and Git proficiency.
• Experience writing automated tests.

Process

• Task vetting instead of an interview: complete a vetting task so the client can assess your task design.
• Paid per task; each task takes about 60–75 minutes.
• Work is delivered task by task, with reviewer feedback on submissions.
• Expected start in early November 2026.

To apply: https://weworkremotely.com/remote-jobs/toptal-ml-inference-engineer-terminal-bench-task-designer