What You'll Be Doing
Author complex, multi-step benchmark tasks grounded in real-world workspace files including technical specs, architecture docs, API references, and codebases
Design tasks that evaluate the ability of language models and systems to reason over technical documentation and follow precise instructions
Pair each task with a clearly defined ground truth output and an objective evaluation rubric
Incorporate web search and code execution into task design to reflect professional, real-world workflows
Contribute to the development of benchmark datasets used to assess LLMs on professional document understanding and instruction following within the Technology domain

What We're Looking For
Strong experience in software engineering, data science, or analytics in a hands-on professional capacity
Strong experience in authoring or evaluating technical documentation, including architecture docs, API references, or codebases
Strong experience in designing structured, multi-step tasks with measurable outputs and evaluation criteria
Strong experience working independently in a remote, self-directed contract environment

Data Scientist | Remote

Similar Data Science Jobs