How Agent-Driven Development Transformed Our Analysis Workflow at Copilot Applied Science

Introduction

As an AI researcher on the Copilot Applied Science team, I recently discovered a way to automate not just repetitive tasks but the very intellectual toil that often consumes our days. Now I find myself maintaining a tool that enables my peers to do the same—a shift that may have redefined my role entirely. This article shares the journey, the lessons learned, and how GitHub Copilot made it all possible.

How Agent-Driven Development Transformed Our Analysis Workflow at Copilot Applied Science — Source: github.blog

The Challenge of Analyzing Agent Trajectories

A significant part of my work revolves around evaluating coding agent performance against benchmarks such as TerminalBench2 and SWEBench-Pro. These benchmarks generate what we call trajectories—detailed records of the agent’s thought processes and actions while solving tasks. Each trajectory is a JSON file with hundreds of lines of code. Multiply that by dozens of tasks per benchmark set, and again by the many runs requiring analysis each day, and you’re looking at hundreds of thousands of lines of code to pore over.

Doing that manually was impossible. So I turned to GitHub Copilot to help surface patterns in the data. Copilot could reduce the lines I needed to read from hundreds of thousands to a few hundred, but I still found myself repeating the same loop: use Copilot to find patterns, then investigate them manually. The engineer in me rebelled: “I want to automate this.”

Automating Intellectual Toil: The Birth of eval-agents

That vision gave life to a project I called eval-agents. Its purpose: to automate the intellectual grunt work of analyzing agent performance, freeing up time for deeper insights and creativity. But automation alone wasn’t enough. I wanted the tool to be a platform for collaboration.

I set three core goals for the project:

Make these agents easy to share and use so the whole team could benefit.
Make it easy to author new agents so anyone could extend the system.
Make coding agents the primary vehicle for contributions, turning code into a collaborative artifact.

The first two goals align with GitHub’s DNA—values I’ve internalized from my time as a maintainer of the GitHub CLI. The third goal was a natural extension of agent-driven development.

How GitHub Copilot Enabled This Transformation

Creating eval-agents wouldn’t have been possible without GitHub Copilot. I used Copilot not just to write code, but to design a system that could reason about trajectories. Copilot helped me prototype agent logic, generate test datasets, and iterate rapidly. The incredibly fast development loop it unlocked meant I could go from idea to working agent in hours instead of days.

Moreover, Copilot’s ability to understand context allowed me to embed domain knowledge directly into the agents. For example, I could describe the structure of a trajectory file, and Copilot would generate parsing code that accounted for edge cases I hadn’t considered. This collaborative coding transformed what was once a solo effort into a partnership.

Design Principles for Agent-Driven Development

The eval-agents project taught me several principles for effective agent-driven development:

Start with the human workflow. Understand what your team does repeatedly, then automate those steps. In my case, the loop of querying, examining, and summarizing trajectories was ripe for automation.
Design for sharing. An agent that only works on your machine is a toy. Build it so others can run it with minimal setup—use environment variables, configuration files, and clear documentation.
Treat agents as evolving artifacts. Your first agent will be imperfect. Encourage peers to fork, modify, and extend it. The goal is a ecosystem, not a monolith.
Leverage Copilot as a pairing partner. Use Copilot to write boilerplate, suggest improvements, and even explain complex logic. This speeds up development and reduces errors.

The Impact on Team Collaboration

Once eval-agents was shared with the Copilot Applied Science team, something remarkable happened. Colleagues who had never built an agent before started authoring their own. They used the existing agents as templates, customized them for new benchmarks, and contributed back improvements. The team’s analysis throughput increased dramatically, and the quality of insights improved because members could focus on interpretation rather than data wrangling.

One unexpected benefit was the reduction in context switching. Previously, analyzing a new benchmark run would scatter attention across multiple tools and files. Now, a single agent handles the full pipeline, from data ingestion to summary generation.

Lessons Learned: Collaborating with Copilot

Throughout this journey, I discovered several best practices for using GitHub Copilot effectively in agent development:

Be explicit in comments. Copilot uses comments to understand intent. Writing “Parse the trajectory file and extract steps where the agent used the terminal” leads to better code than a vague prompt.
Iterate with Copilot. Don’t accept the first suggestion. If Copilot generates a function that’s close but not perfect, refine the comment or ask for alternatives by clearing the suggestion and re-prompting.
Use Copilot for testing. Generating test cases for agents can be tedious. Ask Copilot to create edge cases, then run them against your agent to uncover flaws.
Document as you go. Copilot can also generate documentation from code. Use this to keep your READMEs and inline comments up to date without extra effort.

Conclusion: A New Role Emerges

By automating my intellectual toil, I may have automated myself into a different job—one where I maintain and grow a platform that empowers others to automate their own analysis. It’s a role I didn’t anticipate, but one that feels deeply satisfying. Agent-driven development, powered by tools like GitHub Copilot, is not about replacing humans; it’s about freeing them to do the creative, strategic work that machines cannot. And that, I believe, is the future of software engineering and AI research combined.