All articles

Article · 10-min video

Which developer productivity metrics should you never use?

Oren Keinan, founder of Aidealy · ·

By pressing "Play on YouTube", you agree to load this video from YouTube, a Google service. Google then receives data such as your IP address and may store data in your browser.
Details in our Cookie Policy and Privacy Policy.

Also available with audio in Arabic, French, German, Hebrew, Italian, Japanese, Korean, and Portuguese: choose one in the player's settings.

Watch on YouTube (Opens in a new tab)More from Oren Keinan on YouTube (Opens in a new tab)Read Oren's take on LinkedIn (Opens in a new tab)Read the transcript

In short

The developer productivity metrics that get cheated most often, and that now reach performance reviews, are the ones that count a single activity. Don't read any of them alone: next to each count, put the measure that drops when the work is bad.

These counts now reach decisions about people: performance reviews in some teams at Amazon, and jobs at Coinbase.
Every case here comes from a named, dated source.
Each count then gets its own section, from lines of code to AI tokens, with the number to put next to it, and the article ends with what to use instead.

Do productivity metrics really decide who stays?

Yes: Amazon employees say some teams count AI use in performance reviews, and Coinbase fired engineers slow to set up its AI tools.

In July 2026, 3 current and former Amazon employees spoke to the business news channel CNBC. They said some managers there track their teams' AI activity on internal dashboards, "with certain teams factoring usage into performance reviews".

In August 2025, the tech news site TechCrunch reported what Brian Armstrong, Coinbase's chief executive, had told a podcast. Armstrong fired engineers who didn't set up the company's new AI coding tools by the end of the week. Some of them had a good reason, he said. The ones without a good reason got fired.

You probably measure your engineers already, and the numbers you pick may soon decide who stays.

Should you measure engineers at all?

Yes, but a number about a person can show you where to look, and it cannot tell you why.

So before you act on a number about a person, ask them. The real question is what you count, and what you let the number decide once you have it.

What should you put next to lines of code?

Put next to a line count how much of that code is still there a few months later.

Lines of code is the oldest measure on the list. The idea was that more lines meant more work done. It never did. Now that AI writes much of the code, a line count says even less.

JetBrains, a maker of programming tools, surveyed more than 15,000 developers in mid-2026. On average, developers in the 2026 JetBrains survey said AI agents fully write about 47% of their code.

Measured data shows the same thing. Researchers at the Complexity Science Hub in Vienna checked public code on GitHub and reported in the journal Science in 2026. In the US, about 5% of Python functions were written with substantial AI help in 2022, and 29% by the end of 2024.

Anthropic makes the AI coding tool Claude Code. In June 2026, it wrote about its own numbers: "Lines of code is an imperfect measure, as it measures quantity over quality."

Code that is still there months later is the part that lasted. That one number tells you whether the lines were work, or just typing. By hand, this means following each merged change through the later history of its files, which takes real time on a large codebase.

Should you count commits?

Do not count commits at all: count the changes that got merged.

Commit counts are easy to fake. A free script on GitHub called github-activity-generator had more than 4,000 stars in September 2026. It fills a whole year of your commit history, and its own description says it does this "instantly". Its page also says it is "for educational purposes" and "should not be used to misrepresent professional contributions or coding activity".

A commit count tells you how busy someone's history looks. It can't tell you whether anything shipped. An engineer who spends a whole week on one hard fix may show a single commit for it.

In 2021, Nicole Forsgren and colleagues published SPACE, a well-known framework for measuring developer productivity. They wrote that activity counts such as pull requests and commits "should never be used in isolation either to reward or to penalize developers".

Can you judge engineers by story points and closed tickets?

No: keep story points and ticket counts inside the team, for planning the next few weeks.

Many teams estimate their work in story points. A story point is a guess about how big a task is, made before anyone writes the code. So counting finished points means counting guesses. Closed tickets have a different weakness: split one task into several tickets, and the count rises with no extra work done.

In August 2023, Gergely Orosz and Kent Beck wrote about the leaders who use numbers like these. Orosz writes The Pragmatic Engineer, a newsletter for software engineers. Beck created Extreme Programming, a team style of building software.

The first case they described was a chief technology officer who wants to identify which engineers to fire. One of the ways they listed was to "Decide on a few easy-to-measure metrics like the number of pull requests, number of tickets closed". They also wrote that "decisions are often made based on metrics known to be incorrect."

You can keep your ticket tracker. Just don't measure people through it.

Are pull request targets a good idea?

No: a target on the number of pull requests gets met by splitting the work. So next to that number, put how many pull requests needed a fix soon after they were merged. Also put how long each one waited for review.

Pull requests need more care than anything else on this list. Uber built a dashboard that counted diffs per engineer. A diff is a code change sent for review, much like a pull request. When Gergely Orosz asked engineers there how the dashboard changed things, he heard that people started to create "far, far more diffs". That raised the load on the systems that build and test every change, and the cost of running them.

Some managers and directors then set targets for the number of diffs per month. Those targets, Orosz wrote, "are met with ease: as you just create smaller diffs".

Small changes are a good thing. DORA, which stands for DevOps Research and Assessment, is Google's research program on how teams deliver software. Its guide "Working in small batches" was last updated in December 2025. It says to "Avoid the temptation to generate massive pull requests", because "large changes are difficult to review, test, and integrate safely."

So the trouble at Uber was the target on the count. A big change broken into small ones that each pass review and stay in production is good engineering. The fixes after merge and the review waits are what tell you whether that is what happened. The review waits are in each pull request's history. The fixes take more work: for every later fix, someone has to find the pull request it corrects.

Should you set targets for AI use and token counts?

Counting AI use can help people try the tools, but once use becomes a target, people raise the count without doing better work.

The newest measures count AI itself. Every token an AI tool uses costs money, and some leaders now treat token spending as a sign of good work. Jensen Huang is the chief executive of Nvidia. On the All-In Podcast in March 2026, Huang said: "If that $500,000 engineer did not consume at least $250,000 worth of tokens, I am going to be deeply alarmed."

His argument makes sense. An engineer with good AI tools can get more done, and the company pays for those tools anyway. DORA made a similar case in June 2026. It wrote that a token leaderboard "does provide a relatively low-stakes way to encourage exploration in 'AI-hesitant' developers", and that "teams cannot achieve impact without baseline adoption."

That part is fair. The trouble starts when use becomes the target. In April 2026, the newsletter The Pragmatic Engineer reported on Salesforce. The company expects its engineers to spend at least $170 a month on AI tokens, and flags anyone who spends less.

Also in April 2026, the tech news site The Information reported on a leaderboard at Meta. It ranked more than 85,000 employees by the tokens they used. In 30 days they used about 60 trillion tokens, The Information reported, and some left AI agents running for hours just to raise their numbers. Meta removed the leaderboard.

In July 2026, the news agency Reuters reported on a lawsuit by 26 former Meta employees. The lawsuit claims that Meta relied on factors such as productivity and AI token use when it chose whom to lay off. It also claims that this hurt people who had missed work for medical reasons. Meta denies the claims. A Meta spokesperson told Reuters: "Workforce management and organizational decisions were and are made by people, not AI."

Amazon confirmed to Business Insider in May 2026 that it had closed its own token leaderboard. Dave Treadwell, one of its senior vice presidents, told staff: "Please don't use AI just for the sake of using AI."

What should you put next to a token count?

What you spend on AI for each change that got merged: tokens are a cost, and you judge a cost by what it bought.

I want engineers to use AI, and a lot of it where it helps. My disagreement with Huang starts there. A token bill tells you what the work cost, and only that: what the work delivered is a different number.

DORA's June 2026 article also says what to put next to a token count. It suggests looking at token use "by team, by application, or by use case". It also suggests the "cost per accepted change", meaning what you spend on AI for each change that got merged. Your AI bill does not say which change the tokens went into, so matching the spend to each merged change takes manual work.

Why is a count read alone misleading now?

Because output can rise while the work gets worse, and a count read alone shows only the output.

More AI use has not produced the same amount of shipped software. In 2026, the National Bureau of Economic Research (NBER) published a study of more than 500,000 GitHub developers. In the 2026 NBER study, as developers moved from autocomplete to autonomous AI agents, their commits rose by 240% in total. In the same study, their actual releases rose by only 30%.

In 2026, researchers at Carnegie Mellon University published a study of 806 open-source projects that started to use the AI coding tool Cursor. They compared them with 1,380 similar projects that did not. In the first month, the projects added 3 to 5 times more lines of code, and after 2 months that gain was gone. But warnings from code-checking tools rose 30%, code complexity rose 41%, and both stayed high. Count only the first number, and you'd call that a success.

A 2025 study from Tilburg University found who carries that cost. When GitHub Copilot was first tested in open-source projects, the more experienced developers reviewed 6.5% more code, and their own code output dropped 19%. The researchers warned that "productivity gains of AI may mask the growing burden of maintenance on a shrinking pool of experts". So my warning for anyone planning layoffs based on output counts alone: you may lose the experts who carry the maintenance work.

A pair shows you the cost next to the output, and the people who carry it. It makes cheating visible. It doesn't make cheating impossible.

How can you rank engineers without activity counts?

A ranking can exclude the counts that people can raise without better work, and give credit for merged work that did not need fixing.

I founded Aidealy, which makes software that measures the work engineering teams ship. Its ranking of engineers, the Aidealy Productivity Rank, works that way. The lines an engineer writes, commits, closed tickets and tokens earn no points in it. Among other things, a merged pull request earns credit, and more when it increases code quality or adds tests. Points come off when its code has to be fixed, and more when it breaks sooner. Reviewing a teammate's work earns credit, and keeping them waiting costs points. Working across more languages and repositories earns more too, and so does working with several AI chats at once, as one measure among many.

When someone's rank drops, it shows you where to look. Then you ask the person.

Whether a rank like this gets people fired is a choice leaders make.

Which engineering metrics should you use instead of activity counts?

Look at the code that got merged and lasted, and at how people work; keep AI spend as a cost you track.

The code:

  • Changes that got merged
  • Code still there months later
  • Changes that increase code quality
  • Changes that add tests

How people work:

  • Reviewing teammates' work
  • How long reviews wait
  • Work across languages and repositories
  • Several AI chats at once

Track it, don't rank by it:

  • AI spend per merged change

AI spend per merged change is a cost: track it, but do not rank anyone by it. For every other number you report, one test decides how to use it. If a number can rise while the work gets worse, do not use it alone.

Sources

Watch the video on YouTube

More from the blog

Transcript

Show the full transcript

0:00 When a count decides who stays

In July 2026, a few current and former Amazon employees spoke to the business news channel CNBC. They said some teams there now count AI use in performance reviews. If you lead engineers, you're probably measuring them already. And the numbers you pick may soon decide who stays. By the end of this video, you'll know which measures get cheated most often.

And you'll know what to put next to each one, so the cheating shows. I know the title sounds dramatic. Every case in this video is real, dated and linked below. At Amazon, it's performance reviews. At Coinbase, it was people's jobs. About a year earlier, Coinbase's chief executive fired engineers who hadn't set up their new AI coding tools within a week.

He told the story himself on a podcast. Some of those engineers had a good reason. The ones who didn't got fired.

0:54 A number shows where to look

Now, I think you should measure your engineers. But a number can show you where to look. It can't tell you why. So before you act on a number about a person, ask them. The real question is what you count, and what you let the number decide once you have it.

1:10 Lines of code: pair them with what survived

The oldest measure on the list is lines of code. The idea was that more lines meant more work done. It never did. And AI has made it worse. According to DX's report from mid-2026, AI now writes more than half of all the code in the companies it studies. DX is a company that measures how software teams work.

Faros AI is a company that analyzes engineering data. In its 2026 report, it said this about counting how much work comes out. "Throughput measures what was shipped, not what survived." So here's what I'd put next to a line count: how much of that code is still there a few months later. That one number tells you if the lines were work, or just typing.

1:54 Commits: don't count them at all

[On screen: illustrative example, not real data]

There's a free script on GitHub with more than 4,000 stars that fills a whole year of your commit history. Its own description says it does it instantly. To be fair, its authors say it's meant for learning and shouldn't be used to make anyone's work look bigger. But it shows how little a commit count proves. A commit count tells you how busy someone's history looks.

It can't tell you whether anything shipped. Take an engineer who spent a whole week on one hard fix. That's a single commit. In 2021, a group of researchers published a well-known framework for measuring developer productivity called SPACE. They wrote that counts like commits and pull requests "should never be used in isolation either to reward or to penalize developers."

So my advice for commits is short. Don't count them at all. Count the changes that got merged.

2:46 Story points and tickets: keep them in the team

Many teams estimate their work in story points. A story point is a guess about how big a task is, made before anyone writes the code. So when you count finished points, you're counting guesses. Closed tickets have a different weakness. Split one task into 5 tickets, and the count goes up 5 times. In 2023, Gergely Orosz and Kent Beck wrote about the leaders who use numbers like these.

Orosz writes The Pragmatic Engineer, a newsletter for software engineers. Beck created Extreme Programming, a team style of building software. They described a chief technology officer who wants to decide which engineers to fire. One way they listed was to pick a few easy-to-measure numbers, like pull requests and closed tickets. And they wrote this. "Decisions are often made based on metrics known to be incorrect."

So keep story points and ticket counts inside the team, for planning the next few weeks. To be clear, you can keep your ticket tracker. Just don't measure people through it.

3:50 Pull request targets: pair them with fixes and waits

Pull requests are next, and they need more care than anything else on this list. Uber built a dashboard that counted diffs per engineer. A diff is a code change sent for review, much like a pull request. Gergely Orosz asked engineers there how the dashboard changed things. People started creating far more diffs. That raised the load on the systems that build and test every change, and the cost of running them.

Some managers and directors then set targets for the number of diffs per month. People met them easily. They just made each diff smaller. And small changes are a good thing. DORA stands for DevOps Research and Assessment, and it's Google's research program on how teams deliver software. It recommends small changes. Its guide on working in small batches was last updated in December 2025.

It says to "avoid the temptation to generate massive pull requests." Small changes are easier to review and to test. And they're safer to merge. So the trouble at Uber was the target on the count. Break a big change into small ones that each pass review and stay in production, and that's good engineering. What matters is what you put next to the count.

How many of those pull requests needed a fix soon after they were merged. And how long each one waited for someone to review it.

5:09 AI token counts and tokenmaxxing: adoption is not impact

The newest measures count AI itself. Every token an AI tool uses costs money. Some leaders now treat token spending as a sign of good work. Jensen Huang is the chief executive of Nvidia. In March 2026, he said this on a podcast. "If that $500,000 engineer did not consume at least $250,000 worth of tokens, I am going to be deeply alarmed."

His argument makes sense. An engineer with good AI tools can get more done, and the company is paying for those tools anyway. DORA makes a similar case for counting use. In June 2026, it wrote that a token leaderboard can get hesitant developers to try AI. And a team can't get results from AI if nobody uses it. I agree with that part.

Now look at what happens when use becomes the target. In April 2026, the newsletter The Pragmatic Engineer reported on Salesforce. It said the company expects its engineers to spend at least $170 a month on AI tokens. Anyone who spends less gets flagged. That same month, the tech news site The Information reported on a leaderboard at Meta. The leaderboard ranked more than 85,000 employees by the tokens they used.

In 30 days, they used about 60 trillion. Some left AI agents running for hours just to raise their numbers. Meta took the leaderboard down. Then in July 2026, a group of 26 former Meta employees sued the company. The news agency Reuters reported on their lawsuit. It says Meta relied on factors like productivity and AI token use when it chose whom to lay off.

And it says that hurt people who'd missed work for medical reasons. Meta denies the claims. A Meta spokesperson told Reuters this. "Workforce management and organizational decisions were and are made by people, not AI." According to DX, here's what all that use delivered. It sampled data from 400 companies, from late 2024 to early 2026. AI use went up 65% on average.

The number of pull requests they merged went up by less than 8%. Amazon shut down its own token leaderboard in May 2026. Dave Treadwell is one of its senior vice presidents, and he told his staff this. "Please don't use AI just for the sake of using AI."

7:35 Tokens: judge a cost by what it bought

So here's where I disagree with Jensen Huang. I want engineers to use AI, and to use a lot of it. Well, a lot of it where it helps. But a token bill tells you what the work cost, and only that. What the work delivered is a different number. DORA's own article from June 2026 also says what to put next to a token count.

It suggests looking at token use by team, by application or by use case. And it suggests the cost of each accepted change, meaning what you spend on AI for each change that got merged. That's the pair. Tokens are a cost, and you judge a cost by what it bought.

8:13 Why the pair matters

Here's why the pair matters so much right now. Faros AI followed about 22,000 developers for 2 years. As their AI use grew, the pull requests each developer got merged went up about 16%. Bugs per developer went up 54%. Count only the first number, and you'd count that as a success. And Faros wrote a warning for anyone planning layoffs based on that first number.

"The engineers being considered for cuts are in many cases the ones absorbing the quality gap AI is creating." In plain words, those are the people who review and fix what the AI wrote. That's what a pair is for. It shows you the cost next to the output, and the people who carry it. And to be honest, a pair makes cheating show.

It doesn't make it impossible.

9:01 A ranking built on pairs

[On screen: illustrative example, not real data]

I run a company called Aidealy. We built the Aidealy Productivity Rank on pairs like these. The lines an engineer writes earn no points. Neither do commits, closed tickets or tokens. We count tokens. We just don't rank by them. Among other things, a merged pull request earns credit, and more when it increases code quality or adds tests. Points come off when its code has to be fixed, and more when it breaks sooner.

Reviewing a teammate's work earns credit, and keeping them waiting costs points. Working across more languages and repositories earns more too. So does working with several AI chats at once, one measure among many. When someone's rank drops, it shows you where to look. You can ask Aidealy why it moved, and see the work behind the answer. Then you ask the person.

So would a rank like this get people fired? That's a choice leaders make. We built Aidealy so you can investigate the data behind every number, and understand where your team needs help.

10:01 What to look at instead

Here's what to look at instead, all on one screen, so you can take it to your next leadership meeting. The last column is a cost. Track it, but don't rank anyone by it. If a number can go up while the work gets worse, don't use it alone. The sources I used are all linked below, so read them yourself before you take my word for any of this.

If you want to keep going, pick one of the videos on the screen now, or subscribe for the next one.

See what your R&D is really doing.

Tell us what you're trying to figure out, and we'll get you set up to answer it on your own R&D: your engineers, your AI agents, and the code they ship together.

Talk to us