All articles

Article · 10-min video

Do tokens or lines of code measure developer productivity?

Oren Keinan, founder of Aidealy · ·

By pressing "Play on YouTube", you agree to load this video from YouTube, a Google service. Google then receives data such as your IP address and may store data in your browser.
Details in our Cookie Policy and Privacy Policy.

Also available with audio in Arabic, French, German, Hebrew, Italian, Japanese, Korean, and Portuguese: choose one in the player's settings.

Watch on YouTube (Opens in a new tab)More from Oren Keinan on YouTube (Opens in a new tab)Read Oren's take on LinkedIn (Opens in a new tab)Read the transcript

In short

Measuring developer productivity keeps failing when engineering metrics count one activity, such as lines of code, story points, tickets or AI tokens. Measure the output of the work, not the activity, and pair every measure with a second one that shows when people overdo the first.

Once people are ranked or rewarded by a single count, they raise the number instead of doing better work.
The same mistake showed up in a weekly form at Apple in 1982 and in an AI leaderboard at Amazon in 2026.
Andy Grove, then the president of Intel, described that fix in 1983.
The rest goes through those cases, the counts teams use today, and what a fair ranking of engineers needs.

What happened to Amazon's AI token leaderboard?

In May 2026, Amazon closed an internal leaderboard of AI use. Business Insider reported the reason: it "encouraged some staff to perform tasks that didn't necessarily solve problems, just so they could climb the ranks."

Business Insider described it as an employee-made leaderboard that "tracked AI token use." The leaderboard counted the AI tokens each person used.

In June 2026, 404 Media, an independent news site, reported that some employees said they had "deliberately cheated to climb the leaderboard's ranks."

Amazon isn't the first company to count the wrong thing. Some managers at Apple made the same mistake in 1982.

Lines of code at Apple in 1982

A rewrite that made the code faster and shorter counted as a negative number. The managers soon stopped asking its author for the count.

In early 1982, some managers at Apple asked every engineer to report, each week, how many lines of code they had written.

Bill Atkinson, the main designer of the user interface of Apple's Lisa computer, had just rewritten part of his code with a simpler method. The new version was almost 6 times faster and about 2,000 lines shorter. When he reached the lines-of-code field, he "thought about it for a second, and then wrote in the number: -2000." After a couple more weeks, the managers stopped asking him to complete the form.

Are story points, tickets, pull requests and AI tokens any better?

Story points, tickets, opened pull requests and tokens work for their own job, but stop measuring real work once teams or people are compared on them.

Many teams estimate their work in story points. Ron Jeffries, the first coach of Extreme Programming, a team style of building software, has written about where they came from.

A story point was a renamed "ideal day": the time a pair of programmers would need for a piece of work if nobody interrupted them. The points had one job, which was helping a team decide how much work to plan. In 2019, Jeffries wrote: "I may have invented story points, and if I did, I'm sorry now." He also wrote that comparing teams on their estimates, or on the points they finish, is harmful.

Tickets and pull requests have the same problem. In 2026, a paper from Microsoft's Engineering Thrive team described the pattern. Engineering Thrive is Microsoft's own system for measuring the conditions its developers work in. The paper says: "Organizations often track activities (lines of code, pull requests, tasks) and treat them as proxies for outcomes (value delivered, speed, quality)." Companies often count what people do and then treat the count as if it were the result.

Pull requests show the difference. A pull request someone opened is activity. A merged pull request whose code is still there months later is much closer to a result. Counting the activity means measuring the paperwork. Measure the work, not the paperwork. The mistake is measuring people through the ticket tracker.

The newest single count is AI tokens. Business Insider also reported that Amazon "does track AI token usage to measure costs." That's the right use for them. A token count is a fuel bill, not an achievement. Like the counts before it, it stopped measuring real work the moment people were ranked by it.

What did McKinsey propose in 2023, and why did engineers object?

McKinsey proposed new ways to measure developers, and Gergely Orosz and Kent Beck answered that nearly every measure it added counted effort or output.

In August 2023, the management consulting firm McKinsey published an article called "Yes, you can measure software developer productivity." McKinsey wrote that, compared with functions such as sales, "software development is perennially undermeasured," meaning it gets measured less than those other functions. As most companies become software companies to some degree, it argued, their leaders need to know they're using their most valuable people well. And McKinsey itself warned against overly simple measures such as lines of code.

McKinsey said in 2023 that its approach was in use at "nearly 20 tech, finance, and pharmaceutical companies." It kept measures from well-known research and added its own. One of them, which it called contribution analysis, assessed each person's contributions to the team's backlog. The backlog is the list of planned work, and the analysis started from data in tools such as Jira.

Later that month, Gergely Orosz and Kent Beck published a response. Orosz writes The Pragmatic Engineer, a newsletter for software engineers and engineering managers. Beck created Extreme Programming and test-driven development.

They split software work into 4 stages:

  1. Effort: the planning and the coding.
  2. Output: what that produces, such as the feature and its code.
  3. Outcome: how customers behave differently because of it.
  4. Impact: the value the company gets back, such as revenue.

They also wrote that "the act of measurement changes how developers work, as they try to 'game' the system." To game a system is to raise its numbers without doing better work.

Kent Beck gave an example from Facebook. A developer survey there was meant to show how things were going. "Then those scores became goals," and managers started bargaining with their engineers for better scores.

In Beck's account, one offer went like this: "Give me a 5 & I'll make sure you get an 'exceeds expectations'." That's a strong rating in a performance review. In the end, Beck and Orosz wrote, the company knew "even less about how things are going". Things were also getting worse "because people are now gaming the system."

McKinsey asked a fair question and answered it with the wrong measures. Leaders do need to know where engineering time and money go, but they can't learn it by counting how busy people look.

Can developer productivity be measured at all?

Yes: measure the work that shipped and lasted, not how busy people were, and check each measure against a second one that catches its side effects.

Kent Beck says no. In part 2 of the 2023 response to McKinsey, he wrote a section alone. His answer was: "Measure developer productivity? Not possible."

Behind that answer is Goodhart's law. Charles Goodhart, an adviser at the Bank of England, described it in 1975. Its best-known short form comes from Marilyn Strathern, a professor of social anthropology at Cambridge. Strathern wrote in 1997: "When a measure becomes a target, it ceases to be a good measure."

I think Beck and Goodhart are right about one kind of measure. Count one activity and reward it, and people will raise that number without doing better work. But measuring people changes how they work, and that isn't always bad. It's the reason to measure at all. Which way it changes them depends on what you count.

In 1983, Andy Grove published a book called High Output Management. Grove was the president of Intel at the time. On page 18, he wrote that "a genuinely effective indicator will cover the output of the work unit and not simply the activity involved." His example was a salesman: "you measure a salesman by the orders he gets (output), not by the calls he makes (activity)."

Grove also gave a second rule. Because measures direct what people do, he wrote, you should pair them, "so that together both effect and counter-effect are measured." His example came from software: track when each part of a program will be ready next to what it can do. Watching both should help a team avoid building "the perfect compiler that will never be ready," and also avoid "rushing to finish one that is inadequate."

DORA, short for DevOps Research and Assessment, is Google's research program on how teams deliver software, and it works the same way today. It puts how fast a team ships next to how often its changes fail. DORA's guide to these measures was last updated in January 2026. It says that "speed and stability are not tradeoffs" and that for most teams the two are correlated: they rise and fall together.

What does a fair ranking of engineers need?

A fair ranking counts the work that shipped and lasted, and checks it against quality. Then the way to climb it is to do better work. It follows these rules:

  1. Count the work that shipped and how long it lasted.
  2. Pair every speed measure with a quality measure.
  3. Remove points when the work breaks, and more when it breaks sooner.
  4. A number can show you where to look. It can't tell you why. So before you act on a number about a person, ask them.

With a ranking built like this, being measured still changes how people work, but in the direction you want. One way to climb is to write clean code that ships and doesn't need fixing soon after. Reviews count too: own the code you approve as well as the code you write, and don't keep your teammates waiting for a review.

I founded Aidealy, which makes software that measures the work engineering teams ship. Its ranking of engineers, the Aidealy Productivity Rank, follows these rules. Among other things, it counts each engineer's merged pull requests. Points come off when that code has to be fixed after the merge, and more when it breaks sooner. Reviewing other people's work earns credit, and keeping a teammate waiting for a review costs points. The lines an engineer writes earn no points, and neither do tokens: Aidealy counts tokens, but doesn't rank by them.

What to take to your next leadership meeting about developer productivity

Count one activity, and people will learn to cheat it. So rank on shipped work that lasted, and pair speed with quality. Remove points when the work breaks, and ask the person before you decide anything based on their number.

No ranking can tell you whether the work was worth doing. A rank shows how the work went. Whether it was worth doing is a judgment, and that judgment stays with you.

CaseWhat was countedWhat happened
Apple's Lisa team, 1982The lines of code each engineer wrote that weekA rewrite that made the code almost 6 times faster went on the form as -2000, and the managers stopped asking for that engineer's count
Story points, Ron Jeffries in 2019Story points, a renamed "ideal day" of workThe man who may have invented them wrote that, if he did, he's sorry now, and that comparing teams on them is harmful
Facebook's developer survey, told by Kent Beck in 2023A developer survey scoreManagers bargained with engineers for better scores, and the company knew even less than before
McKinsey's framework, 2023Its own added measures, such as each person's contribution to the team's backlogOrosz and Beck wrote that nearly every one counted effort or output
Amazon's AI leaderboard, 2026How much AI each employee used, in tokensIt encouraged some staff to do tasks that did not necessarily solve problems, and Amazon closed it in May 2026

Sources

Watch the video on YouTube: Why measuring developer productivity keeps failing

More from the blog

Transcript

Show the full transcript

0:00 Amazon's AI token leaderboard

In May 2026, Amazon shut down an internal leaderboard that tracked how many AI tokens its employees used. Some employees later told a news site called 404 Media that they'd cheated to climb it. If you lead engineers, you're probably already measuring them. Or you'll be soon. What you choose to count decides what happens next. Either your engineers start cheating the numbers, or they start doing better work.

By the end of this video, you'll know what to count so they do better work. And you'll know why the number should start a conversation, never become a verdict. Amazon isn't the first company to count the wrong thing. One famous case is 44 years old, and it starts with a weekly form at Apple. Later in this video, I'll also tell you what happened inside Facebook when one simple survey score started to matter.

0:50 Lines of code, 1982

-2000 lines of code. That's what Bill Atkinson wrote on a weekly form at Apple in 1982. Some managers on the Lisa team had asked each engineer to report the lines of code they wrote every week. The Lisa was the computer Apple was building at the time, and Atkinson was the main designer of its user interface. He'd just rewritten part of his own code.

The new version ran almost 6 times faster, and it was about 2,000 lines shorter. So when the form asked how many lines he'd written, he wrote -2000. After a couple more weeks, the managers stopped asking him to fill it in. To me, that one form shows the whole problem. A week that made the software faster and smaller made the number go negative.

1:33 Story points, tickets and AI tokens

Lines of code weren't the last single activity that teams counted. Many teams today estimate their work in story points. Ron Jeffries was the first coach of Extreme Programming, a team style of building software. He has written about where story points came from. A story point was simply a renamed ideal day, the time a pair of programmers would need without interruptions.

Points had one job, which was helping a team decide how much work to take on. In 2019, Jeffries wrote this. "I may have invented story points, and if I did, I'm sorry now." He wrote that comparing teams on their estimates or on the points they finish is harmful. Tickets have the same problem. In 2026, a paper from Microsoft's Engineering Thrive team described the pattern.

Engineering Thrive is Microsoft's own system for measuring the conditions its developers work in. Companies track activities like lines of code, pull requests and tasks. Then they treat those activities as if they were results. Notice that pull requests are on that list. A pull request someone opened is activity. A merged pull request whose code is still standing months later is much closer to a result.

I call counting the activity measuring the paperwork instead of the work. And to be clear, your ticket tracker isn't the problem. The mistake is measuring people through it. The newest single count is AI tokens, the small pieces of text that an AI tool takes in and gives back. That's what Amazon's leaderboard counted, according to the news site Business Insider.

It reported that the leaderboard encouraged some staff to do tasks that didn't necessarily solve problems, just so they could climb the ranks. It also reported that Amazon does track how many tokens it uses, but to measure its costs. Amazon was right to treat tokens as a cost. A token count is a fuel bill, not an achievement. And like the counts before it, it stopped measuring real work the moment people were ranked by it.

3:32 McKinsey 2023 and the response

In August 2023 the consulting firm McKinsey published an article called "Yes, you can measure software developer productivity." I'll give their case in its strongest form first, because it deserves that. McKinsey pointed out that other parts of a company get measured all the time, like sales. But software development gets measured far less. More and more companies are turning into software companies.

So their leaders need to know they're using their most valuable people well. And to be fair, McKinsey itself warned against simple measures like lines of code. McKinsey said nearly 20 companies were already using its approach. It kept some measures from well-known research and added its own. One was called contribution analysis. It looked at how much each person added to the team's backlog, the list of planned work.

It started from data in tools like Jira. About 2 weeks later, 2 respected engineers answered. Gergely Orosz writes The Pragmatic Engineer, a newsletter for software engineers that says it has more than a million readers. Kent Beck created Extreme Programming and test-driven development. Together they split software work into 4 stages, from effort and output to outcome and impact.

Effort is the planning and the coding. And output is what that produces, like the feature and its code. Then the feature reaches the customers. The outcome is how they behave differently because of the feature. And the impact is the value that comes back to the company, like revenue. And they wrote that nearly every measure McKinsey added counted effort or output.

Their central point was simple. Measuring developers changes how they work, because people start to cheat the system. And here's the Facebook story I promised. Kent Beck described what happened to a developer survey there. The survey was meant to show how things were going. Then the scores became goals, and managers started bargaining with their engineers for better scores.

In Beck's telling, one manager offered this deal. "Give me a 5, and I'll make sure you get an 'exceeds expectations'." That's a strong rating in a performance review. In the end the company knew even less about how things were going, as Beck and Orosz put it. And things were getting worse, because people were now cheating the survey.

Here's how I see that debate. McKinsey got the question right. Leaders do need to know where engineering time and money go. Actually, no. Let me put that more carefully. McKinsey asked a fair question and answered it with the wrong measures. You can't learn where the work went by counting how busy people look.

6:07 Goodhart's law and the fix from 1983

So can developer productivity be measured at all? Kent Beck says no. In the second part of their response to McKinsey, he wrote his own answer. "Measure developer productivity? Not possible." There's a law behind that answer, called Goodhart's law. An adviser at the Bank of England named Charles Goodhart described it in 1975. The best-known short version comes from a Cambridge University professor named Marilyn Strathern, in 1997.

"When a measure becomes a target, it ceases to be a good measure." I think Kent Beck and Goodhart are both right. But only about one kind of measure. Count one activity and reward it, and people will raise that number without doing better work. But measuring people changes how they work, and that's not always bad. It's the reason to measure at all.

What matters is what you count. One year after the Apple form, Andy Grove published a book called "High Output Management." That was 1983, and Grove was the president of Intel at the time. He wrote that a good measure covers the output of the work, not simply the activity involved. His example was a salesman. You measure him by the orders he gets, not by the calls he makes.

And because measures direct what people do, he wrote that you should pair them. You measure the effect, and you measure the counter-effect next to it. He even gave a software example. Track when each part of a program will be ready, and also what it can do. Watching both keeps you from building a perfect program that's never ready.

It also keeps you from rushing out one that isn't good enough. DORA stands for DevOps Research and Assessment, and it's Google's research program on how teams deliver software. It works the same way today. It puts how fast a team ships next to how often its changes fail. DORA publishes a guide to those measures, last updated in January 2026.

And that guide says speed and stability aren't a trade-off, because for most teams they rise and fall together.

8:02 Ranking done right

A ranking of engineers needs 4 things if you want it to make people better, not make them cheat. First, count the work that shipped and how long it lasted. Not how busy people looked. Second, pair every speed measure with a quality measure. Third, take points away when the work breaks. And more when it breaks sooner. And fourth, treat the number as the start of a conversation.

Never as a verdict. Measuring people still changes how they work. With a ranking built like this, it changes how they work in the direction you want. The way to climb is to do better work: write clean code that ships and doesn't need fixing soon after. Own the code you approve, not just the code you write. And don't keep your teammates waiting for a review.

[On screen: illustrative example, not real data]

I run a company called Aidealy, and we built the Aidealy Productivity Rank around these 4 rules. Among other things, it counts each engineer's merged pull requests. A change that leaves the code cleaner earns a little more, and one that leaves it messier earns a little less. It takes points away when that code has to be fixed after the merge, and more when it breaks sooner.

It gives credit for reviewing other people's work, and it counts how quickly someone reviews. The lines an engineer writes earn no points, and neither do tokens. We count tokens. We just don't rank by them. And the rank is where a conversation starts, never a verdict.

9:25 What to take to your next meeting

Now, there's something no rank can tell you. It shows how the work went, but it can't tell you whether the work was worth doing. That's a judgment, and it stays with you. So here's what to take to your next leadership meeting. Count one activity, and sooner or later people will cheat it. Rank on shipped work that lasted, pair speed with quality.

Take points away when the work breaks, and let the number start a conversation. The next video to watch is called Developer productivity metrics that get people fired. It goes through the measures that get cheated most often, one by one. Subscribe if you want it. The sources I used are all linked below, so read them yourself before you take my word for any of this.

See what your R&D is really doing.

Tell us what you're trying to figure out, and we'll get you set up to answer it on your own R&D: your engineers, your AI agents, and the code they ship together.

Talk to us