BT
x Your opinion matters! Please fill in the InfoQ Survey about your reading habits!

Opinions: Measuring Programmers' Productivity

by Sadek Drobi on Oct 05, 2008 |

In the field of software development, just as in any other area, managers need to appreciate performance of their programmers and the progress of their projects. However, defining appropriate metrics to this end appears to be a tricky task.

Measuring source lines of code (SLOC) is one of commonly used approaches that presents however a number of important limitations highlighted recently by Shahar Yair and Steve McConnell. First of all, the amount of lines of code does not allow effectively measuring the progress of a project because it focuses on activity rather than results. LOC do not have any value as such: the value of the final product depends on its performance and quality, not on the amount of code. Hence, focusing on the latter is actually a very limited approach to productivity.

SLOC doesn’t tell anything about the complexity of the issue at hand or about the quality of the final product in terms of maintainability, flexibility, extensibility, etc. With regard to quality, it can actually be negatively correlated. Refactoring as well as some design patterns result in reducing LOC while improving code’s quality. Larger code base may mean more noise, more unnecessary complexity, less readability and flexibility.

What is particularly risky about adopting such a reductive view of programmers’ performance is that it creates wrong incentives. Rather than optimizing their work in terms of final product, developers may be encouraged to favor the quantity of code to the detriment of quality and even intentionally write more verbose code. “What gets measured gets done”, recalls Steve McConnell.

He points out that some of these issues can be solved by using function points as measuring metric. The program size is then determined by the number of inputs, outputs, queries and files. Nevertheless, this approach also has its downsides. McConnell mentions some practical issues like the necessity of having a certified function point counter and the difficulties with mapping each function point back to individual programmers. A certified function point specialist Daniel Yokomizo highlights in his comment other limitations of such approach: lack of tools to measure function points’ complexity and to take into consideration things such as code sharing, frameworks, libraries, etc., that affect the time needed to create a feature.

Even though many commentators involved in the discussion about measuring approaches agree on their limitations, they do not necessarily dismiss the need for measuring programmers’ performance. Many insist on the fact that SLOC, for instance, can be used as a baseline for a more complex analysis combining different factors. This goes along the lines of four principles outlined by McConnell that should guide any analysis of programmers’ productivity:

1. Don’t expect any single dimensional measure of productivity to give you a very good picture of individual productivity […]

2. Don’t expect any measures […] to support fine-grain discrimination in productivity among individuals. [They] give you questions to ask but they don’t give you the answers […]

3. Remember that trends are usually more important than single-point measures […]

4. Ask why you need to measure individual productivity at all […]

In which context measuring programmers’ productivity is actually meaningful? What criteria can be used for it? How can they be combined? Many questions are still open for discussion and if your experience has brought you some answers, don’t hesitate to share.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

A minor nitpick by Daniel Yokomizo

Nice article, but I have a minor nitpick. I was a certified function point specialist by IFPUG, I'm not anymore since 2005 (IIRC, I don't remember the exact date), as I said in my comment on Steve McConnell's blog. OTOH I was still working regularly as a specialist after that. IFPUG has a time limited certification, when came the time to certify again I didn't felt the money was worth it (as I didn't want to work primarily with function points anymore).

Why do I care? by Bruce Rennie

Why do I care about programmer productivity? I'm not selling lines of code, am I? What, exactly, is one programmer producing?


Where does the desire to measure developer productivity come from? Do they measure the productivity of doctors? Engineers? Plumbers? And those don't tend to work in teams.

LOC by Stefan Wagner

If I would have the choice to choose from 2 projects, which do the same thing, one with x lines of code, and one with 2*x lines of code, I would - of course - choose the smaller one (assuming no other information). Code-reuse means less code! More code encourages copy and paste.

If you bundle some subprime measurements together, do you get a triple-A measurement, or do you get junk?

Maybe I would start writing code-generators, if LOCs would be measured, but most probably I would leave.

Re: Why do I care? by Sadek Drobi

@Bruce: It can be quite interesting to trace where this productivity measuring culture comes from...

Re: Why do I care? by Thomas Cagley

Off course they measure productivity of doctors, engineers, plumbers and even football players (sacks per game is a productivity measure). The goal is improve efficiency. When money is loose it is less important however when the economy slows or contracts it becomes critical to try to squeeze every cents worth of value out of projects if for no other reason than to protect the jobs in organization.

Tom Cagley
www.spamcast.net

Re: Why do I care? by Bruce Rennie

Heh, good comment about football players.


Maybe doctors do have productivity measurements (patients seen per hour), but they also have a strong professional code that trumps any productivity measurement. Doctors talk about lives saved, not surgeries per day. Ditto for engineers: only the bridges that are still standing count.


As I also pointed out, things get even more fuzzy when a team is involved. In a team environment, measuring and optimizing at the individual level feels like I'm ignoring Theory of Constraints. I should be measuring and optimizing for overall throughput.


As a final question: does measuring productivity on a failed project make any sense?

Re: Why do I care? by John Reynolds

To Bruce's last point "As a final question: does measuring productivity on a failed project make any sense?", more meaningful to a development team would be:

- actual development time compared to estimated time
- re-work / bug fixing time vs. actual development time (this can be useful if you think about it)
- customer satisfaction / feedback survey

John

Danger - will destroy teams by Mark Levison

When you measure the performance of individuals you destroy teams. Once I know I'm being measured for producing code, here are somethings I will never do again:



1. Spend time in planning/retrospective meetings

2. Mentor jr team members

3. Pair Programming

4. Code Reviews

5. Simplify code, remove duplication, anything that might reduce my function point count

6. Anything else that would take time away from creating function points



Further more I will learn what function points are and learn to game them by writing code that maximizes the number of function points.


Find a measure of the individual and I will show you how it destroy's teams. Net result I recommend against measuring individuals or even teams. Instead I focus on outcomes.

Hudson Continuous Integration Game by Zachary Shaw

We've started "playing" the Hudson Continuous Integration Game I think it's a great tool to start talking about productivity / performance. I don't know if I'd love to be judged on it by management, but I do think it's a good tool for intra-team measurement. I talked about it a little on my blog recently.



The basic gist is that you get / loose points based on your checkin's. The more you checkin, the more tests you write, the more warnings / TODO's you clean up, the more points you get. It's true so far here "What get's measured get's done", warnings have gotten cleaned up, more tests have gotten written, and people checkin more frequently. It doesn't necessarily measure who's really productive, but it measures who is creating future work, so maybe in a way that is a measurement of productivity.

Re: Danger - will destroy teams by Zachary Shaw

So I agree that performance measuring can hurt, but if you measure on things you care about, code coverage, tests written, warnings removed, builds not failing etc. your measurement might start having a positive effect. (Just a side note, the Hudson Game would encourage #5)



I think it's important to measure things that you care about your team doing. I also think it's important for the team to agree on what they're being measured on. If you get team buy-in then it can lead to healthy competition.



I do not think that SLOC is a good measurement. It doesn't speak to quality. But there are a bunch of static analysis tools that do speak towards quality, that hook into a build. I think the idea of the Hudson game is to use that information to measure developers. Honestly, we've just started using the Hudson game, and it seems good, so I may be a little blinded by a new shiny toy, but can anyone see a downside to the Hudson game?

Re: Danger - will destroy teams by Mark Levison

My objections were to the measures as described. My real concern is there is any linkage between pay, bonus etc and any individual measure. See: gojko.net/2008/08/07/paying-programmers-are-bon... for a description of pay and performance.



I'm familiar with Hudson and think it an excellent way of highlighting quality issues as long as the information is kept private and within the team. I would be concerned if the team started to take it too seriously or management got involved.

Re: Why do I care? by John Jimenez

To follow your example: If you have two engineers, both create bridges that "still stand" but one designs two bridges for every one the other designs. Wouldn't you want to know that? Wouldn't the more productive engineer deserve more $$ or a promotion? Maybe there is something hindering the other engineer's performance. Isn't there value in knowing that?

Thanks.

Re: Danger - will destroy teams by John Jimenez

I believe you're right in the case you are working on a high-performing team. But, what if you have a team which isn't reaching deadlines or under performing? I believe you need empirical data (or as close to it as possible) to address members who are under-performing. Without it management could be accused of favoritism or worse.


Ideally one would prefer a culture to perform come from peers directly, through healthy competition, peer evaluations, etc. However, if this doesn't exist from within then it's management's job to make is so and productivity metrics are a vital tool.

Re: Danger - will destroy teams by Mark Levison

A couple of problems is this "lack of productivity" real or a sign of something else? Measure my productivity on a team and you will find its low. I'm the guy who acts as a mentor to everyone, I help review alot of code and I'm always refactoring and improving the infrastructure. When I'm part of team quality slow improves. Am I productive or not? By your measure not all.


What really matters here is the output of the team and are the team happy working together. If the team is producing high quality well tested, simple code - why does it matter how productive they are? If they're happy then who cares that one team member is really just acting as the glue.


When I coach I encourage management to set goals for the team and measure whether they meet those goals. Anything done on the individual level will destroy the hard won gains at the team level.

Know why you're measuring, and keep it squeaky clean by Robert Merrill

I ran the metrics program for a small (15-20 developers) project shop for 5+ years. We measured productivity for only one reason--project estimation for client proposals. We estimated two ways, using an agile-style planning game, and also using function point analysis. (I was a CFPS from 2002-2005). I counted each project as built, and used the time records to determine h/FP.


When it was time to estimate, I would count the requirements and consult the project history to predict h/FP. Having an independent estimation method was very helpful, and it also held up better under client pressure than the fully subjective methods. (Function Point Analysis is also an excellent way to spot requirements gaps).


To keep the whole thing squeaky clean (even though it was a high-trust culture already), I was the only one with access to the project history. Management never saw it, except in summary form.


Metrics are an amplifier. They will make a high-trust organization even more effective, and will make a low-trust organization even more toxic.


www.ufunctional.com

Quantitative performance measure will be the disaster to team. Be careful! by Wu James

I worked in a company and it ever pushed a productivity improvement program, "2X", which means double your teams' productivity. It was not at individual developer level. We measured the productivity in code lines. You can guess what we got eventually, tons of redundant code in system.

Deming said that you never measure your people in digits. That is not only true in manufactory, but also in SW industry. SW development is more complicated than assembly line.

When you link personal performance with individual's productivity, you definitely can get the number increased. However, it is not your revenue or quality. Your people is smarter than you. They can figure out everything to make your happy to increase their salary and bonus.

As Bruce said, "Why do I care about programmer productivity?". Exactly right. We are selling quality products not the code lines or even funtional point. Sofware is not built by blocks.

If you DO want to measure sth of performance, following DON'T/DOs maybe useful:

1. DON'T use the digits that the team can control. Use inputs or outputs from/to customer. Do link it with your revenue.

2. DON'T compare data across teams. Do compare it with your past to see the trend.

3. DON'T use complicate/fancy methods. That will definitely will increase overhead, or effort waste, to your team. DO use some indicators which are really simple and easy to collect.

If you can use your leadership, forget the digits.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

16 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT