@johncutlefish's blog

Check out Iterate.fm, my podcast with friend and coworker Tareq

Measurement Malpractice

Published: January 22, 2018

1 ThQ9aRV8ir6eYi6  W3tGw 2x

Why are you measuring that?

To understand how the team is doing!What do you mean by “doing”?

Oh you know…their performance. How things are going…No. I mean what decisions are you hoping to inform?

I’m not sure what you mean.What would do more of, do less of, add, remove, spend more money on, or spend less money on based on the data? Or are you looking to justify a decision or argument you’ve already made, perhaps?

Hmmm. Well, a lot of this is just how you do it. All the books suggest it. So I’m not going to second guess what they say. It is simply what you do when you’re working with a team.But let’s say this number was low, or high, what action would you take?

Well, I know for one thing that managers and executives love to see this stuff. They want to know that people are motivated, accountable, and committed. They want to see progress.Does this measurement do that?

I mean it gives them a general picture. On a high level, yes.Does the number being low guarantee that the team is NOT motivated, accountable, and committed? Can that manager take immediate action based solely on that data?

Well, no. I mean there are a ton of things that could make that number low. But it would be like a warning sign that something is off. They’d need to go ask questions…Are those questions that haven’t been asked before?

Well, I mean, a lot of this stuff comes up in 1:1s already, but…Does the measurement give you any clue whatsoever about what is off?

Hah. OK. Let me tell you the back story here. Please keep it on the down-low. There is this ongoing debate between the CTO and the CEO about our team here. The CTO claims that technical debt is piling up, and that a lack of company focus is bringing the team to a complete halt. The CEO, based on conversations with her friends in other companies, is persuaded that we just don’t have the right team in place.Wow. That sounds pretty serious, and very common actually. You can boil many of these requests for measurement back to some sort of unresolvable personal dispute. So are you suggesting that the measurement is meant to settle that debate? It sounds serious.

Well, not settle. But I think the CTO wants to show that we are doing our side of the bargain, and to maybe shore up his case. And maybe, I’m not sure, he might be wondering about a couple of the tech leads. As in… are they pushing people hard enough. I mean sometimes a sprint finishes and we have no idea why the ball dropped.No idea? Do you have a retrospective?

We do. And stuff comes up. But the same stuff…so we have no idea if it is really that serious. I mean, maybe people ARE slacking.So the CTO wants to know whether to fire his tech leads, but overall thinks it is the CEO’s fault for swamping them with work, and not funding debt work-down? OK. This really is serious, and again not uncommon.

Back to an early question. Will this data help unravel the assumptions of the CTO and the CEO? How does it indicate whether technical debt is “piling up”? What does a lack of focus look like? Will your measurement approach inform any of that?

I guess not so much.And what about the teams? What are they to make of all this? It sort of screams “we don’t trust you!” How might that impact their behavior?

They’ve already start to push back a bit. I found out that a team wasn’t putting their work in the tool recently, because they thought they might get penalized. One engineer asked whether any of these numbers had anything to do with delivering value to customers. I didn’t know what to say.I’d probably be asking the same things. Did you have an answer? What do you tell the teams when they ask Why?

I tend to say it is just how you do things. That more data is better than less data.Do you believe that?

Yes. In principle at least. But you’ve got me thinking. The big problem now is that everyone at the executive meeting LOVES to see these charts. It gives them a sense of confidence…About?

That work is actually happening! That no one is slacking. You can’t deny that. The numbers might not be perfect, but it does fill that need…Hmmm. At the potential risk of getting people fired, and sowing distrust. What exactly is the executive need?

Hmmmm. Trust. Wait no. Yeah….trust.