Measurement Fails in Business, Part II: Real-World Examples and Solutions
“The numbers don’t lie!” is a mantra proclaimed thousands of times daily. Odds are you’ve used it many times yourself. Whether in a corporate boardroom, town council, local school board meeting, or amongst a group of staff members working on next year’s budget, it is declared with authority and the sincere belief in the equation N = T (Numbers = Truth).
But my decades of studying numbers used to manage business, not to mention many other aspects of our lives, demonstrate—with apologies to George Gershwin—“It ain’t necessarily so.” Every time I hear someone say, “The numbers don’t lie,” it puts me on skeptical high alert, as it should everyone.
My first memorable experience of how seemingly straightforward numbers can be manipulated came when I was a 22-year-old turn foreman at U.S. Steel’s South Chicago Works, the world’s largest structural steel mill at the time, employing 20,000 workers. Following each shift, we updated the steel report, which showed the tonnage of raw material—in the form of 10-ton ingots—that went in the front end and the quantity of rolled I-beam product that came out the back.
Simple, right? Not if you were trying to make the mill appear to be running smoothly—a virtually impossible task on a site with roots put down before the Civil War and suffering from a decades-long shortage of maintenance.
There were multiple rolling mills within the complex. To balance the material coming in, the gambit was to make deals and trade ingots, mostly on paper, with another mill. To balance the material coming out, we would pile excess product without recording it, in a massive storage area when exceeding quota and then draw from it when we fell short.
The practice was an open secret; everyone accepted it, everyone did it and believed every other mill played similar games. But those running the big picture in Pittsburgh never had a clue what was really going on in Chicago, yet they managed the mill using our manipulated numbers. I wonder how often the Pittsburgh decision-makers were overhead proclaiming “The numbers don’t lie!
RELATED
- Measurement Fails, Part I
- Data-Driven Builders Leave the Guesswork Behind
- Can We Make Home Building a Little Less ‘Interesting?’
When It Comes to the Accuracy of Business Metrics, Be a Skeptic
My January column, “Measurement Fails, Part I” laid out the many common obstacles standing in the way of achieving accurate measurements, including estimates presumed or disguised as facts, and ratios based on bad numerators, bad denominators, or both.
Remember my proclamation on ratios—An estimate divided by an estimate is bull$#%@!—which originated with a particularly frustrating meeting with a national home builder years ago at which highly suspect numbers were tossed around. Last issue’s article reviewed how graphs are often either poorly conceived (resulting in distorted data) or are deliberately distorted to get the desired spin. Either way, the results are bad numbers, and even if not willfully dishonest, they don’t represent the truth.
Graphs are often either poorly conceived (resulting in distorted data) or are deliberately distorted to get the desired spin.
That column ended with a pledge to dig into more detail, with real-world builder examples. But before we talk specifics, I challenge you and your company to never accept anyone’s numbers at face value. Be skeptical. Review their origin. Examine their background components, measurements, and calculation methodologies. Determine if the data purveyors are pushing a particular agenda.
Often as not they are, yet this doesn’t necessarily mean the numbers are “cooked.” We use numbers to present a case, to persuade, to sell, and that’s a good and normal practice. Still, keep your eyes wide open and watch for it. Two people with opposing agendas can manipulate the same data set to offer very different interpretations.
As described in Part I, simply compressing or expanding the scales on graphs, or cutting off the scales at the top, bottom, or both, can completely alter the message and influence decisions. What follows are some of my favorite examples.
Examples of Real-World Measurement Fails in Home Building
Model Traffic/Closing Ratios: How Do You Count Them?
While this appears simple, even mundane, it’s so messy it’s almost impossible to compare builder against builder.
How do you count traffic through your models? Let’s take a typical “buying unit”—a mom, dad, and two kids—all walking your models together on Sunday afternoon. Put that down as a single traffic count. Then dad returns on Tuesday morning to talk pricing and ask about available deals. Is that another traffic count? Mom then comes back in alone on Thursday afternoon. She really takes her time and asks a ton of questions—a significant indicator this sale is progressing. Another count? On Saturday morning, mom and dad return with the kids and slowly walk and talk them through mom’s favorite model, gauging the kids’ reaction. This family is ready! They then return on Sunday and sign a purchase contract.
Is this one traffic count or four? You could argue either way. Builders are keen for high “closing ratios,” and if you count this as just one unit of traffic, it’s simple: one traffic count, one contract. Salespeople quickly learn: Higher closing ratios look better. But is recording it as one contract after four visits a more realistic picture, capturing more accurately the amount of time and effort the salesperson expended? What, exactly, are we measuring here?
And now the head of sales considers the bundle of cash she fought for to spend on ads and billboards—an expense she knows will be judged by traffic count, at least in the near term. In this scenario, she perhaps counts this as four visits—or not. What’s the more important metric, closing ratio or traffic count, and why? You’ll have to decide. But make it a thoughtful, fully aware decision on how you’ll generate the numbers. Stay consistent and make sure your salespeople have no incentive to be less than completely honest.
Now, are you comparing your traffic and closing ratios with a recommendation you read in a magazine, or to another member of your 20 Club? Numbers don’t lie ... right?
Schedule/Cycle Time: At What Point Do You Start Measuring?
My TrueNorth colleagues and I have spent decades working on cycle-time measurement and have developed a “Saved Day Calculator” Microsoft Excel template for figuring the total value of a day in the schedule (to request the calculator, see the info at the end of this article). But, especially in these days of inflated schedules, the measurement is fraught with complications.
First, at what point do you start measuring cycle time? Do you include time spent in the permitting process? Permitting is hyper-local and can vary dramatically even within a single metro area, taking from two weeks to two months or more based on location and jurisdiction. And if you’re building multifamily units, then double or maybe triple that time. My advice: Measure and track permitting, but keep it separate, not as part of cycle time per se.
My TrueNorth colleagues and I have spent decades working on cycle-time measurement (and) the measurement is fraught with complications.
So, begin tracking cycle time at the foundation start, right? But wait, are you building on slabs, a crawlspace, or poured walls? And if it’s a slab, is it monolithic, post-tension, or one of countless variations? Or perhaps it’s a basement but it has a precast wall system, saving five to 10 days. And try comparing the cycle time of a basic slab unit on stable, flat soil in Oklahoma City with a poured-wall home on rocky soil and challenging topography in Harrisburg, Pa. You can’t meaningfully assess those differences as a single metric within overall cycle time.
The only solution is to pull foundations out of the cycle-time measurement, as we did with permitting. Yes, you need to count and track the foundation process, but again, keep it separate; that is, unless you build using just one foundation method and look only at your local company results, not benchmark them against other builders.
Having removed permitting and foundations, let’s simply call it “construction cycle time” and begin with frame start. You may encounter some hiccups to that logic as production progresses, but let’s jump to the end, which is defined … how? Final inspection? Certificate of occupancy? Close of escrow? There are reasonable, good arguments for each, and others you may consider the “end” of the construction cycle. To make the call harder still, throw in your spec units, which many builders intentionally stop at some point short of final, perhaps to allow color, cabinets, and counter selections. Do you include that downtime in the home’s cycle time, even if the home sits for months? Consider all of the factors, decide on what indicates cycle time best for you, stick with it in your operation, but again, beware of comparisons with other builders; it’s highly unlikely they’ll be apples to apples.
RELATED
- Business Success: Easy as 1, 2, 3?
- Roll Up Your Sleeves and Get Serious About Cycle Time
- The 5 Most Common Home Builder Measurement Fails
Variance: You're Underestimating Its Effect
Look up the money you paid out on variance purchase orders last year and multiply by 20. That’s the true total cost of your VPOs. Impossible, you say? I can prove it. I’m so obsessed with variance, I wrote a four-column series about it (see the info at the end of this article to request a PDF version), and I was just getting started. I’ll boil those 10,000 or so words down to a few paragraphs here.
The first issue is the quality of baseline budgets. Are they accurate and fair? Was there enough time to do a good job with estimating and bidding, or was completing the budgets rushed, with a lot of “best guesses”? Did Purchasing play it straight? Or were they pushed into screwing the numbers down to get costs as low as possible, or were they able to build in some slack? Or maybe, in some areas, a lot of slack, to protect themselves from never-ending price increases.
The bottom line is: Everyone focuses on the additional costs of variance, yet we rarely look back at the original budgeted numbers to see if the variance is the result of changes, errors, omissions, rework, or things gone wrong in the field, or whether variance was induced by using flawed numbers to start with. A close examination of these cases shows the initial reported numbers are virtually always wrong.
Harder Still: Measuring Variance
That’s the easy part; more difficult is counting the variance as it comes in from the field. Go look at your last 10 VPO requests (maybe you call them FPOs or field purchase orders, EPOs or exception purchase orders, or simply Variance). What you will see are entries under Labor and/or Materials. You’ll find significant dollars there, but deeper analysis will show these costs represent the minority of the variance cost. In fact, the majority comes from process and overhead costs incurred from dealing with variances, not just for the builder but also borne by suppliers and trades. These are huge costs, on average far exceeding labor and material costs. And although difficult to measure, open minds, honest analysis, and courageous calculation will get a handle on them.
As bad as that is, now consider this: TrueNorth has information from nearly 5,000 suppliers and trades who have participated in our LeanWeeks, indicating they submit less than 20% of all potential VPOs and at best 50% of those are paid. (For some builders, the pay rate is as low as 10%.) Thus, even if you track labor and material variance well, you’re seeing just 10% of the true cost. Process and overhead will at least double that number, which means all of the money you paid out for VPOs last year was less than 5% of the true total cost—strong evidence that the true total cost of variance is 20 times what our typical numbers show
ROI: More Reliable for Measuring Builder Success
I could go on and on about this topic, and we could perform the same analysis on quality and inspection numbers, performance metrics, and one of my favorite current subjects, the true total cost comparison of on-site versus off-site construction methods. As an industry, we measure absolutely none of these aspects well.
Finally, everyone is obsessed with gross profit and net profit. That’s fine, but especially with the overly inflated schedules we now experience due to labor and material shortages, along with a host of process problems, I argue that ROI (return on investment) is a far more meaningful number for measuring builder success. Builder B may show two or three more points to the bottom line in a year than Builder A. But if Builder A builds in half the time, he is beating Builder B, badly. ROI will show that; pre-tax net profit will not.
If that isn’t perfectly clear to you, sit down with your CFO or accountant and hash it out. Be careful though, I told you not to take anyone’s numbers at face value! Remember Mark Twain’s warning: “There are three kinds of lies: lies, damned lies, and statistics.”
If you would like a PDF of Scott’s columns on “Measurement Fails” or the Excel template mentioned in this column, please email [email protected] with your request.
