Consumer Report responds to criticism of their review

You have to look at masses of data and realize there are undocumented, hard limits, and a lot of 'red herrings.' Many of their metrics are not (or have not yet) been used in the "score."

My contribution to the methodology is to only concentrate on the bottom score vehicles. It doesn't take long to find the few, pass-fail, metrics that drive the score. Think of all cars starting with 100 and then points are taken away when hard limits are violated.

I don't think they are that clever. I suspect they are also aware that once their "score" algorithm is published, car manufacturers will use them as requirements for their design teams. In a couple of model cycles, 3-6 years, like the kids at Lake Woebegone, everyone is above average.

Funny you should mention this but the VW Beetle was slammed in the July 2012 issue. At first I couldn't figure out why until:

• 3 - Noise
• 3 - Driving position
• 3 - Emergency handling

It had passed the acceleration threshold so no points were taken away. But we have already figured out that "Noise" and "Driving position" are big point killers. Add poor braking or "Emergency handling" and we are starting to get an idea of how they calculate their score. With enough data points, we can eventually map out both the thresholds and penalties.

Bob Wilson

 

Exactly. Much like the IIHS and NHTSA crash testing methods, but at least for those, that's part of the intent.

It's not unlike their mysterious reliability ratings. You'll never see the total number of samples for a vehicle, standard deviations or any other statistical data. Also, everything is normalized so you don't see any absolute numbers. This makes it difficult to know what you are seeing. If you take some of the overall information about problems vs. years chart for each make they sometimes publish in the April auto issue, you find that the differences between an "above average" reliability car and a "below average" one are not as big as you might think from a half-red and half-black circle. It amounts to less than a couple more problems of unspecified severity and cost over the course of 5 years. That's because vehicles from all makes are pretty reliable and it might not be enough to sell copy if they didn't normalize the numbers to exaggerate small differences.

I have no data to prove this, but in the last few years I have a hunch they give extra penalties to vehicles they feel are worse than the previous generation model or a "sister" model from the same make. The Prius C and Civic are good examples of that. Even though they may still compare reasonably well to other cars in their class from other makes, once you see words like "decontented", it's likely that an unusually low rating will follow. Also, I think the objective thresholds and weightings, if they do exist, may vary considerably from one class vehicle to another.

 

What makes a good driving position is obviously very subjective. I find my Prius c Three very comfortable, comparable to the 1996 BMW 318ti I just switched from (another bottom of the line car).

Here's a video describing CR's criteria for a good "driving position" - it covers more than just the driver. The narrator is the infamous Tom Mutchler, who slammed the C in the magazine then posted this defensive and sarcastic reply to criticisms of his review on the CR blog.

 

I would love to see "scores" instead of, or better yet, in addition to, the vaguely worded handling explanation this review gave us on the C. I don't have the article in front of me, so this is paraphrasing, but he said the car 'handled soundly on our test track but wasn't particularly agile'. 'It lost it's grip early but the stability control system kept it secure.' What does that mean? Did it lose it's grip or not? Did it feel like it was going to lose it, but the stability control system prevented it? All too vague.

Then he goes on to say that it did 'decently in our avoidance maneuver'. Wouldn't the car have to be at least somewhat "agile" to perform well in that test? And what does decently mean? How about some substance here. A rating from 1-10, a comparison against other cars in its size/class, a MPH at which it no longer performed "decently". There "are" some positives to be noted in his explanation, but they all seem couched in negatives, as if the reviewer had the hardest time admitting to anything good about the car.

The reviewer revealed his bias in the response he wrote to another car review website's criticism of his Prius C review. Apparently the car already had one strike against it before he even opened the door, simply because it was 'traffic cone orange'. I don't know how much his bias colored the review, but I certainly would have liked a better written one, with more charts and numbered ratings, as well as more clearly worded descriptions of the handling characteristics. Unfortunately, a page and a half is typical length for a CR car review, a restriction he probably had to work within, which is why no one should make a car buying decision based solely on their words.

 

Ahhh HA! I see the problem. His wallet was fatter from the money saved driving the Prius c and it hurt his butt.

Bob Wilson