Consumer Report responds to criticism of their review

I appreciate that one can find Consumer Reports web pages that summarize some of their reports, the best and worst. But these 'metric' pages don't help to explain the CR "score" used to rank cars.

The curious behavior of Consumer Reports "score" is more evident looking at page 64 of the July 2012 paper copy. This page has a full table listing their metrics but not their 'scores' for the cars reviewed. So I wrote the CR "scores" above each car. So taking one cost related metric, "Annual cost, 12k mi. at $4.00/gal" we find:

• 77 - $1,565/yr - Ford Focus
• 72 - $1,805/yr - Chevrolet Cruze
• 71 - $1,550/yr - Hyundai Veloster
• 61 - $1,450/yr - Honda Civic
• 60 - $1,875/yr - Volkswagon Beetle
• 53 - $1,110/yr - Prius c

The worst rated car is cheaper to operate, substantially cheaper. Yet the Consumer Reports "score" gives no clue.

Several years ago, I encoded a similar page from another Prius review to see if I could ferret out a function that gives similar "scores" based upon the Consumer Reports metrics. I didn't have much luck but I was assuming a linear relationship. I now suspect non-linear weights and selected metrics are used.

Comparing the Consumer Reports "defense" with pp. 64 of the July 2012 issue, here are the Prius c metric - the defense "complaint" - [ranking top score to lower]:

• 2 - "loud" :: [4, 4, 3, 3, 3]
• 3 - "driving position" :: [4, 4, 4, 3, 3]
• 11.3 - "acceleration" :: [8.5, 10.0, 9.2, 9.7, 8.6]
• 3 - "nimble handling" :: [5, 4, 4, 3, 4]
• 3 - "ride is always busy" :: [4, 4, 3, 3, 4]

The strongest correlations are "noise" and "driving position" across all vehicles. But the VW Beetle was hit with a low score even though it had the second fastest acceleration. This suggests a minimum, acceleration. Since the second best "score" vehicle came in at 10.0 seconds the minimum acceleration is between 10.0 and 11.3 seconds. Their web site of "worst acceleration" seems to have a floor of 11.0 seconds suggesting this may be the threshold, the upper limit.

We can treat Consumer Reports as any other 'black box' and with enough metrics, come up with a model that predicts their "score." But on this limited sample, we can hypothesize the CR score:

1. "noise" and "driving position" are major parts
2. "acceleration" has a threshold requirement greater than 10.0 and less than 11.0 seconds
3. "operating cost" plays little or no part

In contrast, Prius owners are often "operating cost" driven. We drive efficiently and that means "acceleration" is not so important and this reduces "noise." We also drive at or under the speed limit and choose slower routes which further reduces road and engine "noise."

Bob Wilson

 

toronado455, you make a lot of irrational comments, (much like the reviewer), I will try to help.

1. It is the modeler's job to decide how the car will most likely be operated. e.g. implementing the 80/20 rule, while that would leave out hypermilling on one end, it would also leave out the apparent drag racing they were attempting to do on the other- which lead to the inexplicable 37 mpg they claim they got.

2. There isn't enough evidence to answer why they would have any agenda against the car, but a scientific mind would NEVER react in the petulant and hyper defensive manner this reviewer did. There were valid criticisms of the initial review, which by its nature includes a great deal of subjectivity, so one would expect a person without an agenda to concede on some minor points (like the 37mpg) in order to bolster the validity of their overall model. Doing the opposite of that there is only one reasonable conclusion; this author, and perhaps Consumer Reports as a whole, has some sort of agenda.

3. >Who is to say what test is appropriate for a hybrid?
The mean and median of hybrid drivers, see answer #1.

4.>If hybrids are so different from other cars that they require a special driving technique, then maybe the manufacturer should disclose that to the consumer. Otherwise, how would a testing organization like Consumer Reports know that they are supposed to alter their testing technique to accommodate the special needs of the hybrid?

​

What fatuous nonsense. If the majority of hybrid drivers gravitate to these "techniques" (e.g. not flooring it continually) and figure out without such "disclosures" on their own, how can Consumer Reports, which is holding itself up as an authority to test cars year after year, supposed to be taken seriously if they allegedly are too simplistic to also uncover this secret Easter-egg of car operation themselves? I might grant some credence to this idea if it were 2001 and the Prius were new and had a small group of loyalist driving it, but its been around for over a decade and has 4 million on the road. If Consumer Reports requires a white-paper on how to operate an econo car, they need to reconsider the assumption that they have the wherewithal to review cars and should stick to vacuum cleaners.​

 

"Significant"? Are you trying to kid me? Do you expect me to believe that a human being without the aid of a stopwatch would even be able to perceive the difference between .4 seconds of an 18 second acceleration?

Which brings up the broader question, why in the name of Sir Isaac H Newton is Consumer Reports testing a Prius at nearly 80 mph in the 1st place, and even more broadly, if such metrics weighted significantly in a decision, why would said person be considering a car optimized for fuel efficiency?

cwerdna, if you or CR took .4 seconds of a break from an excel spreadsheet to really think through what this test is claiming to evaluate, you would realize their structure of measurement is so patently silly that it's to the point of uselessness.

 

My head just exploded. I've never seen that detailed of an analysis on... anything... on an internet forum.



I think the bottom line here is the hostility and clear bias the report is written with. Their completely unprofessional reply to the criticism bears this out as well. The problem with most automotive reviews was already pointed out by someone else. If it's not a race car, most reviewers aren't interested. They don't have to cope with real-world operating costs and wear and tear. The more practical a car is, the worse of a review it's going to get, even while it continues to sell like mad.

 

Actually, I have to disagree. A half second in the quarter mile is the difference between a Camry V6 and a Mitsubishi Evo. It seems insignificant on paper, but a car that's half a second and 3-4mph quicker in the quarter mile is completely walking the slower car in a race. A car covers a lot of ground in a half second when it's going 75+mph.

Also, having driven the regular Prius and the C, there is a very noticable difference in acceleration.

 

"Quarter mile" "race" and "Prius" are three things that should never be mentioned in the same discussion. Quarter mile times are useless in real world situations. I've been driving for 15 years and have never come across a situation where that much acceleration is needed. There is no reason to ever have to come from a dead stop to merge doing 80 mph. There are traffic signals and merge lanes for a reason. I drive to and from DC on a daily basis and the C has more than adequate acceleration to be able to handle driving with the maniacs here. There is no such thing as a fast car with hybrid like efficiency (excluding EVs). There are still cars on the road with slower acceleration and they do just fine.

SGH-T989 ? 2

 

I actually think it seems more significant on paper than in the real world. Don't underestimate the qualitative considerations here affecting that perception. I believe we're talking about apples and oranges, there are a lot of brand perceptions, as well as quantitative differences between the Camry V6 and the Evo, combining all those differences together makes the acceleration difference seem significant because you overall feel the difference between the brands. Also, if I'm walking the .4 seconds to go 1/4 mile would make a significant difference to my perception of difference, but I'm going to be hard pressed to notice the additional feet I've traveled driving those speeds. None of this alters my original point; a human being is not going to be able to feel a "very noticeable" difference that it took 18.4 vs 18.0 seconds to go 1/4 mile. I'd like to see them run that test 30 times, I bet .4 seconds would be very close to come within the margin of error within each car's tests' times, it certainly doesn't matter.

I have no reason to doubt you that you perceive them to be slower, my suggestion is that you are subconsciously bringing in a number of other variables into that distinction between the Liftback/C because I simply do not believe anyone is perceiving that fractional difference, unless you are Rainman.

I want to be clear, in no way am I claiming the C is anything other than a slow car. Those times are horrible compared to other cars, but my original point, which I believe has yet to be refuted, is that all 3 times are horrible (this assumes that this metric is relevant for this class of car, but that's an entirely different discussion) Therefore, if the Liftback and V are not weeded out of contention for the magical "recommendation" status, the C being dinged so heavily for the fractional and insignificant difference is simply not logical.

 

science. if their test never allows the hybrid to get to optimal temp, then the test is useless, because in reality the car will most likely get to temp. so they are testing a condition that most of us will never see, a worst case scenario if you will. its nice to know what would happen in that instance, but that is not the norm. so shouldnt they test and report on the norm as well?

ok yeah we all dont drive the same way. lets say someone does drive like their test cycle and gets that mpg. well shouldnt they also report on others who dont drive that way? they represent one side of the story and totally ignore the other side as if it doesnt exist. THAT is the problem.

the question is DO they know? if no, then why dont they know? wouldnt you as a journalist question why you cant even get EPA mpg? wouldnt you question why others are getting way more mpg and you cant? and if they DO know, then that's an even messier situation...

 

I've been drag racing cars since I was 16. I'm saying there IS a noticeable difference. All three cars are slow, but the C is noticeably slower in the real world, not just on paper. People spend thousands of dollars to knock .4 seconds off a 1/4 mile time.

CR is still stupid for knocking the car for running that time, given what it is, but there is no debating it is under powered compared to the full size.

 

OK, football players spending thousands in personal trainers to knock a tenth of a second off their 4-40 times, that's their job, so what? I'm not going to evaluate my needs in a jogging routine against their's. Why would anyone care about car nerds spending habits on fractions of a second relating to this topic?

How do I put this nicely; a drag racer's perspective falls outside the mainstream. That's OK, but own up to that and don't pretend you're average. Why would you assume I care about the opinions of people who value car racing as it relates to hybrid car evaluations? This is almost as nonsequitar to discuss as the totality of CR's review. For the 3rd time, .4 seconds is imperceptible to anyone without a stopwatch, I understand that difference matters to you, and even that I'll grant the possibility that you might have the gift of tracking such a difference in your mind, I'm just saying it is insignificant between the Liftback and C, and I don't care.

 

You keep missing my point. It's not about nit picking and bench marking. 0.4 seconds is 44 feet at 75mph. That's over two car lengths . It's not insignificant and even grandma will feel the difference. So while it's dumb to knock an intentionally slow car for being slow, saying that it has noticeably worse acceleration than the full size is definitely true. The difference in power was the one thing that made me hesitate to go with the C over the full size.

 

Where exactly is CR's alleged "bias"? I've seen this accusation several times in this thread but have yet to see anyone spell out any evidence for it. Is CR biased against small cars? Is CR biased against cars that get good fuel economy? Is CR biased against practical cars? Is CR biased against Japanese cars? Is CR biased against Toyotas? Is CR biased against the Prius? From years of reading both Consumer Reports and many other sources of automotive reviews, I would say, clearly, "no" to all of the above.

Maybe if you were talking about Motor Trend or Car & Driver, I could understand that criticism. Those publications do have a reputation for using the BMW M3 as their "reference design" by which all others are judged. But Consumer Reports doesn't do that. They publish much more balanced reviews geared toward a more mainstream consumer than those of the enthusiast mags.

Actually, they keep their cars for a while before they sell them and have experienced problems with their cars, such as with their Fisker Karma.

This is probably the exact opposite of Consumer Reports.

I sure hope that Consumer Reports NEVER factors in sales numbers into their criteria for how they rate and rank cars. I can't think of a worse metric to include in an evaluation. Just as I wouldn't rate the food at McDonald's highly just because it sells well.

 

How long does it take a Prius c to get to optimal operating temp?

Even factoring in the period of the ICE running more while cold, CR's 150-mile test should give a reasonably realistic real-world result. Unless you expect hybrid owners to wait until the car warms up before beginning their MPG measurements, you should not expect CR to do that in their test.

 

I'm in agreement w/AWDstylez. It sounds like you've never taken cars drag racing before. I have w/my former Maxima and 350Z before. At tracks, they don't use stopwatches. The margin of error is FAR too great. There are timing systems installed a tracks and they'll give you very accurate 1/4 mile time and speeds, reaction time, 60' times, etc. I've attached an example (I ran the 14.908 @ 94.32 mph). I distinctly remember that one as my I got my nice person KICKED by that Civic.

0.4 seconds is somewhat significant and yes, the faster car would be walking away the whole time. As for nearly 80 mph (76.2 mph in the Prius c's case), 1/4 mile time and speed is a pretty standard test of acceleration. The speed is the speed you're at when at the 1/4 mile mark. I've observed all sorts of cars at the track (partly due to having to wait so long to get my turn). 17+ seconds in the 1/4 mile is very slow. 17s and 18s are what econoboxes get.

It's just another objective test. Should they NOT run objective acceleration tests just because it's a Toyota, a Prius c or a car designed for FE? No. There's no reason they should be excluded.

Would you be happier if they resorted to the "butt dyno" and said "we're not going to run any acceleration tests beause it was designed for FE"? What if they said "we're not going to run any acceleration tests because we feel the car is slow?"

 

No, they shouldn't run acceleration tests. It hurts people's feelings.

 

A 1/4 mile race between two cars separated by 0.5 seconds if the times are about 5 seconds, is quite a bit different than if the times are about 10 seconds.

Back of the envelope calcs say the fast cars differ by about 25% power between them, while the two slower cars have about a 10% difference in power. 25% you can feel, 10% -- I doubt it. The absolute differences in the feeling of being pushed into your carseat are of course even greater, about 100 %.

 

I can't see where either the CR mileage numbers or 1/4 mile times are relevant to how
"good" (every person's definition of good will differ) the Prius c is.

Like the EPA mileage numbers, the CR test and resulting numbers are meant solely for
comparison to other cars that have been likewise tested. While both organizations will
say that there test is valid, they are different. But it doesn't matter...
What matters is people can compare cars that were tested alike.

It would be simplistic to demand that the tests reflect the "real world." Whose real world?
A driver in the mountains of Colorado? A LA commuter? A doctor commuting to work in
the early morning on semi-rurlal roads across gently rolling Mid-west farmlands? Yours?
Mine?

There are many automotive related tests/competitions that are likewise irrelevant for
everyday driving. NASCAR, F!, hill climbs, World Rally, and auto cross to name a few.
Even Fuel Efficiency challenges are meaningless. What significance can be made of a
race where the cars tear up the road at ~25 MPH?

Are there metrics which might be meaningful? Yes, but I doubt that there would be much
agreement on what they are. IMHO, ones to consider are:
* 30-60 MPH (time and distance) relating to freeway entry
* 60-80 MPH (time and distance) related to overtaking/passing

If you want the low down on any car, you find car-specific sites like PRIUSchat and lurk
for a while…. maybe a good long while until you can judge the competence of the posters
that routinely post. And you take what you read with a grain of salt, in some cases a fistful
of salt.

That's what I did before I bought my '08 Gen II. After I read CR, Car&Driver, Motor Trend,
Edmunds, etc, and the EPA tests, I lurked at PRIUSchat for a month. And everything I read
here was not positive. Members were tearing Toyota up for what were and still are
problematic. (Not being able to open the lift gate when the 12V battery is dead, the relative
fragility of the 12V battery, the manufacturing defect in the '05-'06 Multi-Function Display, etc.)

But I bought the car anyway. We haven't looked back, and it has proven to be a "good"
purchase for us. I don't waste time defending my purchase or being an apologist for hybrids in
general. Ignorance and/or hard headiness is its own reward.

So, in the range of material that should be considered before buying what is essentially a new
type of car (for the first-time buyer) what CR has to say should be just a blip on the screen,
a single data point. Nothing to get all wrapped around the axle about….

either before or after you buy the car.

 

Actually I have been to the track too many times and in too many cars to count. The C's acceleration is more than adequate to navigate through traffic you just have to plan accordingly. It wasn't meant for speed, it was meant for efficiency. I don't see any 50mpg Z's on the road nor would I expect to. So don't expect a Prius to have the quarter mile time of a Z. When only comparing the Prius family a .4 second time difference is nothing. Consumer Reports is ridiculous in saying that the acceleration is good enough in the liftback and V but not in the C. No one in an econobox is going to jump out in front of any car moving at a high rate of speed unless they are absolutely stupid. The track is the track and no one, except apparently you, cares what the quarter mile time is because it is irrelevant in daily driving. They just want a good, reliable, efficient car.

SGH-T989 ? 2

 

I knew someone would say this (like this) eventually. The truth is that demographic analysis allows for a much more accurate test than what CR is doing- the EPA test is a great example. Their number is still low, but it's heckuva lot closer than what CR is doing.

To give valid comparisons to other cars and predictions for real-world driving, you don't just take any old piece of crap test and use it forever for the sake of consistency. It's not only pointless to drive a car extremely hard in a test (since virtually no one wants to destroy their car), it gives skewed comparisons between the cars themselves. A high-rev engine is going to do much better in such a test than a lower-rev one, for example.

And there's no need to do that. This is all about bell-curves and statistics. Do the research and stick the test in the middle of the "use" bell curve (for all cars- ala the EPA). That's at least a good place to start.

But, if as a consumer product evaluator you REALLY want to do your job right, you can take it farther than that. There are a ton of different ways that variables can be changed to give people more information about how a car performs under different conditions. Just one idea is to come up three or four basic weather scenarios- cold, warm, hot, etc.- and test and report on those, too.

Then- especially if driving a hybrid- TRY HYPERMILING WITH IT! Holy crap, THAT'S CRAZY! YOU CAN'T COMPARE THOSE NUMBERS TO OTHER CARS YOU DIDN'T HYPERMILE WITH!! Well, no, but you're still telling people exactly what the car can do! Isn't that the point?

Just run lots of tests that are based on REALITY and be forward with your results. Be thorough. Listen to valid criticisms, and don't report numbers that aren't even close to real.

 

What else can be said? The topic of fuel-efficiency measure has been controversial for years, but limited to certain markets. Now that Toyota has penetrated into the lower-cost realm in large enough numbers to draw attention, the pot is getting stirred even more. To make matters even more complicated, adding a plug adds an entirely new dimension to the already confused comparison scales.

How will middle-market consumers make sense of so much variety? Each design differs well beyond the typical understanding of what's under the hood. Trying to apply that to your own particular driving circumstances is a fruitless effort, even without considering seasonal change.

Automakers will provide specifications. Publications will state their observations. But like in the past, it will be owner word-of-mouth endorsements contributing heavily to sales.