Is Consumer Reports Letting Readers Down?

micheal · 04-25-2007, 07:59 PM

I'm not sure if this is the best place or not, since this really isn't a specific news article about hybrids. I was afraid it would sucked in the black hole that is FHOP, and it is related to news about hybrids.

Anyway, I just recently got a chance to look at my latest edition of Consumer Reports and was able to see all of their apologies and analysis of what went wrong with the Carseat Fiasco of 2006. I was curious if anyone else was a little irritated with the CR "correction" of the hybrid hype analysis and the CR correction of the carseats. I noticed both the letter to the reader and a several page description of what went wrong in their tests and reporting. Meanwhile, the correction for the hybrid hype shenanigans was I think only mentioned in the letter to the editors portion and the new graph put on the website along with an acknowledgment of error. I think the press release that CR first put out about the carseats did not even mention the hybrid miscalculation as one of their errors over the years.

This could be an apples to oranges comparison, but it seems that CR is taking their error on carseats was much more serious than their reaction to the error for hybrids. Is anyone else seeing this or irritated by this situation?

I wonder if the carseats being taken more seriously only because it was a safety issue or is there something more sinister in this situation? Whatever the reason, I think CR has lost a lot of their credibility with me, although I probably won't stop subscribing.

JimboK · 04-25-2007, 08:09 PM

I don't think it's anything sinister. I really do believe they try to be objective. My take is that they uncovered some significant methodological flaws with their car seat research, whereas the only (admitted) error with the hybrid report was with the math. Yes, their methodology there is open to scrutiny, but they thought -- and apparently still think -- it was good. So the response was probably appropriate to what they genuinely believe the seriousness of each issue to be.

Wildkow · 04-26-2007, 06:13 AM

Who knows but the car issue has a picture of the Prius in the box talking about reliability.

Wildkow

adamwmcanally · 04-26-2007, 08:47 AM

<div class='quotetop'>QUOTE(micheal @ Apr 25 2007, 06:59 PM) [snapback]430184[/snapback]</div>

Quote:

I'm not sure if this is the best place or not, since this really isn't a specific news article about hybrids. I was afraid it would sucked in the black hole that is FHOP, and it is related to news about hybrids.

Anyway, I just recently got a chance to look at my latest edition of Consumer Reports and was able to see all of their apologies and analysis of what went wrong with the Carseat Fiasco of 2006. I was curious if anyone else was a little irritated with the CR "correction" of the hybrid hype analysis and the CR correction of the carseats. I noticed both the letter to the reader and a several page description of what went wrong in their tests and reporting. Meanwhile, the correction for the hybrid hype shenanigans was I think only mentioned in the letter to the editors portion and the new graph put on the website along with an acknowledgment of error. I think the press release that CR first put out about the carseats did not even mention the hybrid miscalculation as one of their errors over the years.

This could be an apples to oranges comparison, but it seems that CR is taking their error on carseats was much more serious than their reaction to the error for hybrids. Is anyone else seeing this or irritated by this situation?

I wonder if the carseats being taken more seriously only because it was a safety issue or is there something more sinister in this situation? Whatever the reason, I think CR has lost a lot of their credibility with me, although I probably won't stop subscribing.
[/b]

I think it might be a case of apples to oranges. When the initial carseat findings were published, they were on radio and television calling for a recall of most of the seats tested as well as governement investigations. This whole fiasco has hurt their credibility, but knowing CR they investigated how this happened and will probably be a better organization because of it.

PriusRos · 04-26-2007, 09:12 AM

<div class='quotetop'>QUOTE(micheal @ Apr 25 2007, 07:59 PM) [snapback]430184[/snapback]</div>

Quote:

Anyway, I just recently got a chance to look at my latest edition of Consumer Reports and was able to see all of their apologies and analysis of what went wrong with the Carseat Fiasco of 2006. I was curious if anyone else was a little irritated with the CR "correction" of the hybrid hype analysis and the CR correction of the carseats. I noticed both the letter to the reader and a several page description of what went wrong in their tests and reporting. Meanwhile, the correction for the hybrid hype shenanigans was I think only mentioned in the letter to the editors portion and the new graph put on the website along with an acknowledgment of error. I think the press release that CR first put out about the carseats did not even mention the hybrid miscalculation as one of their errors over the years.

[/b]

I didn't even see the correction about the hybrid calculations -- what is the miscalculation you are referring to and is the correction in the same issue that analyzed the carseat test problems?

I do think that they had to put out the report about the carseat testing problems because that was a major big deal. They had mistakenly TRASHED several of the leading carseat brands

Wiyosaya · 04-26-2007, 10:32 AM

Unfortunately, Consumer Reports is about the only consumer product testing agency out there, and as such, I see the potential for many mistakes like this. There almost certainly will be future instances like this.

Product testing is a science, and as a science, product testing should have verification testing done. This is not the case with CR. You take CR's word for it, or you don't. There is no other organization like CR so that you might compare their report with CR's to see if the two reasonably agree. That, as I see it, is a problem.

My wife purchased a couple of Maytag appliances because they were recommended by CR. We have had several problems with each. My bet is that there are other consumers out there like us who have had similar results with the same and/or other products.

Also, its my opinion that CR's methodology often includes purely subjective statements. For example, their vehicle reviews have statements like "plenty of pep" and/or things like "handles well." Unfortunately, there is no standard for what exactly is meant by statements like those, and IMHO, including such statements in their reviews "cheapens" their reviews. Those types of statements strike me as perhaps, just perhaps, one notch above vehicle manufacturer commercials that try to entice the "macho" male to buy a vehicle.

Their reliability data may be slightly better since it is collected from CR subscribers that respond to surveys, however, I think it could be even better if CR were to open up that reliability data collection so that people beyond their subscribers could provide input. (AFAIK, CR does not do this. If I'm wrong, please correct me.) Only collecting data from your subscribers who, presumably, only buy products that you recommend just might set up a catch-22 situation that skews the data toward one product or another. In one sense, it could be construed as a bias (though presumably unintended) towards particular products.

So, I suggest taking CR with a grain of salt. Unless another agency comes along that does the same or similar testing and data collection on similar products, there can be no comparison with another set of data and all the consumer has to go on is CR's word.

JimboK · 04-26-2007, 10:39 AM

<div class='quotetop'>QUOTE(PriusRos @ Apr 26 2007, 09:12 AM) [snapback]430411[/snapback]</div>

Quote:

I didn't even see the correction about the hybrid calculations -- what is the miscalculation you are referring to and is the correction in the same issue that analyzed the carseat test problems?
[/b]

The hybrid (mis)calculation and subsequent correction were in last years' issues. I think the original report was sometime in the spring and the correction a month or two later.

I don't remember the mistake exactly. I think it had something to do with double-counting the Prius' depreciation.

I agree with Wiyosaya, especially about the surveys. Survey respondents are not selected from among the overall population of users of a given product. There is inevitably a selection bias. Even among CR subscribers, they have introduced another potential source of bias. I recently received my annual survey, and beginning this year they are asking for a monetary donation to help fund the survey process. Sorry, but they get my money with my subscription. I didn't return the survey.

DaveinOlyWA · 04-26-2007, 11:47 AM

the hybrid correction, if i remember right was actually pointed out to CR by a Priuschat member. and the car seat correction should have been taken more seriously. the hybrid premium article was just a generalized article not pointed at one car manufacturer or model (they did make a mistake on the Prius)

because of that CR article, car seats have been a nearly impossible thing to buy. the top-rated snug ride is virtually un-buyable. we were able to get one, but cannot buy an additional base for it. (know of anyone who has one for sale??) we have been on a waiting list from the factory for over 3 months. we are now forced to move the base from car to car and its a hassle.

also the companies slammed in the carseat evaluation, some of them are suffering financial hardships, so ya, it should be taken MUCH more seriously.

PriusRos · 04-26-2007, 12:03 PM

<div class='quotetop'>QUOTE(JimboK @ Apr 26 2007, 10:39 AM) [snapback]430465[/snapback]</div>

Quote:

The hybrid (mis)calculation and subsequent correction were in last years' issues. I think the original report was sometime in the spring and the correction a month or two later.

I don't remember the mistake exactly. I think it had something to do with double-counting the Prius' depreciation.

[/b]

Okay, I seem to remember the original article (which came out before I bought my Prius) and it struck me that the depreciation was way overestimated. I don't remember seeing the correction.

<div class='quotetop'>QUOTE(DaveinOlyWA @ Apr 26 2007, 11:47 AM) [snapback]430523[/snapback]</div>

Quote:

the hybrid correction, if i remember right was actually pointed out to CR by a Priuschat member. and the car seat correction should have been taken more seriously. the hybrid premium article was just a generalized article not pointed at one car manufacturer or model (they did make a mistake on the Prius)

[/b]

I agree. The carseat issue is much more serious.

skruse · 04-26-2007, 12:18 PM

Consumer Reports is a starting, not an end, point. While CR strives to be objective, too often an "East Coast" bias comes through because they are not aware of similar products sold in different regions of North America.

The "frequency of repair" tables are an indicator, not an absolute.

04-25-2007, 07:59 PM	#1
micheal I feel pretty, oh so pretty. Join Date: Apr 2005 Location: Lubbock, TX Posts: 842 My Car: 2004 Prius Model: Package: #6 Thanks: 1 Thanked 0 Times in 0 Posts Friends: 0	I'm not sure if this is the best place or not, since this really isn't a specific news article about hybrids. I was afraid it would sucked in the black hole that is FHOP, and it is related to news about hybrids. Anyway, I just recently got a chance to look at my latest edition of Consumer Reports and was able to see all of their apologies and analysis of what went wrong with the Carseat Fiasco of 2006. I was curious if anyone else was a little irritated with the CR "correction" of the hybrid hype analysis and the CR correction of the carseats. I noticed both the letter to the reader and a several page description of what went wrong in their tests and reporting. Meanwhile, the correction for the hybrid hype shenanigans was I think only mentioned in the letter to the editors portion and the new graph put on the website along with an acknowledgment of error. I think the press release that CR first put out about the carseats did not even mention the hybrid miscalculation as one of their errors over the years. This could be an apples to oranges comparison, but it seems that CR is taking their error on carseats was much more serious than their reaction to the error for hybrids. Is anyone else seeing this or irritated by this situation? I wonder if the carseats being taken more seriously only because it was a safety issue or is there something more sinister in this situation? Whatever the reason, I think CR has lost a lot of their credibility with me, although I probably won't stop subscribing. __________________ "Miren al pepino. sus suaves movimientos tal como mantequilla en un chango pelon." ("Watch the cucumber. Oh, how smooth his motion like butter on a ... bald monkey.")

04-26-2007, 11:47 AM	#8
DaveinOlyWA 3rd Time was Solariffic!! Join Date: Apr 2004 Location: South Puget Sound, WA Posts: 12,378 My Car: 2010 Prius Model: IV Package: Solar Roof Thanks: 24 Thanked 121 Times in 103 Posts Friends: 10	the hybrid correction, if i remember right was actually pointed out to CR by a Priuschat member. and the car seat correction should have been taken more seriously. the hybrid premium article was just a generalized article not pointed at one car manufacturer or model (they did make a mistake on the Prius) because of that CR article, car seats have been a nearly impossible thing to buy. the top-rated snug ride is virtually un-buyable. we were able to get one, but cannot buy an additional base for it. (know of anyone who has one for sale??) we have been on a waiting list from the factory for over 3 months. we are now forced to move the base from car to car and its a hassle. also the companies slammed in the carseat evaluation, some of them are suffering financial hardships, so ya, it should be taken MUCH more seriously. __________________ My Blue 2010 : Last tank 541.9 @ 49.48 pump (56.7 MFD) 5.54 CPM, 21 MPH, Lifetime:5442.2 miles 56.5 MPG pump. (62.22 MFD) 4.72 CPM. Summer MPG 57.4 Winter: 49.5 My 2006 SPM: Last tank 376.6 miles @38.21 pump (40.8 MFD) 7.17 cpm winter mpg 49.64 summer mpg 53.41 lifetime: 42,563.5 miles 51.5 mpg pump (52.7 mpg MFD) 5.51 cpm My 2007 Zenn total "fuel cost" $166.58 on "about" 9599.7 miles. 1.74 cents per mile (granted i plug in for free at work!!) My Plate: DUALPWR (Dual Power)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Consumer Reports Admits Error	Tempus	Prius and Hybrid News	43	03-10-2006 09:55 PM
Consumer Reports article	Springtime	Gen II Prius Main Forum	20	03-07-2006 09:27 AM
Consumer Reports March 2006	JackDodge	Prius and Hybrid News	5	02-11-2006 05:01 PM
Consumer Reports...The most satisfying car is...	jaguaraja	Gen II Prius Main Forum	4	03-11-2005 02:55 PM
Have you filled out a consumer reports survey?	malorn	Prius and Hybrid News	0	12-31-1969 07:00 PM

04-25-2007, 08:09 PM	#2
JimboK One owner, low mileage Join Date: May 2006 Location: Chesterfield, VA Posts: 2,548 My Car: 2005 Prius Model: Package: #2 Thanks: 4 Thanked 88 Times in 69 Posts Friends: 6	I don't think it's anything sinister. I really do believe they try to be objective. My take is that they uncovered some significant methodological flaws with their car seat research, whereas the only (admitted) error with the hybrid report was with the math. Yes, their methodology there is open to scrutiny, but they thought -- and apparently still think -- it was good. So the response was probably appropriate to what they genuinely believe the seriousness of each issue to be.

04-26-2007, 06:13 AM	#3
Wildkow Senior Member Join Date: Jan 2006 Posts: 5,274 My Car: 2006 Prius Model: Package: #8 Thanks: 84 Thanked 36 Times in 33 Posts Friends: 11	Who knows but the car issue has a picture of the Prius in the box talking about reliability. Wildkow

04-26-2007, 10:32 AM	#6
Wiyosaya Senior Member Join Date: Mar 2006 Posts: 276 My Car: 2006 Prius Model: Package: #5 Thanks: 0 Thanked 0 Times in 0 Posts Friends: 0	Unfortunately, Consumer Reports is about the only consumer product testing agency out there, and as such, I see the potential for many mistakes like this. There almost certainly will be future instances like this. Product testing is a science, and as a science, product testing should have verification testing done. This is not the case with CR. You take CR's word for it, or you don't. There is no other organization like CR so that you might compare their report with CR's to see if the two reasonably agree. That, as I see it, is a problem. My wife purchased a couple of Maytag appliances because they were recommended by CR. We have had several problems with each. My bet is that there are other consumers out there like us who have had similar results with the same and/or other products. Also, its my opinion that CR's methodology often includes purely subjective statements. For example, their vehicle reviews have statements like "plenty of pep" and/or things like "handles well." Unfortunately, there is no standard for what exactly is meant by statements like those, and IMHO, including such statements in their reviews "cheapens" their reviews. Those types of statements strike me as perhaps, just perhaps, one notch above vehicle manufacturer commercials that try to entice the "macho" male to buy a vehicle. Their reliability data may be slightly better since it is collected from CR subscribers that respond to surveys, however, I think it could be even better if CR were to open up that reliability data collection so that people beyond their subscribers could provide input. (AFAIK, CR does not do this. If I'm wrong, please correct me.) Only collecting data from your subscribers who, presumably, only buy products that you recommend just might set up a catch-22 situation that skews the data toward one product or another. In one sense, it could be construed as a bias (though presumably unintended) towards particular products. So, I suggest taking CR with a grain of salt. Unless another agency comes along that does the same or similar testing and data collection on similar products, there can be no comparison with another set of data and all the consumer has to go on is CR's word.

04-26-2007, 12:18 PM	#10
skruse Senior Member Join Date: Apr 2004 Location: Great Central Valley, Fresno, CA Posts: 1,206 My Car: 2007 Prius Model: Package: #6 Touring Thanks: 6 Thanked 1 Time in 1 Post Friends: 2	Consumer Reports is a starting, not an end, point. While CR strives to be objective, too often an "East Coast" bias comes through because they are not aware of similar products sold in different regions of North America. The "frequency of repair" tables are an indicator, not an absolute.