"Gilbert Report" on accelerator weakness

Lot’s of technical analysis here. I did not read all of this thread, but one thing I do question. If the ETC failed in the sense that Mr. Gilbert said or as some of you have said that the circuit is fried in a way to cause the accelerator failure, would not that car still exhibit that symptom and Toyota service would find that problem and fix it. As far as I know that has never been the case. There might be something going on here in the real world, but I would find it difficult to believe that it would fix itself in some manner as indicated in the Gilbert method. It think if there is something intermittent in the ETC there would be transients (as the short happens) that would cause a failsafe mode.

 

If toyota actually tested the cars right after the incidents you would expect them to find something. I see numerous reports of Toyota NOT testing and refusing to provide owners with ERD output. These are part of the complaints against Toyota.

I have also seen a lot of handwaving by toyota experts saying they have ruled out electronics, and when pinned we have reports that some problems are software related. Given the public tests that toyota has described, I find it hard to believe that they have adequately tested the hardware and software.

 

This is one approach.....

Reality Check free online comic strip library at comics.com
<a href="http://comics.com/reality_check/2010-03-02/" title="Reality Check"><img src="http://c0389161.cdn.cloudfiles.rackspacecloud.com/dyn/str_strip/312070.full.gif" border="0" alt="Reality Check" /></a>

 

What I am saying is that according to the Gilbert scenario, the ETC would not just fix itself, before taking it to the dealer. It should continue to fail. The short or electronic circuit is not going to fix itself (the car would continue to accelerate). Dealers would not be saying they can't reproduce the problem. Therefore I believe it has to be something else. Could still be electronic, but not how Gilbert created it.

The issue with the ERD output is that no Toyota dealer can read it right now. There is only one prototype reader in existence in North America. There will be 100 beginning of April and about 150 at the end of April. Eventually they will be commercially available. The EDR standard does not even go into effect until 2012, so there was no requirement to have it yet.

 

OK. First, terminology. ETR means Electronic Throttle Control, which probably everything from the sole of your foot to the flapper on top of the engine.

What Gilbert found was a potential flaw in the way that the pedal to Engine Controller portion of this was error checked, and, further, an error in the architecture of the redundant pedal sensors.

Further, my speculation, based upon comments made by Gilbert during the House hearings, is that, depending upon how wires and chips are set up in the EC, there's a potential for a single point failure in the EC that would set up Gilbert's scenario, complete with a runaway car.

There are reasons both why and why not a potential fault might manifest itself in reality. Frankly, my opinion is that the probability of this fault actually existing is ridiculously low. However, one of the things one learns in field troubleshooting is dammed things do happen from time to time. In those cases of which I'm aware, those "dammed things" can often be traced back to some kind of other fault that got by, somehow. A phrase often used is "a chain is only as strong as the weakest link" - and, sometimes, it's not just one link that's gone bad, it's a bunch.

Yeah, one can get hardware intermittents. It happens. Repeatable hardware intermittents - that's rarer, but I've seen those, too. In general, blown parts are a lot more common.. But those tend to be easier to find and fix.

Are there higher probability things to go chasing? Sure: Software, funny stuff in the EC, lots.

Look at it this way: The whole business is a Sherlock Holmes problem. Right way to handle it: Leave your assumptions at the door. Listen to the clues. Try and weigh those clues with what you know, what you might think you know, and what other people know. Assign probabilities. Go for the low-hanging fruit. Make sure that you don't engage in wishful thinking. Try to cover all bases. It's hard work and it requires serious intellectual honesty.

Where we are here in this forum? Coming up with lots of hypotheses. That's required. Maybe there's one that people can test - heck, Prius Forums seems to be the place where the brake problem was first noted, identified, and bitched about, and there were a lot of hypotheses that got tossed (it wasn't there, user error, lawsuit happy people, etc. etc.) on the way. In the end, finding repeatable experiments that validated the flaw was probably dead useful for the Toyota people. Not to mention allowing the people who had experienced the thing to feel that they (a) weren't losing their minds and (b) had company. We, as end users, can possibly provide some useful input here. (But only if it doesn't get people killed - see a bit later on!)

About the only thing I've been able to think of (hypotheses, again) that might be the root cause of the problem has to do with the cruise control. The CC switch has direct input into the ETR, it's not duplicated (like the gas pedal), and is under almost complete software control (which is neither positive nor negative in the scheme of likely things to go bad, but what the hey.). Hey, maybe Smith hit the CC switch with her knee in a funny way on the way to the freeway and, crossing over 45 mph or something, triggered a software fault? Sure, why not? Besides, wasn't there some guy kvetching back in October of something that he got (at around 90 mph, playing, as he put it, with the CC) unwanted acceleration out of a Pruis?

Durned if I know. But, at this point, all doors are open.

KBeck.

 

Yes. I agree.

 

First, I'd like to say,that the Gilbert scenario is unlikely to be what is causing the problems.

But,... I would like to see honest investigations into the problems, and that includes testing and openess to hypothesis. Has toyota been testing for intermittent electrical problems? How have they been testing? Do they test crashed car electronics?

Simply saying that toyota has determined that its not electronics, leaves open the big question - What is causing all this unintended acceleration?

Then the next question is why toyota thought it does not need to let drivers look at the ERD. Did they think they are safer than all the American and european car manufacturers?

I would say they didn't provide it because they were affraid of liability. Now they are being forced to make this information available. Yes toyota will beat the deadline, but questions will remain until they provide this information? How can we verify toyotas reliability testing? I sincerely hope that it is because a drastic drop in incidents so that we don't need to analyze what is going on.

 

Confused now. ETC according to the Toyota webinar means Electronic Throttle Control. Actually they use ETCS the S standing for System. Where did ETR come from?

I might have remembered incorrectly but i thought they mentioned in the webinar that the CC as dual input and that it limits the amount of change that it accepts at any one time, so that a maximum throttle setting would not be allowed. But that does not mean there could not be an issue with any of it. I have been a software engineer long enough working on some really difficult problems that I know anything is possible. Complicated software is not perfect, ever. The hardest part to solving the problem is to have a scenario that will consistently create the issue. If that can be done then a fix can be worked out fairly quickly. This is assuming it is software issue, but it might be a combination of software and hardware, who knows.

I'm not sure how truthful Ms. Smith was. (I listened to all of these hearings). All of these reports are lacking in enough detail to figure out exactly what happened. This news article i read indicated that there were two sets of floor mats in the car. Also that the car had been sold to another person (3000 miles on it) and they have had no problems with it so far. :

http://priuschat.com/forums/prius-hybrid-news/77015-toyota-witch-hunt.html

I have downloaded the NHTSA complaints database to my computer. Currently I looked at only the Prius data. I searched the description field for 'acceleration' keyword and got a number of hits for all Toyota Prius years including for Gen. I models from 2001 and up. However the pertinent preliminary findings are:

1. Most of the entries were submitted recently since this issue was reported in the news. They were old occurrences but just reported now.

2. Most of the descriptions were rather poor, meaning: it could be caused by anything. Some did not even report a specific problem, just that the NHTSA needs to investigate these issues.

3. Many confused the unattended acceleration with the regen-brake to real brake transition, literally. Reported for other than the 2010 prius, even for Gen I. But all recent entries.

4. Some sounded like they hit the accelerator pedal by mistake. Example (paraphrased): "I was going 2 miles an hour into a parking spot, when the Prius accelerated. Hitting the brakes did nothing. I ended up crashing into parked cars." That sounds like a accelerator/brake pedal mixup to me.

5. One really compelling accident in 2006 reported in 2007 for a 2005 Prius. There might be others, I have not gone through them all yet.

I want to do the same for other Toyota models and other manufacture's to see if Toyota's have a bigger, problem than other manufactures, at least as reported in the NHTSA complaints database.

The hard part in going through the descriptions is to see if they are relevant to the acceleration problem. There is no field in the record that categorizes the data for you. You also have be sure when counting in that you don't double or triple count since the database contains duplicate records if they have been updated in anyway rather than just changing the fields. They give the new record a new id with a "original" reference tying them all together.

 

I'd be curious to hear about what other modern devices that handle
many analog inputs actually send those inputs to physically separate
chips, rather than bringing them to a typical MCU with several
PIOs so that the code can run around and read them. Show me an
automotive ECU that takes that sort of break from industry
standard practice, especially when it's using automotive-rated
parts.
.
_H*

 

Hokay. No objections to your hypothesis whatsoever. As regards Gilbert's..

I completely agree that if Gilbert's scenario requires (a) two independent faults and/or (b) a physical short with damage in a single device in the EC, then, if the runaway acceleration is caused by this, we'd be up to our keesters in failed cars. Which isn't happening.

Then, to make my hypothesis work, following the Sherlock Holmes model of What Would Have To Be Possible, we'd need to have a (1) transient fault that (2) hits the chip. Umm.. That is possible. There's this kind of hardware fault called CMOS latch-up. If too much current is put into or taken out of an input or output to above VCC or below ground, on a CMOS device, a buried intrinsic PNPN SCR transistor can get turned on. Once it's on, it stays on, until either things go blooey or power is removed. "Blooey" happens when there's enough voltage out there to drive enough current to melt or damage the silicon. If blooey doesn't happen, then Odd Things may happen inside the device, all depending upon the design of the device.

Ground faults can also cause latch-up on CMOS. Improper power sequencing (not likely here) where the different voltages in a system are turned on out of order can do it, too.

Oh, yeah. All commercial CMOS that I'm aware of have this "capability", it's a side-effect of the silicon process that builds CMOS wafers. But the amount of current (typically) is right up there in the 20 mA range. Bipolar (BJT or JFET technology) don't have this particular problem. This is one of the reasons why, in robust uncontrolled environments, it's generally not a good idea to connect CMOS parts to the I/O. Although it can be done, if one is very, very careful. CMOS has some advantages, though, especially when one is playing analog (no input bias currents, rail-to-rail operation, etc.), so sometimes one finds a CMOS part in a design where no CMOS part should have gone.

I believe that once upon a time Harris Semi used to make rad-hard CMOS parts with deep trenches and such that weren't subject to the problem. Don't know if that kind of stuff is still common.

To make this idea work the analog part in question would maybe have to have a medium sized capacitor on its input (to provide the current for the surge) along with a bobbling power supply so the internal diode(s) get forward biased too far. Or maybe the cap would be connected to a different ground than the chip, then the grounds diverge for some reason, that kind of thing. Finally, when the car is turned off, power to the device would also have to be removed, thus clearing the fault. A lot of what-ifs, any one of which would sink the idea without a trace.

Mind you, I still think that an electronic pedal fault is a very low runner. I like software, busted wires (hmm.. intermittent wires on ground?), or something nasty in the ECU better. But an electronic pedal fault is still a possibility.

KBeck.

 

That is exactly where I am on the thought process. The troubling part of the testimony to me was that there was no fault detected by the system.

For Hobbit, I do not know what standard industry practice is for automotive. Toyota may very well be following the standard practices, but I would prefer they would use best practices. I've worked on much more expensive systems, and our best practices were to optically isolate all our sensor inputs from each other and the rest of the electronics. I don't think toyota needs to go that far, and industry good practice is to override the throttle with the brakes. I am satisfied with that solution for the rare cases that there are run away throttles. At a minimum though Toyota should be logging error codes somewhere for unintended acceleration, which seems like standard practice.

When it come to the Gen III prius, I am comfortable with the braking response when pressing the throttle at the same time. I am also confortable with the ability to get the car quickly into neutral. That is unless someone has a scenario I have not seen. I am also a little troubled that people seem to think gilbert has been proved wrong without evidence.

 

You say that the gas pedal signal is not part of the loop. It is part of the loop as "desired throttle opening" (loop A).

Any "pedal error" is detected by its own separate feedback loop (let's call it loop "B") prior to sending "desired throttle opening" to feedback loop A. If there is an error in pedal feedback then "desired throttle opening" is to close throttle.

You say "If the resistance between the two duplicated signals gets to a relatively low value, the error checker over at the EC doesn't notice." This is a gilbert created short.

You say "Once that happens, the pedal position can be anything, and the EC thinks that's what the customer wants." Wrong, it cannot be anything, it has to be "desired throttle opening".

And what did gilbert do with "desired throttle opening"? He introduced a non-pedal voltage signal into the circuit changing the output voltage of loop A. You say "Full throttle forever?"--only if gilbert wants it full throttle forever. Watch his abcnews video where he does the demonstration. There is no pedal input to throttle level, he manipulates the "desired throttle opening" via a box held in his hands not a pedal under a driver's foot. He introduces voltage as full throttle open and when he's finished with it he removes the voltage and the car comes to a stop. Gilbert is manipulating the voltage to a feedback loop A that has been shorted.

 

The Associated Press: Toyota disputes critic who blames electronics

PDF copy of Exponent Analysis of Gilbert Demonstration
http://a.abcnews.go.com/images/Blotter/ht_exponentgilbert_100305.pdf

 

For those who didn't read the Exponent report:

They were able to create the same "malfunction" in all of the above cars without setting any codes using a similar technique to the one Gilbert used. In fact for the Mercedes, they used exactly the same resistance values as they did for the Avalon.

 

In other words, Exponent is catching up to the fact that we
100% debunked Gilbert right here a week and a half ago. He's
done, time to move on to the next cockamamie theory.
.
_H*

 

Yes, but they also made this cool graphic:

 

OK. Let's take this from the top.

Let's try a little terminology; it may be that you're misunderstanding what I'm saying.

There's a control system that starts at your foot and ends up torque applied to the wheels of the car. In ye olde time days the foot went to a pedal, the pedal went to a cable, the cable went to the thottle plate in a carburetor, the engine used the air/fuel mixture inserted through the carburetor, the engine burned the mixture and created torque on the flywheel, which worked its way down to the wheels.

OK. We still have the pedal. Next, we have two independent sensors on that pedal (no cable). The two, 1+1 error-checked signals from those sensors go straight into the ECU (Engine Control Unit).

I freely agree that there's two signals. I also agree that they're checked against each other. If they both say full throttle, then full throttle is what you get. If they both say no throttle, no throttle is what you get. If one says max and the other says something else, then the design intent is for the software to note that the signals are not the same and to Do Something Intelligent With Error Recovery. So, far, I think we still agree. But, at this point, we're not talking about a feedback loop. Protected signals, yes, but not a loop.

Now, let's talk about the feedback loop. So, assuming that we get past the error checking stage, I take a wild swing at what Toyota's done in the ECU and state that software calculates a throttle opening. Let's say, for example, that, for some reason, the ECU software decides that a 50% throttle plate opening would be a good idea. There's likely a huge amount of complexity in that, at least on a Prius: how much charge in the battery, whether the car's accelerating or not, engine temperature, etc., etc.

So, that 50% number likely exists strictly in software. Now, there may be a little filtering on the 50% number (we might not want to go from, say 10% to 50% in one clock tick), but then some message gets sent to the servo on the throttle body. Said servo then cranks itself open and, with a bit of luck, gets to 50%.

However, something so critical deserves an error check. So, on the side of the throttle body, there's a pair (two!) sensors, that send the measured throttle opening back to the computer. Assuming that the error check on the sensors passes (these are 1+1 detected sensors, rather like the gas pedal sensors), then the actual throttle opening is compared to the desired throttle opening inside the ECU.

This, then, is the loop: (calculated desired throttle opening in ECU) -> (filter in ECU) -> throttle servo -> throttle sensor -> (compared to desired throttle opening in ECU). It's a "fed back" signal, from the ECU, right back to the ECU. (So, "feedback" and "control loops" are very nearly the same thing.)

I want to be particular about this terminology. When engineers are talking about control loops, they are talking about loops like these. If things are protected, like the 1+1 signals from the gas pedal and the 1+1 signals from the throttle body sensor, the protection, itself, is not a loop, because, literally, we're not going around in circles.

In this loop, we start in the ECU and we end up back in the ECU, going "around the loop". I started this with an example of 50% opening desired: If the throttle sensor says, say, 30% or 80%, then, necessarily, the ECU software would throw an error and the driver and car saved.

Now, it's barely possible that the, I do not know, voltage out of the ECU to the throttle servo is fixed for whatever opening is desired. I highly doubt that. Instead, and what would be extremely typical in EE land, there would be a fair amount of gain in this control loop.

See the below picture:


Now for the math. E(s), the error signal, is the difference between the direct input of the loop and the feed back signal. Hence,

E(s) = X(s) - Y(s)*G(s)

But, since Y(s) is simply the amplified E(s) through F(s), we get

E(s) = X(s) - E(s)*F(s)*G(s)

which gets us to

E(s)*(1+F(s)*G(s)) = X(s)

which gets us to

E(s) = X(s)/(1+F(s)*G(s))

Since Y(s) = E(s)*F(s), then E(s) = Y(s)/F(s), and

Y(s)/F(s) = X(s)/(1+F(s)*G(s)), and

Y(s)/X(s) = F(s)/(1 + F(s)*G(s)), which is the standard control system function.

In this example, the "Y(s)" is the throttle flapper position. F(s) is whatever gain and filtering is in the ECU to the throttle sensor, and G(s) is whatever gain (if any) the protected sensors have back to the ECU. The subtraction is done in the ECU. Under normal operation, the closed-loop feedback system drives the E(s) term to zero, or near zero, depending upon the control loop parameters. Hence, an easy check to make is if the magnitude of the E(s) term exceeds some threshold - if it does, do error recovery and kill the engine. If you watch the webinar where Toyota talks about their electronic throttle control, it's pretty clear that when they talk about "control loop", it's this loop, ECU -> throttle servo ->throttle sensor -> ECU, that they're talking about.

Yeah, well, here's where we disagree. From my eyes, I don't see any loops around the pedal sensors. I see two independent sensors on the pedal assembly: I see them checked against each other in the ECU: But I sure don't see any feedback terms going backwards here. Hence, no loop.

What Gilbert noticed is that if he takes the two signals going to the ECU, connects a resistor between the two, then slowly decreases the value of the resistor, the detector in the ECU doesn't notice any problems, even if those two signals are within 0.02 V of each other. The signals are nominally 0.8 V different. If they do get truly shorted to each other (0V difference), then the ECU wakes up and makes a fuss, correctly. However, not-quite-zero-ohm resistances can happen.

Here we go. The "desired throttle opening" is not what the gas pedal is registering - it's the math function I mentioned between the pedal position and the X(s) in the above diagram.

Not quite right, here. The "desired throttle opening" is that math function, the complicated one that the Toyota engineers came up with that takes everything in creation into account, including the gas pedal position. But, under normal operation, the gas pedal position can and should be anything. The driver want it floored? The ECU sees a floored gas pedal and adjusts its desired throttle opening to the appropriate level. The driver takes his/her foot off the pedal? Then the ECU closes the throttle.

From this perspective, the gas pedal is not part of the control loop. It's part of the input to the control loop, but it's not in the ECU->servo->sensor->ECU loop. Hence, the loop doesn't check the gas pedal.

Now, Gilbert's doing experiments. He's got himself this resistor decade box and some jumpers. After looking through the repair manual and going, "Hmm..." he clips the resistor decade box across the two sensor leads and drops the resistance until the voltage hits 20 mV. No fault. Double, "Hmm..". Now, put another resistor to VCC, so there's a simultaneous near-short to VCC. Oh, yeah, the engine speeds up! And there's no error code!

He also does some tracing around of the signals inside the ECU. I heard him state, during testimony, that those analog signals went through one piece of silicon. Oops. In the hurly-burly of voltage transients, ESD hits, bad manufacturing, and Things Going Wrong silicon parts do have internal faults between pins and such, and a simultaneous short between pins and to VCC at the same time, in the presence of something gone wrong, would not be considered unusual.

Right thing to do: At least route those two pedal sensor analog signals to two different pieces of silicon. Better thing to do: make the slopes of the two sensors different, a la Honda/GM.

So, a flaw in the error detection architecture.

Do I think that this particular fault is likely? Nope. I still think software is a more likely culprit. Is it possible that one or more of the out-of-control Toyotas out there had this particular flaw? Sure.

KBeck.

 

I just read the thing.

They didn't report anything about the ECU, beyond saying "nice doggie" when they talked about it.

That's odd. Very extremely odd, since the ECU is the most likely place for something to go wrong with the reception of signals from the pedal assembly. I would have expected some analysis and/or statement saying that there was redundant silicon in there, or that no single-point fault in the ECU could cause unwanted acceleration without a DTC. Those statements, or something like them, are notable by their absence.

The graph is pretty, but is also very extremely odd. It's like somebody saying that, compared to the entire electromagnetic spectrum, we can see only one exceedingly thin slice, and showing the plot. Then implying that we can see only in black and white since we can't see very much, can we.

I'd say a lot more about this thing but right now I'm too upset about it to say something calmly.

I really, really want to be proven wrong about this "single silicon device" thing. Hobbit, did you ever get a good look in there with an ohmmeter?

KBeck.