SUDDEN UNINTENDED ACCELERATION (SUA)

what i'd like to know is, how were there so many cases of u/a in a short period of time, and then so few?

 

Must have been a great job! But what more can you measure about the throttle than the trottle position? And if that item is faulty and reports 100% when the throttle is not pressed, that is what is being recorded and being logged. So you read the logs and say "well, it says here the throttle was pressed completely. Who did that then?". The only other option is to check if the throttle position sensor is faulty, but if it is an intermittent fault, you might come to the conclusion that it works fine and that it has to be operator error.

I assume the Space Shuttle had some redundancy to rule out faulty sensors or even sensors to check for faulty sensors. I don't think our cars do.

 

First, let me say that I have no idea how any digital throttle is designed. Without knowing the strategy for how it works and how faults are discovered it is pure speculation.
But let's just say that the data for it varies from 0 to 255. I would design it so that full throttle, mechanically, only went to some value like 240. The first detectable fault is an error where the value suddenly spikes to 255 (all ones).
Second, I'd have two sensors and have the software detect a mismatch of more than some value as a fault.
Third I'd do something like have one sensor vary from 0 to 255 while the other went from 255 to 0 (and then invert it after being received in the CPU). All these values would need to be logged.
There are other intermediate values that could be logged. For example, the throttle has different sensitivities in the Prius when you are in ECO, normal and power mode. You'd want to log the input and output to this translation.
These are just ideas I came up with in two minutes. Far different than how, for example, a gamer joystick works because if it fails your drving game just crashes.
I did work on the joystick that controlled the large arm on the Shuttle (made by Canada...the Canadarm). We did check a number of things such as the rate of change of the commanded position before allowing it -- so you couldn't jerk the arm around too quickly. Probably don't want that in a car because you want to allow quick acceleration.

Mike

 

I hope my assumptions are faulty; I know the data is collected from several different ECUs across the CAN bus (interestingly if the CAN bus was cut at the moment of impact that might be a problem). The question (which nobody except Toyota knows) is what data is actually being recorded; I hope Toyota are using a good data recording method but I also know that no car manufacturers have to follow aerospace standards (let alone NASA standards!)

Post #95 by Subjective shows that he got this data from Bosch (pre-crash data):
Timestamp Speed Brake Switch Accelerator rate. Engine RPM
-4.4sec 1.2mph ON 0.78 1200
-3.4 sec 1.2mph ON 0.78 (?) (not stated)
-2.4 sec (not stated). ON 0.78 (?). (not stated)
-1.4 sec (not stated). OFF (not stated) (not stated)
-0.4 sec (not stated). OFF (not stated) (not stated)
0 sec 3.7mph ON 0.78 1200

This suggests that Bosch makes the recording unit (possibly); as opposed to Denso who make most (?all) the ECUs.

What is interesting is that there seems to be some missing data. The '?' is where he stated it was 0.78 later in the post but didn't last it in the raw data. He stated that at -1.4 and 0.4 seconds the accelerator rate wasn't recorded.

From this data, it may be that the unit is not recording all the data at each 1-second sample point - or it may just be it wasn't typed into the post.

The other concerning thing would be the accelerator rate being 0.78 most or all of the way through - and the engine speed being the same. If the accelerator rate is the gas pedal, then it would mean his wife was holding it very carefully (and not holding it down fully either) with one foot, and managing to keep the gas pedal held perfectly still while pumping the brake with the other foot. The engine RPM being 1200 at the start and end is odd (given it is exactly the same and not e.g. 1197 then 1205). This could be something as simple as this RPM being stored in a byte and thus having lower resolution.

It would be good to know the actual algorithm that captures the pre-crash data. I will assume from Subjective's data that the car keeps 5 seconds of data (for a total of 6 entries - 5 entries, and an extra one at the point of impact, airbag activation, or whatever).

There are some ways this could be done:

Algorithm 1
REPEAT
'Send requests to each ECU for the data points (speed, brake switch, accelerator rate, engine rpm)'
'Record the results into a block of memory' 'Record 'NO ANSWER' if no answer received'
'Store that block of memory into a rolling 6-entry buffer (like a very short flight data recorder)
'Wait one second, then repeat'
IF (crash detected) then : Send another request, collect all the data, add to rolling buffer, record it into EDR, then STOP
UNTIL FALSE

This algorithm would be nice in that if no reply was received from e.g. the throttle control unit, then a 'NO DATA RECEIVED' would be recorded in the EDR. The quality of the data the WAS received, however, still depends on the algorithms in whichever ECU / code is sending that data. This algorithm also involves having to wait for a second, and would need a dedicated chip to do it and so on. Not very efficient use of CPU time etc.

However, another way this could be done is as follows:

Algorithm 2

Have a 5-entry buffer (one entry contains all the data points being recorded) in the RAM of the EDR module.
Have a 'current' 5-entry buffer in RAM, where new data from the ECUs is stored temporarily
Have a 5+1 entry Flash / non-volatile RAM buffer in the EDR to store the data when a crash occurs.

(Set up a timer to send an interrupt once every second)
(Set up an interrupt that occurs when an actual crash is detected)
(Set up interrupts for when data is received from the network)

WHEN timer-based interrupt is received (once per second), THEN:
'Store the block of memory (containing data points of speed, brake switch, accelerator rate, engine rpm) into the rolling buffer in RAM (5 entries?)
'Send out a request for the relevant ECUs to send updated speed, brake switch, accelerator rate, engine rpm'
[END of code for timer-based one-second interrupt]

WHEN crash-detection interrupt is received, THEN:
'Store the block of memory (containing data points) into the rolling buffer (in normal RAM)
'Send out a request for the relevant ECUs to send updated data points'
'Store the 5-entry rolling buffer into the EDR non-volatile RAM'
'Wait a certain time period for replies from the relevant ECUs'
'Store this new, final, updated data into the 'final' entry in the EDR non-volatile RAM'
[END of code for 'crash' interrupt]

WHEN 'data-received-from-network' interrupt received, THEN:
'Read the network input buffer. Determine the data type, and data value'
'Store that value into the temporary RAM-based buffer' (not the rolling buffer - that update is done once per second).
[END of code for 'data-received-from-network' interrupt]

The second algorithm would allow the CPU to be doing all the other tasks it needs to do. The tasks that need to be done for each interrupt would take miniscule amounts of time. The second algorithm could also explain why Subjective's data is -x.4 seconds all the way through, but the final data entry is at zero seconds, because the data was captured and stored in a rolling buffer prior to the crash, but the actual crash occurred 0.6 seconds before the next 'standard' data capture event.

If the second algorithm was used, and if it relied upon interrupts for events where data was received from the network, then, if no reply was received from a particular ECU, it is possible that the previous entry could be 'carried forward' - given the old entry might still be in the temporary buffer. That old entry would then get stored into the rolling buffer, then into the EDR.
- This would mean, for example, that if whatever process is responsible for reporting the accelerator rate stopped working, it is possible that the accelerator rate could be recorded as staying in a certain position even if that was not the case, as no more data was being received.

In Subjective's example, if whatever components were responsible for sending the 'accelerator rate and 'engine RPM' back to the EDR module were no longer responding to those requests, then (depending on the exact code within the EDR module), you could end up with constant values being recorded throughout the whole event.

All of this is, of course, my subjective opinion. However it would be very interesting to know how the algorithm works, and *where* the actual EDR is physically located (I bet this is a very closely guarded secret, but it is accessible via the OBD port; it will be hidden somewhere in an ECU!)

If Subjective is reading this, it would be cool if you could update the table I made above, to specify if there are actual missing data points in the data they sent to you?

Mike - designing for NASA - much respect - I cannot imagine the attention to detail and level of skill needed!

 

I bet it was the floor mats! (Also, it would be interesting to see the percentage of vehicles affected during the spike, and compare that to the legal situation in the country; in New Zealand you couldn't sue Toyota, for example, in the US you could become very rich very quickly).

 

funny thing about floor mats, the first major incident was a loner lexus. the dealer had put two non lexus floor mats on top of the oem. the driver floored the pedal and it caught on top of the mats.
an analysis by paralysis revealed that the oem mat could ride up under the pedal because it was not clipped in.
of course, this had nothing to do with the accident.

 

The key to a fool proof algorithm is to eliminate virtually every logic statement in your pseudo code. Again, I have no idea how any modern system like this works. The inner loop of the telemetry logging system I built was, literally, something less than a dozen 8086 assembly language instructions.

Load a pointer to some memory where a variable of interest is stored
Read the data
write the data
increment pointers
decrement loop counter
loop​

The key is that the above logging always works (assuming the hardware is working) at a guaranteed speed
Numerous other threads (actually just interrupt driven events) deposit data in their working variables as the normal course of what they do. And each event sets or increments bread crumb-type info as it does things.
Then by analyzing the telemetry stream you can see the state of various processes and the values of variables they were using. (You had to disallow local stack variables for important data that had to be saved in a global memory space.)
The key was the tool used to create the list of pointers to get the data desired. You could repeat any pointer at whatever frequency you wanted to get the repeat rate you wanted.

In any case, there are numerous ways to do it...and with processors 1000x the clock speed today you can do things much more sloppy and still get away with it.

Mike

 

Your pseudocode is perfect. What annoys me today is that, while processors have hugely increased clock speed (and also, usually 1 cycle per instruction), we are much more sloppy in our code. I believe that NASA and also Boeing / Airbus etc use this kind of pseudocode, but, I have read articles that suggest that many car ECUs run an 'OS' that is multitasking (pre-emptive). I first programmed on a ZX81 (I confess that it *DID* have the 16K ram expansion pack); my first proper learning of machine code was RISC ARM26 in the 1980s (Acorn, ARM2, ARM250, ARM3 CPUs). I am very glad that there has been a division between high-quality code (aeroplanes and above) and 'normal' code (amazing computer games) - I saw a passenger 747 crash aftermath when I was young. It would be nice it auto manufacturers used code that was tested to perfection (I can find many quotes that Toyota code was perfect, but also direct quotes that analysis showed coding flaws that could cause SUA events). In New Zealand, the government has been asked whether to ban Nissan Leafs from the road (I have one in addition to my Prius etc) due to the issues with the braking ECU).

I wish you could be employed by car manufacturers to write the mission-critical code portions - but I know, with much faster CPUs, assembly language is old-fashioned - much higher level languages are good enough - but higher level language programming can result in compiled code which misses 'rare' conditions. There is one road I drive down where there is a 'Police Speed Camera' shortly after a manhole cover (fairly steep downhill in an urban setting). I manage to gain about 8 km/h after hitting that manhole due to brake failure, but have not tickets as I just go to 40km/h before the manhole, reaching 48 km/h and thus avoiding the 50km/h camera. I am sure the braking algorithm could be better; I pay very careful attention, because if a child ran out from behind a bush just as my tyre hit the manhole cover, my brakes would barely work.

 

Make that "throttle position sensors", plural. My memory of one of the final reports was that Toyota's throttle-by-wire system did use two sensors, not one. When their readings did not match closely enough, an error would be flagged.

The logging system should also record something about the brakes, independent of throttle reading. If the driver was pressing on the brake at the same time an erroneous full throttle reading was occurring, that should show up in the log. And the engine management system should note the apparent conflict and give precedence to the brake. So a later investigator ought not simply dismiss this single-fault case as "it works fine and that it has to be operator error. " It should require a multi-fault event to create that appearance.

 

The report by NASA gave a lot of interesting details on how that stuff works, even more than are in the New Car Features manual.

The NASA report wasn't the last word, and should be read in conjunction with the later Barr report, which had access to information NASA did not. NASA, for example, had been led to believe the ECU used error-detecting memory, and drew conclusions assuming that.

One of the Barr findings was that certain of the logging and monitoring responsibilities were assigned to the same ECU task that would, if it hung, cause the problem in the first place.

 

The NECV850E2v3 RISC microprocessor brought a nice feature where the actual microprocessor does time-scheduling between multiple processes, and no matter what happens (e.g. if one process locks up), it will still keep the others running.

However, it was released in 2009, so that means specifically that the Gen 2 Prius cannot have that luxury, and the Gen 3 *might* if they were very lucky!

 

Still not much help if the assignment of responsibilities to tasks is such that one task can hang that is responsible for mapping your go-pedal input to throttle movement and detecting that you're trying to override with the brake and logging the corresponding trouble codes....

 

I hadn't got round to looking this up, but
http://www.safetyresearch.net/Library/Bookout_v_Toyota_Barr_REDACTED.pdf
An Update on Toyota and Unintended Acceleration « Barr Code

Thank you for pointing this out; reading the PDF now; P49 has an awesome quote (I haven't read further yet);
19 Q My question goes back to this: Is there any software
20 that you're aware of that does not have bugs?
21 A No.

Again, I am not dissing Toyota; it is so true that all software will always have bugs! No software will ever be free of bugs. In the example of the Prius, there is the chance of bugs killing you while driving the car (my Leaf is much more dangerous though), but non-hybrids are also susceptible, and the only solution is to go back to more basic designs. If I wanted to be safe I could drive my 200SX every day. It eats fuel so badly, but, it has an ECU that controls the fuel injectors. The throttle is manual (hopefully it doesn't stick one day). Other than the throttle wire becoming stuck, my 200SX is a much safer car. Very little software. But if we all drive it, we all get to die faster due to climate change

(On a side note, my UK 200SX that I bought was the second one I was going to buy; I agreed to buy the first one, but to come down the next weekend with cash and drive it back up). This never transpired; the owner was driving in a 30mph zone in a densely populated area, the throttle stuck and the upgraded turbo went to full boost. It was rear wheel drive. Slamming on the brakes helped except that the front brakes are more powerful, and it is rear wheel drive. The brakes probably didn't do much without vacuum either. This car was powerful and got to 60mph in maybe 4 seconds; during the test drive the week before as a passenger it literally induced fear due to acceleration (I was young). It ended up going backwards through a brick wall then a brick house wall and into someone's (empty) living room. He didn't get in trouble as it was a random event, but it was a SUA event caused by purely mechanical means.

I seriously think I will add a kill switch onto my Prius and Leaf (all my other cars are very old). A fuel pump kill switch, combined with a kill-switch for the 12v controller for the hybrid system (or the equivalent in the leaf) on the dash could be ready for use. It would probably cause lots of codes, but it would help in an emergency.
-> However, the reality is that this is ridiculously paranoid as the risk of a problem is so extremely low. The main risk is of falling over between your front door and your car and dying of a head injury, or of another driver crashing into you. I will still make a kill-switch for my mum's Prius however

 

This. There are all kinds of things that have improved safety since the old 200SX, including a lot of the stuff the software does. I'm not sure I'd be able to say the 200SX was really a safer car.

The difference is today bad things can happen that seem freakishly random and out of the blue (such being the nature of software bugs) and nobody feels comfortable with that, but it requires a crazy specific confluence of conditions to happen, and ends up being very rare compared to a lot of other things one might worry about.

 

then you throw in gas pedals that are too long, mats that slide up, and people who have their foot on the gas thinking it's the brake. what's an investigator to do?

 

It may be safer from software bugs, but that doesn't make it safer overall. The mechanical faults in the old systems typically turned dangerous more frequently than did the firmware bugs in their replacements.

But there remains plenty of room for improvement ...

 

The first 200SX I nearly bought went backwards at 70mph into a house due to the throttle cable getting stuck, a few days before I was going to buy it! It was showroom condition, beautifully maintained too! If it hadn't spun it would have probably been a fatal 'SUA' and more extreme than many of the SUAs people talk abut today!

 

Stick or slushbox?

I've had my own pilot-induced SUA experience under extreme fatigue. But it was in a manual transmission vehicle, and a reflexive jab at the clutch was one of the things that saved the moment. So if that car was a stick version, a regular driver of manual transmissions might have saved at least the body.

The engine might have been a different matter. But I have been present at multiple incidents with farm machinery where the operator had the presence of mind to immediately shut off the ignition before an initial problem escalated to any collateral damage.

 

When my Gen 2 was reprogrammed after all the stuck throttles one of the key things they did was program it so that the brakes had priority over the throttle. If the cars gas pedal was stuck to the floor and you applied the brakes I believe all fuel was cut off and the car could be stopped with the brakes.
All modern cars have redundant TPS sensors in them, some have two and some have three and the ECU is constantly comparing them to make sure they are in agreement or the throttle is put into a limp in mode.

 

Constantly comparing them, I'm sure Michael Barr would add, at all times when the ECU task responsible for doing that hasn't stopped.