1. Attachments are working again! Check out this thread for more details and to report any other bugs.

Capturing data from CAN-View

Discussion in 'Gen 2 Prius Audio and Electronics' started by roperld, Mar 29, 2006.

  1. roperld

    roperld Junior Member

    Joined:
    Sep 4, 2005
    30
    2
    0
    Location:
    Blacksburg, Virginia
    Instructions for capturing data from the serial port of CAN-View:

    If your laptop/tablet-pc has only USB ports, you will need to purchase a USB to serial adapter. When it installs, note the com port number it is using.

    Download and install the free RealTerm program (http://realterm.sourceforge.net).
    After installing it, create a shortcut for it on your desktop as follows:
    "C:\Program Files\BEL\Realterm\realterm.exe" port=5 baud=115200 display=2 capfile=C:\Prius\test
    Here I used com port 5 and a capture file name of C:\Priue\test ; you may need to change the com port number. (display=2 means hex)

    For serial setup on the Prius CAN-View, set "Data source" to RAW .

    Click on the RealTerm shortcut to start the RealTerm program with the parameters set in the shortcut.
    Then click on Capture. If you want to see in the terminal window the data being captured, uncheck the Direct Capture box.
    Then click on Start Overwrite or Start Append, depending on what you want to do.

    Then, to actually start the capture, touch the "Output is" button on the MFD to change it from OFF to ON.

    Drive around to capture data. Then to quit capturing data, touch the "Output is" button on the MFD to change it from ON to OFF.

    Then click on Stop Capture on the RealTerm window.

    A sample output file is at http://www.roperld.com/science/prius/raw

    You can use some program such as UltraEdit to view the hex data.

    Now we need a program to massage the data.

    -Dave Roper
     
  2. Presto

    Presto Has his homepage set to PC

    Joined:
    Mar 27, 2005
    1,326
    24
    0
    Location:
    Vancouver, BC
    Vehicle:
    2005 Prius
    Wow! Thanks for the info, Roper! Exactly what i'm lookin for, and I'm sure others as well. As you mentioned, now we need a program to process the data.
     
  3. roperld

    roperld Junior Member

    Joined:
    Sep 4, 2005
    30
    2
    0
    Location:
    Blacksburg, Virginia
    <div class='quotetop'>QUOTE(Presto @ Mar 29 2006, 09:12 PM) [snapback]231992[/snapback]</div>
    Glad you are pleased. It took me about two weeks of help from Norm of CAN-View and from two other persons to figure all that out. I believe in sharing things that interest me.

    I think one or two people are already working on programs to massage the data. First may be a conversion of the binary data to cvs to be loaded into Excel. Then one can use Excel to plot parameters versus time.

    It would be nice to have several programmers compete to create the best programs to massage and plot the data.

    My goal is to be able to plot various hybrid system parameters versus time on the same graph, so that one can study how they are related.

    I plan to capture a file soon for a longer trip up and down our Virginia Allegheny mountains. I will put it on the web and post the url here.

    By the way I just completed teaching a 3-hour course on hybrid vehicles for the Open University of the YMCA at Virginia Tech. Any one interested in the huge PowerPoint slide show can contact me for it on a CD ROM. I would also include a rough draft of a book I am writing entitled "Triple Threats for the Human Future". Contact me from the members list.

    Dave Roper
     
  4. frenchie

    frenchie Member

    Joined:
    Oct 19, 2004
    150
    48
    0
    Location:
    France, not far from PARIS
    Vehicle:
    Other Hybrid
    Model:
    N/A
    <div class='quotetop'>QUOTE(roperld @ Mar 29 2006, 10:58 PM) [snapback]232144[/snapback]</div>
    hello Dave thanks for the link.
    I used seriallogger to collect data but realterm is free.
    It is easy to read the log file from excel and to convert the data in hex or dec.
    here is the code I wrote some time ago for interpreting the AVCLAN codes
    in this code everybody can find the basic tricks to read a log file and put the interesting data in the right columns. after that, it is easy to put them on graphs.
     
  5. roperld

    roperld Junior Member

    Joined:
    Sep 4, 2005
    30
    2
    0
    Location:
    Blacksburg, Virginia
    <div class='quotetop'>QUOTE(frenchie @ Mar 30 2006, 03:05 AM) [snapback]232236[/snapback]</div>
    Frenchie,

    Great! I downloaded your file into Excel. A few questions:

    1. Are those data the data I collected or data you collected?
    2. If my data, how did you get it into Excel?
    3. Please give some explicit instructions on how to manipulate the data in Excel to plot hybrid parameters versus time.
    4. What is AVCLAN?

    Dave
     
  6. frenchie

    frenchie Member

    Joined:
    Oct 19, 2004
    150
    48
    0
    Location:
    France, not far from PARIS
    Vehicle:
    Other Hybrid
    Model:
    N/A
    <div class='quotetop'>QUOTE(roperld @ Mar 30 2006, 07:26 AM) [snapback]232268[/snapback]</div>
    1) these are some data I collected for analysing the AVCLAN (Audio Video) to help norm understanding how the Nav Unit "speaks" to the MFD.
    2) no object
    3) I will come back later with an operational solution to show on the same graph MG2 rpm (speed) /ICE rpm / MG1 rpm & some other parameters. (no time before sunday!!!)
    the idea is to collect the data during a trip and later on to show on graphs how they interact.

    the macro I sent is be used by people with a good knowledge of excel/vba.
     
  7. kk6yb

    kk6yb Junior Member

    Joined:
    Oct 24, 2005
    36
    0
    0
    Location:
    San Diego, CA
    Vehicle:
    2011 Prius Alpha
    Model:
    Plug-in Base
    <div class='quotetop'>QUOTE(roperld @ Mar 29 2006, 02:06 PM) [snapback]231838[/snapback]</div>
    I have been working with Dave for a few weeks on this and I now have a program that is available to convert the data.

    My CAN-View decoder program reads the binary file that RealTerm created. It must contain RAW data (not formatted) from CAN-View. The program decodes 28 of the most interesting fields from the CAN packets and creates a .csv file with the data. You can then load the CSV file (here's a sample) into a spreadsheet program (such as Excel) and create graphs and do analysis and play with the data.

    The following web page has a description of the program, download links, and descriptions of the fields that are output. The program is free and the source code is available for download as well.

    CAN-View Decoder Program Web Page

    Suggestions and comments are welcome. I'm not a windows programmer (I work on embedded systems) so I didn't do a Windows user interface, but the program works.

    --GaryM
     
  8. JimboK

    JimboK One owner, low mileage

    Joined:
    May 1, 2006
    2,817
    187
    49
    Location:
    Chesterfield, VA
    Vehicle:
    2005 Prius
    Dave, Gary, or anyone else that can help a "techno-peasant" with data capture: :)

    I've tried Dave's instructions with the Realterm application and Gary's decoder for my new Can-view, and I'm having some problems. Reader's Digest version: After (presumably) logging data for 30+ minutes, the resulting CSV file has only 12 lines, with several blank columns and other columns with obviously invalid data (e.g., battery temp of 224C).

    The long version:

    No problems downloading and installing the Realterm application. I got the shortcut configured with the parameters as Dave describes. Realterm opened with no problem. My first attempt at capturing data generated an I/O error 103 (or something to that effect). A quick web search revealed that error is often related to not being to find or open a file, so I created the Prius folder on C: drive and ... voila! The error went away. So far, so good. Woohoo!

    After giving Can-view the MFD command to start sending data, I started seeing what I assume are binary characters (i.e., symbols and gibberish) in the Realterm display window. I clicked the Start:Overwrite button and, interestingly, the characters stopped scrolling. But then the character counter in the lower left corner started. I figured that whether data was displayed or written to a file was an either/or thing. I rode for a half hour or more and then clicked Stop Capture. The character count was in the 350K range.

    I later ran the Can Decoder application, with the expected prompt within the DOS window to enter the input file. I entered c:\prius\test and <enter>, whereupon multiple lines rapidly displayed and the window closed, all within a fraction of a second. Upon opening the resulting CSV file, the results were as described above. I re-ran the Can Decoder repeatedly to try to visually capture some of the text in that briefly appearing DOS window. Partial results (I think): "**** Warning, bad data, field out of range" followed by what seems to be a different field name on each line.

    Any suggestions? Thanks in advance!!!
     
  9. D0li0

    D0li0 New Member

    Joined:
    Oct 1, 2005
    118
    0
    0
    Location:
    Seattle, Wa
    <div class='quotetop'>QUOTE(JimboK @ Jun 5 2007, 03:03 PM) [snapback]456025[/snapback]</div>
    When you start recording it does stop displaying the characters, that's normal.
    Though you should not see "gibberish" when not recording, you probably have the character set wrong.
    Use the HEX (space) setting, there are multiple HEX settings, one of them should result in something that looks like this:

    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 34 56 76 H4
    A8 B3 C8 00 00 00 00 00 00 34 56 76 H4 A8 B3 C8 00 00 00 00 00
    00 00 00 34 56 76 H4 A8 B3 C8 00 00 00 00 00 00 00 00 00 00 00
    00 00 00 00 00 More numbers and letters from A-F 00 00 00 00 00
    00 00 00 00 00 the 0-9,A-F is "HEX" a base-16 number system 00
    00 00 etc etc scrolling down or up the screen...

    This should start and stop scrolling as you tell the CAN-View to start and stop sending data....
    Once you see this stuff you're ready to try recording and all should be well...
    This is what the display=2 setting is supposed to do, perhaps it's not correct?

    ps. Thank you Gary and Dave for the decoder app, and it's source! I'm thinking that it would be swell to setup a web script at the eaa-phev wiki site ( and elsewhere ) such that someone could create an "article" containing the raw hex and get it parsed and outputted to a table and multiple types of graphs, etc... As you can see from my graphs each one is different, so this is a project that I hope to get to work on sometime...
     
  10. JimboK

    JimboK One owner, low mileage

    Joined:
    May 1, 2006
    2,817
    187
    49
    Location:
    Chesterfield, VA
    Vehicle:
    2005 Prius
    <div class='quotetop'>QUOTE(D0li0 @ Jun 8 2007, 08:39 PM) [snapback]458379[/snapback]</div>
    Ryan, thank you very much for your detailed response. I haven't had time to do any more troubleshooting, though I did hear back from Gary Morris by e-mail. He was gracious enough to have me send him my raw binary file and the output file from Realterm so he could peek at them. His conclusion: "So you are not getting a good capture of data, it's either the data rate is mismatched or the capture program isn't working, or the cable, or the CAN-View is not in the right mode."

    It probably boils down to a common problem I have with computers: a loose screw about two feet above the keyboard. I'll keep working on it.
     
  11. JimboK

    JimboK One owner, low mileage

    Joined:
    May 1, 2006
    2,817
    187
    49
    Location:
    Chesterfield, VA
    Vehicle:
    2005 Prius
    OK, we're making headway. Woohoo!

    Ryan, I had the Hex[space] setting selected before your post. Interestingly, what I've since discovered is that if I select one of the other "Display As" options, then go back to Hex[space], I now see the hex values as you describe. That alone didn't work, but at least the display behaved as you describe. I have to repeat that routine each time I open Realterm; else I see the characters and gibberish I first described.

    Based on Gary's and Ryan's responses, I double-checked the settings and data rates; they were set properly. I've been using an old serial cable, so with the possibility that it could be bad I bought another. Meanwhile, I had some e-mail dialog with Norm to help verify the correct Realterm settings. His response: "You should be setting 8 data bits, parity, 1 stop bit (with parity as odd/even/space whatever, anything as long as it knows there is that 9th bit)." My settings matched all that except parity, which I had set to None.

    So I changed that, plugged in the new cable, pushed all the buttons, and held my breath. The result: good data! Double woohoo!!!

    Some lingering questions, if anyone has answers. Norm's site says, "The RAW message dump is a contiguous 58mS burst of (currently) 55 CAN packets each of 11 bytes, roughly once per second." From my brief driveway testing this evening, the resulting CSV files have somewhat less than 1 line per second, maybe 40-45 per minute. Does this seem about right? Can I assume each line represents one data burst from CAN? For later analysis of events that I'd like to put a precise time stamp on, is the time between each data burst exactly the same? The wording "roughly once per second" could be taken to mean it doesn't.

    Thanks again to all! This is going to be fun once I start logging real world data!

    As an aside, Norm tells me his father recently died. I know many of you think highly of him -- heck, I do without having met him -- so I thought you'd want to know.
     
  12. efusco

    efusco Moderator Emeritus
    Staff Member

    Joined:
    Nov 26, 2003
    19,891
    1,193
    9
    Location:
    Nixa, MO
    Vehicle:
    2004 Prius
    Model:
    N/A
    <div class='quotetop'>QUOTE(JimboK @ Jun 15 2007, 06:36 PM) [snapback]462663[/snapback]</div>
    Congrats on getting things working...and thanks for posting up here as I suspect some others will need to know this too. I've got an old lap top I don't really use that sort of works...might be fun to do some logging just for the hell of it.


    Very sorry to hear about Norm's dad.
     
  13. dmckinstry

    dmckinstry New Member

    Joined:
    May 4, 2006
    1,034
    4
    0
    Location:
    Cheney, WA (Near Spokane)
    Vehicle:
    2005 Prius
    <div class='quotetop'>QUOTE(JimboK @ Jun 15 2007, 03:36 PM) [snapback]462663[/snapback]</div>
    I'm sorry to hear that. I know that his father was going to have some testing done several weeks ago, but hadn't heard any more. Thanks for letting us know.

    Dave M.
     
  14. TheForce

    TheForce Stop War! Lets Rave! Make Love!

    Joined:
    May 30, 2005
    3,461
    537
    0
    Location:
    Wheelersburg, Ohio
    Vehicle:
    2006 Prius
    Model:
    N/A
    Well I wanted to start capturing data with my Can-View and I thought it was working before but I guess I thought wrong.

    After trying to capture data the program to turn it into a cvs only converts 99 lines. Also when doing a short capture of say about two minutes there are only about 10 lines or so of data.

    The realterm program shows the hex before I start the overwrite but I noticed that the break and error lights are always on or the error will flash ever two or so seconds. Also when I tried to uncheck the direct capture box during capture it will display garbage instead of hex as well as give a warning type sound every second the packet of information is received from the Can-View. I have realterm setup just like the tutorial but cant seem to figure out what the problem is.

    Does anyone have any ideas.
     
  15. mrbigh

    mrbigh Prius Absolutum Dominium

    Joined:
    Sep 6, 2005
    3,686
    699
    2
    Location:
    Long Island, NY
    Vehicle:
    Other Electric Vehicle
    <div class='quotetop'>QUOTE(TheForce @ Oct 17 2007, 04:50 PM) [snapback]527071[/snapback]</div>
    Jay, the symptoms that you describe with the data capture are not the first time publish here, it has been another thread referring to the same problem, for what I can remember, are the settings of the capture program, it been a few months back, do a deep search in PC.
     
  16. TheForce

    TheForce Stop War! Lets Rave! Make Love!

    Joined:
    May 30, 2005
    3,461
    537
    0
    Location:
    Wheelersburg, Ohio
    Vehicle:
    2006 Prius
    Model:
    N/A
    Well I cant find it and I used all the search terms I know to use on both PC and Google. Thats strange because I'm pretty good at finding stuff. I guess it will be lost until I stumble on it or someone points it out to me.

    Can you tell me if its normal for the break and error lights to be on? No matter what I try I can only get 1 line if I capture for 10 seconds. Shouldn't I be getting 1 line of data for every second?
     
  17. mrbigh

    mrbigh Prius Absolutum Dominium

    Joined:
    Sep 6, 2005
    3,686
    699
    2
    Location:
    Long Island, NY
    Vehicle:
    Other Electric Vehicle
    <div class='quotetop'>QUOTE(TheForce @ Oct 17 2007, 07:27 PM) [snapback]527120[/snapback]</div>
    The LEDs should be like dancing while at work.

    This was the thread, read it from the beginning and you may find the answer ; dOliO had a good pointer. Have you replaced your serial cable? Hope this helps
     
  18. TheForce

    TheForce Stop War! Lets Rave! Make Love!

    Joined:
    May 30, 2005
    3,461
    537
    0
    Location:
    Wheelersburg, Ohio
    Vehicle:
    2006 Prius
    Model:
    N/A
    I thought you might have pointed me to this thread but since I tried what was here I was still searching elsewhere.

    Yup replaced the serial cable as well as trying the different parity settings. Maybe I have two bad serial cables? I'll have to see if I can find another one.

    The thing is is that I can see the correct hex as it comes in but when I record I don't seem to get all the data. I could record for 30 seconds and then convert it to cvs and it will only have a few lines of information. Shouldn't it have one line of data for every second of recorded data? And if not why would it only convert 99 lines of data and quit?
     
  19. JimboK

    JimboK One owner, low mileage

    Joined:
    May 1, 2006
    2,817
    187
    49
    Location:
    Chesterfield, VA
    Vehicle:
    2005 Prius
    Jay, I'm sorta glad you're having the problem. I am too. Makes me feel a little less inferior that someone with your technical smarts isn't able to get it working. :p

    Ryan (aka, d0li0) tried (unsuccessfully) to help me troubleshoot it at Hybridfest, and more recently tekn0wledg (not a CV user, but a programmer familiar with communication software) has tried to help here locally. Tek has given me some additional troubleshooting tips that I haven't had the chance to try yet. (Frankly, I've been preoccupied with buying a house.) Please keep me posted if you're able to resolve it.

    Horacio, was there supposed to be a link in your post? Was it, by chance, the thread where I first described my issues?
     
  20. mrbigh

    mrbigh Prius Absolutum Dominium

    Joined:
    Sep 6, 2005
    3,686
    699
    2
    Location:
    Long Island, NY
    Vehicle:
    Other Electric Vehicle
    <div class='quotetop'>QUOTE(JimboK @ Oct 17 2007, 10:05 PM) [snapback]527170[/snapback]</div>
    Yea, this was the thread in the back of my mind last night.

    After trying to help out Jay on the hardware side on things that he already wisely checked, was typing some more small guidance and lost PC connection o the server hang-up again. I was ready to pinpoint him to the capture program (realterm?) settings or probably limitations or probably in need of an update, unfortunately I do not have it nor I used yet. But by all means I will keep an eye on the thread for some real progress and others headache relief.