Overlapping signals

[kdschlosser]

I have noticed that there is an extremely large number of dbc files that have overlapping signals. This makes it impossible to use the dbc file to properly encode and decode a can message.

I spent a long number of hours fixing the gm_global_a_* dbc files so there are no more overlaps. I am not sure to what extent the others have overlaps I would have to check each file that is not generated. I know I have loaded a few files other then the gm_global_a_* files and there are signal overlaps in the ones I loaded.

[pd0wm]

We should probably have a python script that checks for overlap in CI. I would definitely merge that PR. See https://github.com/commaai/opendbc/blob/master/can/process_dbc.py for parsing.

[kdschlosser]

I am working on writing a DBC file parser in Python. There is one available but I have found it to be slow and it is not complete. I wish I was able to get my hands on a newer version of the specification for DBC files as there is much that has been added in newer versions of the spec. It would make writing it easier to do.

[kdschlosser]

This is what I have for ya in terms of errors.

****************************************
loading acura_ilx_2016_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading acura_rdx_2018_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading acura_rdx_2020_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading ford_lincoln_base_pt.dbc
"Invalid syntax at line 500, column 1: ">>!<<VAL_ 935 SodRight_D_Stat 7 "NotUsed_3" 6 "NotUsed_2" 5 "NotUsed_1" 4 "Invalid" 3 "Disabled" 2 "On" 1 "Trailer_Tow_Off" 0 "Off" ;""

****************************************
loading gm_global_a_lowspeed.dbc
Standard frame id 0x10630000 is more than 11 bits in message DriverDoorStatus.
Standard frame id 0x10400000 is more than 11 bits in message Chime.
Standard frame id 0x1020c000 is more than 11 bits in message BlinkerStatus.
Standard frame id 0x10240000 is more than 11 bits in message SteeringWheelAngle.
Standard frame id 0x102cc000 is more than 11 bits in message GearShifter.
Standard frame id 0x102ca000 is more than 11 bits in message GasPedalRegenCruise.
Standard frame id 0x10250000 is more than 11 bits in message BrakePedal.
Standard frame id 0x106b8000 is more than 11 bits in message WheelSpeed.
Standard frame id 0x10210000 is more than 11 bits in message VehicleSpeed.
Standard frame id 0x10758000 is more than 11 bits in message CruiseButtons.
Standard frame id 0x10756000 is more than 11 bits in message CruiseButtons2.

****************************************
loading gm_global_a_lowspeed_1818125.dbc
The signals TopTrvlCltchSwActV and TopTrvlCltchSwActGroup are overlapping in message Engine_Information_1_LS.
The signals TopTrvlCltchSwAct and TopTrvlCltchSwActGroup are overlapping in message Engine_Information_1_LS.

****************************************
loading honda_accord_lx15t_2018_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_accord_s2t_2018_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_accord_touring_2016_can.dbc
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals APPLY_BRAKES_FOR_CANC and ZEROS_BOH are overlapping in message ACC_HUD_2.

****************************************
loading honda_civic_hatchback_ex_2017_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_civic_sedan_16_diesel_2019_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_civic_touring_2016_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals APPLY_BRAKES_FOR_CANC and ZEROS_BOH are overlapping in message RADAR_HUD.

****************************************
loading honda_clarity_hybrid_2018_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals APPLY_BRAKES_FOR_CANC and ZEROS_BOH are overlapping in message RADAR_HUD.

****************************************
loading honda_crv_executive_2016_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals SET_ME_X00 and SET_ME_X00 are overlapping in message STEERING_CONTROL.

****************************************
loading honda_crv_ex_2017_body_generated.dbc
Standard frame id 0x12f8bfa7 is more than 11 bits in message BSM_STATUS_RIGHT.
Standard frame id 0x12f8be9f is more than 11 bits in message BSM_STATUS_LEFT.

****************************************
loading honda_crv_ex_2017_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_crv_hybrid_2019_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_crv_touring_2016_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals SET_ME_X00 and SET_ME_X00 are overlapping in message STEERING_CONTROL.

****************************************
loading honda_fit_ex_2018_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_fit_hybrid_2018_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals APPLY_BRAKES_FOR_CANC and ZEROS_BOH are overlapping in message RADAR_HUD.

****************************************
loading honda_hrv_touring_2019_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.

****************************************
loading honda_insight_ex_2019_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_odyssey_exl_2018_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals APPLY_BRAKES_FOR_CANC and ZEROS_BOH are overlapping in message RADAR_HUD.

****************************************
loading honda_odyssey_extreme_edition_2018_china_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals STEER_CONFIG_INDEX and STEER_STATUS are overlapping in message STEER_STATUS.

****************************************
loading honda_pilot_touring_2017_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading honda_ridgeline_black_edition_2017_can_generated.dbc
The signals BOH and SET_ME_X41 are overlapping in message LKAS_HUD.
The signals LDW_ON and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals LDW_OFF and SET_ME_X48 are overlapping in message LKAS_HUD.
The signals CLEAN_WINDSHIELD and SET_ME_X48 are overlapping in message LKAS_HUD.

****************************************
loading hyundai_i30_2014.dbc
The signals CAN_VERS and TQ_STND are overlapping in message EMS2.
The signals CONF_TCU and CAN_VERS are overlapping in message EMS2.
The signals OBD_FRF_ACK and CONF_TCU are overlapping in message EMS2.
The signals CAN_VERS and TQ_STND are overlapping in message EMS2.
The signals CONF_TCU and CAN_VERS are overlapping in message EMS2.
The signals OBD_FRF_ACK and CONF_TCU are overlapping in message EMS2.

****************************************
loading hyundai_kia_generic.dbc
"Invalid syntax at line 1568, column 10: "BO_ 1191 >>!<<4a7MFC: 8 XXX""


****************************************
loading luxgen_s5_2015.dbc
Multiple messages named 'undefined'

****************************************
loading mazda_2017.dbc
"Invalid syntax at line 270, column 10: "BO_ 1275 >>!<<2017_5: 8 XXX""

****************************************
loading mazda_3_2019.dbc
The signals NEW_SIGNAL_7 and NEW_SIGNAL_6 are overlapping in message NEW_MSG_12.
The signal NEW_SIGNAL_4 does not fit in message CAM_KEEP_ALIVE_1.
The signal NEW_SIGNAL_4 has in incorrect bit index in message CAM_KEEP_ALIVE_1.
The signals NEW_SIGNAL_6 and NEW_SIGNAL_5 are overlapping in message NEW_MSG_21.

****************************************
loading toyota_2017_ref_pt.dbc
Standard frame id 0x40140639 is more than 11 bits in message BDB1F01_14.
Standard frame id 0x4016063b is more than 11 bits in message BDB1F03_16.
Standard frame id 0x40100620 is more than 11 bits in message BDB1S01_10.
Standard frame id 0x40110621 is more than 11 bits in message BDB1S02_11.
Standard frame id 0x40120622 is more than 11 bits in message BDB1S03_12.
Standard frame id 0x40190623 is more than 11 bits in message BDB1S04_19.
Standard frame id 0x40130638 is more than 11 bits in message BDB1S05_13.
Standard frame id 0x40170630 is more than 11 bits in message BDB1S06_17.
Standard frame id 0x40180631 is more than 11 bits in message BDB1S07_18.
Standard frame id 0x401a0624 is more than 11 bits in message BDB1S08_1A.
Standard frame id 0x401d0627 is more than 11 bits in message BDB1S11_1D.
Standard frame id 0x4098063c is more than 11 bits in message BDB1S17_98.
Standard frame id 0x4099063d is more than 11 bits in message BDB1S18_99.
Standard frame id 0x409a063e is more than 11 bits in message BDB1S20_9A.
Standard frame id 0x409b063f is more than 11 bits in message BDB1S21_9B.
Standard frame id 0x40d606d1 is more than 11 bits in message DMS1S02_D6.
Standard frame id 0x40820634 is more than 11 bits in message IDT1S03_82.
Standard frame id 0x40830635 is more than 11 bits in message IDT1S04_83.
Standard frame id 0x40200610 is more than 11 bits in message MET1S01_20.
Standard frame id 0x40210611 is more than 11 bits in message MET1S02_21.
Standard frame id 0x40230619 is more than 11 bits in message MET1S04_23.
Standard frame id 0x4024061a is more than 11 bits in message MET1S05_24.
Standard frame id 0x40270612 is more than 11 bits in message MET1S08_27.
Standard frame id 0x40280613 is more than 11 bits in message MET1S09_28.
Standard frame id 0x40290614 is more than 11 bits in message MET1S10_29.
Standard frame id 0x402a0615 is more than 11 bits in message MET1S11_2A.
Standard frame id 0x402b0616 is more than 11 bits in message MET1S12_2B.
Standard frame id 0x402006f3 is more than 11 bits in message MET1S22_20.
Standard frame id 0x40e406c0 is more than 11 bits in message PMN1F03_E4.
Standard frame id 0x40000682 is more than 11 bits in message YGW1S01_0.
Standard frame id 0x40000683 is more than 11 bits in message YGW1S02_0.
Standard frame id 0x40000686 is more than 11 bits in message YGW1S05_0.


****************************************
loading volvo_v40_2017_pt.dbc
The signals TextUnderSign and NEW_SIGNAL_6 are overlapping in message FSM5.
Multiple messageas with name 'ECM1'

****************************************
loading vw_golf_mk4.dbc
The signals Timeout_Bremsenbotschaft and Indiziertes_Istmoment__Slave_ are overlapping in message Slave_1.
The signals Pumpentemperatur__3_2_2_ and Zylinderzaehler__3_2_2_ are overlapping in message PSG_1.
The signals Pumpentemperatur__2_1_ and Pumpentemperatur__3_2_2_ are overlapping in message PSG_1.
The signals Frei_Fahrwerk_1_1 and Einstellung_Fahrwerkdaempfung_4 are overlapping in message Fahrwerk_1.
The signals Frei_Fahrwerk_1_2 and Frei_Fahrwerk_1_1 are overlapping in message Fahrwerk_1.
The signals Checksumme_Parkbremse and Erreichte_Spannkraft are overlapping in message EPB_1.
The signals Schalterinfo_Parkbremse and Checksumme_Parkbremse are overlapping in message EPB_1.
The signals Status_Neigungswinkelgeber and Checksumme_Parkbremse are overlapping in message EPB_1.
The signals Verzoegerungsanforderung__EPB_ and Neigungswinkel are overlapping in message EPB_1.

****************************************
loading vw_mqb_2010.dbc
"Invalid syntax at line 36, column 199: "BU_: Airbag_MQB BAP_Tester_MQB BMS_MQB Datenlogger_MQB Gateway_MQB Getriebe_DQ_Hybrid_MQB Getriebe_DQ_MQB LEH_MQB Motor_Diesel_MQB Motor_Hybrid_MQB Motor_Otto_MQB SAK_MQB Waehlhebel_MQB Vector__XXX >>!<<9 l c i XXX""

[rbiasini]

Nice :)
Is there anything wrong with the way that extended IDs are defined? What would trigger this?
Standard frame id 0x40830635 is more than 11 bits in message IDT1S04_83

[kdschlosser]

the CANdb++ specification states that for a can ID that has a length greater then 11 bits the ID needs to have bit 32 toggled to the on position.

I do not believe that a CAN bus in a vehicle will have mixed 11 bit and 29 bit frame ID's. So all of the ID's in the file would need to be masked with 0x80000000. The toggling of this bit is what tells us if the frame id is extended or not.

There are also other"" special" considerations that need to be accounted for as well. with GM if the dbc file is using extended frame ID's then the following lines also need to be in the file.

BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_ "UseGMParameterIDs" 0;

I believe that there may also be a special marker for J1939 as well.

BA_DEF_ BO_ "isj1939dbc" INT 0 0;
BA_DEF_DEF_ "isj1939dbc" 0;

or it could be

BA_DEF_ BO_  "VFrameFormat" ENUM  "StandardCAN","ExtendedCAN","StandardCAN_FD","ExtendedCAN_FD","J1939PG";
BA_ "VFrameFormat" BO_ {MESSAGE_ID} 4;

I haven't yet tinkered with the J1939 protocol so I am not sure yet what triggers the special handling of those ID's

[kdschlosser]

IDK if you have this or not. It is the old specification but it will help

DBC_File_Format_Documentation.pdf

[TheMutley]

the CANdb++ specification states that for a can ID that has a length greater then 11 bits the ID needs to have bit 32 toggled to the on position.

I do not believe that a CAN bus in a vehicle will have mixed 11 bit and 29 bit frame ID's. So all of the ID's in the file would need to be masked with 0x80000000. The toggling of this bit is what tells us if the frame id is extended or not.

There are also other"" special" considerations that need to be accounted for as well. with GM if the dbc file is using extended frame ID's then the following lines also need to be in the file.

BA_DEF_  "UseGMParameterIDs" INT 0 0;
BA_ "UseGMParameterIDs" 0;

I believe that there may also be a special marker for J1939 as well.

BA_DEF_ BO_ "isj1939dbc" INT 0 0;
BA_DEF_DEF_ "isj1939dbc" 0;

or it could be

BA_DEF_ BO_  "VFrameFormat" ENUM  "StandardCAN","ExtendedCAN","StandardCAN_FD","ExtendedCAN_FD","J1939PG";
BA_ "VFrameFormat" BO_ {MESSAGE_ID} 4;

I haven't yet tinkered with the J1939 protocol so I am not sure yet what triggers the special handling of those ID's

I can assure you that there's mixed 11 bits/29 bits CAN network in the wild (VAG for example). Also, the reason there's the UseGMParameterIDs in the DBC, is that some high end CAN software use this to unlock the GMLAN special addressing decoding.

[kdschlosser]

I can assure you that there's mixed 11 bits/29 bits CAN network in the wild (VAG for example). Also, the reason there's the UseGMParameterIDs in the DBC, is that some high end CAN software use this to unlock the GMLAN special addressing decoding.

OK that's good to know about the mixed frame ID's. I know that the specification for CAN does allow it, I did not know if it is actually done. I have seen 2 different buses in a vehicle where one has 11 bit and the other has 29 bit, I have not seen a single bus running both 11 bit and 29 bit.

also CANdb++ I would not consider "high end" because it does not have any price attached to it that would make it high end. It is free software and it is also made by the company that wrote the DBC specification. CANdb++ responds to UseGMParameterIDs, I would think that all dbc files generated or not generates should open in CANdb++ but all support every feature within it. After all if it doesn't load in CANdb++ it would be a pretty safe bet that the file is not to spec.

with UseGMParameterIDs it is important to have it where it is needed because of how a GM frame ID is structured.
for 29 bit ID's it is structured as
bits 0-7 = ecu id
bits 8-12 = reserved
bits 13-25 = arbitration id
bits 26 - 29 = priority

and with 11 bit ID's
bits 0-7 = arbitration id
bits 8-10 = request type

with 29bit ids it is really important to have the UseGMParameterIDs flag set because the only information that is needed from the frame ID is the arbitration id, this is what is used to match a frame to a dbc message because the ecu_id can change and so can the priority, but the data in the frame will still be able to be correctly decoded by the dbc file.

If you create a dbc file and have that attribute set then use CANdb++ to edit the file you will be given a new dialog where to enter the message id, and that dialog is only going to allow you to enter the arbitration id and nothing else.

[kdschlosser]

when dealing with J1939 the frame id has to also get broken apart.

bits 0-7 = source address
bits 8-15 = destination address/group number
bits 16-23 = PDU Format
bit 24 = Data Page
bit 25 = Reserved
bits 26-28 = priority

where the destination address/group number and PDU format are what is used to match a frame to a message in a dbc file.

[kdschlosser]

I have managed to put together a list of DBC attributes along with a description of what they do, minimum and maximum values, default values, choices if it is an ENUM and also the object type that the attribute gets used on.

I have been working on a program that may help with people contributing decoded CAN data. It is an analyzer that is going to work with an assortment of CAN interfaces. It shows all traffic in and out of an interface, it also will group frames together that have the same frame id and show the bits or bytes that are changing. It will allow you to apply factors and offsets to groups of bits in a frame so a user can see the data as it comes in. It is also going to allow "spoofing" so if a frame is received it will immediately send out a frame to the same id using data provided by the user. handy when wanting to test something like a temperature gauge or an rpm gauge. It is also going to allow the user to save information they have input into it to a DBC file.

the program will run on Windows, Linux and Mac

Oh and it also will calculate interface timing registers for non standard bitrates.

[romybompart]

Hi @kdschlosser

A can bus can mix 11 and 20 bit frame ID.

I do not believe that a CAN bus in a vehicle will have mixed 11 bit and 29 bit frame ID's. So all of the ID's in the file would need to be masked with 0x80000000. The toggling of this bit is what tells us if the frame id is extended or not.

For example here you can see there are normal and extended bit frame ID. This is a real vehicle can dbc.

I've also noticed the overlapping, this is possible to handle if the message has a multiplexor signal.
The multiplexor can provide a mapping where some signals are overlapped, but depending on the multiplexor signal value the mapping may change.

So here I got an example, I got these 4 signals overlapped:

But as you can see, they coexists thanks to the Multiplexor (mode column), when M=0 the signal associated to that mode will take place: pass_seatbelt_n_latch_req and trans_in_d_req.

So probably there is no support of multiplexor yet ?

[kdschlosser]

in the DBC files that have overlapping signals there was no multiplexors in use. and also just because a freame ID is say 0x416 as you have in your image doesn't mean that it is an 11 bit ID. it can still be a 29 bit ID. 0x00000416 <--- 29 bit ID the leading 0's are assumed. 0x416 and 0x00000416 are the same number they just have a different number of leading 0 bits. and actually the correct hex format for an 11 bit ID would be 0x0416. the application you are using doesn't respect the leading 0's

[romybompart]

Hello @kdschlosser ,
You are right about the representation of the data, nevertheless, I have done the test before so I know you can use extended and not extended frame id at the can bus. To explore the message property you can use cantools and read check the property is_extended_frame, or even the python-can using the is_extended_id.

Here is the dbc file where you can see the message 1440= 0x5A0, which is not extended, and the 2651930624 = 0x9E114000, they are in the same database

[kdschlosser]

I am sure that a bus is able to support both version 1 and 2 of the specification, I personally have not seen both being used on the same network.

When dealing with a v1 frame vs a v2 frame in the Vector DBC files the id for the frame bit 31 is going to be set to a 1..

the ID 2651930624 is 0x9E114000 in HEX and in binary it is 10011110000100010100000000000000.. note the 1st bit is a 1.
The actual frame id in binary is 00011110000100010100000000000000 and in hex is 0x1E114000 and decimal is 504446976.

Also most vehicles have multiple CAN networks in them, at least 2 typically there is a low speed and a high speed. the high speed network is for critical data, messages between the ECM and TCM is an example. the low speed network is for the body, things like lights, locks, steering wheel controls. stuff that isn't critical. So a DBC file can be written to handle multiple networks in a single file.

I have a version of cantools that I have upgraded so it handles that kind of a scenario, It also has better handling of multiplexors and it properly propagates default attribute definitions. I also trimmed out everything that is not specific to DBC files. I added support for GM parameter frame ID's and also J1939 frame id's. This is important for decoding because there is data that is contained within the frame ID that would cause a message to not get decoded and that data shouldn't be used when matching an incoming message to a message in the DBC file. I also added the full gambit of Vector product attributes, there are a lot of them probably over 200 different attributes. a node can have it's RX and TX messages iterated over and also specifically used for decoding. this way you can isolate a specific module on a vehicle bus to match messages to. There is a heap load of changes I made to it. I did a code overhaul that increases the speed in which the library is able to locate and decode a message. I also added special handling for encoding a message where it will fill in the different bits based on what the default value has been set to. This is a nice feature that allowed me to create a DBC file for OBDII that properly encodes a request and decodes the response, to make the request all a program needs to have knowledge of is a signal name. If a response contains more then one signal it doesn't matter I only need to know the signal name that I want to request and it does all the work and fills in the blanks to build the frame.

I spent a lot of time on the phone with Vector going over the relationships of the objects and the different attributes. The guy on the phone was super helpful and was willing to share the information even tho it is under lock and key so to speak....

There area a few limitations to the DBC file format, it is not able to handle multi frame messages and it is not able to handle frames that have a payload that is greater then 8 bytes. It appears like they may not be adding to the DBC file format anymore. There is another format that they work with that has far more abilities and this is what they are using now between all of the different software they make. I was not told this specifically it is just what appears is going on.

[romybompart]

Good to know @kdschlosser, thank you for the info.

Now that you pointed the j1939 dbc features, Are these added into the latest release of cantool?

I didn't know about the vector version 1 or 2. I am using a standard dbc file. What is in the ID message will be taken as it is by the nodes, in my case.

[kdschlosser]

yes and no to the j1939 spec being added to cantools. It is 1/2 way added i guess you can say.. It is not going to properly match a frame to be decoded if any portion of the frame id changes that really isn't the "id" so to speak.

I will explain using GM parameter id's as I know this right off the top of my head and I do not have to go and look it up again, but it is the same kind of scenario when using the J1939 protocol.

a GM parameter id is the frame id but it contains information that shouldn't be considered when matching a frame to be decoded. a GM parameter ID can be either an 11 bit or a 29 bit frame id and they get broken down a little differently, I am going to go over the 29bit id.

this is how the frame id gets broken apart when it is a gm parameter id.

 28                                                                   0
  1 1 1     1 1 1 1 1 1 1 1 1 1 1 1 1      1 1 1 1 1    1 1 1 1 1 1 1 1
 priority         arbitration id            reserved    sending node id

the only piece of information that is needed to match a frame that is to be decoded with a message in a DBC file is the arbitration id. the rest of the information is not needed. the arbitration id is what dictates how the payload is to be decoded. more then one module (node) on the network can send the same data and can use the same arbitration id. the sending node id would be set to the id of the sending node. The priority is just that. it is the priority the frame is given on the network. This is not relevant when making match either. so these things need to be stripped off the frame id of the frame to be decoded when looking for a possible match.

so you can see why this is very important to consider, cantools does not do this. and it does not do this when handling the J1939 protocol either, and it should be in order to properly match a message to be decoded to a message in the database.

It is also important to tie the RX and TX signals to a node when dealing with these types of frame id's. Take this as an example. the RKE (Remote Keyless Entry) frame id will be different then say the DDM (Drivers Door Module). so if I press the unlock button on the remote it is going to create a frame that the BCM (Body Control Module) will see and know to unlock the door. If I press the unlock button on the driver door it is going to do the same... The payload of both frames is going to be identical. the arbitration id's are going to be identical the frame id's are not.. the sending node id will differ.. so if I have the doors locked and the factory alarm armed if I send a frame using the frame id from the DDM this is going to trigger the factory alarm. but if I send the frame using the frame id that the RKE uses the factory alarm will not get triggered.

the updates I made to cantools take this kind of a scenario into consideration. There are Vector specific attributes that handle this kind of a thing and allow for id's to be applied to nodes.

The attributes are

• TpAddressingMode: standard or extended address mode
• TpAddressExtension: if extended mode the address extension
• TpRxIdentifier: id for receiving
• TpTxIdentifier: id for sending
• ECU: name of the ecu the node belongs to. There are cases when a single "module" carries out more then a single function. an example would be a PCM (Powertrain Control Module) which is an ECM (engine control module) and a TCM (Transmission Control Module) combined into a single physical device or "ECU"

all of these things get considered when using a node to encode/decode a message.

All of the above information is relevant to the J1939 protocol as well. the frame id gets broken apart differently then the GM parameter id, the working principal is the same.

There are a slew of Vector attributes that exist, and they all serve a purpose. a lot of them are to carry out various operations in other software they make, they still can be used in a DBC parser to make it behave in a manner that it should. Vector's main purpose is they write software that aids in building a complex CAN network. With their software the communications system in a vehicle can be duplicated and also simulated. I am pretty sure that vehicle manufacturers use either Vectors software or other software like it to design and test the communications systems in a car. then they use the output files from that software (like the DBC file) and place them on the individual modules and the modules use those files to encode and decode messages on the network.

there are attributes that control when to send a frame, how often to send it, and how many times to send it. only one purpose for those being there and that is to instruct a module so it knows what to send and when to send it. these attributes are used in the simulation software but also can be used by a module directly if the DBC file is flashed to the module.

• GenMsgCycleTimeActive
• GenMsgStartDelayTime
• GenMsgCycleTimeFast
• GenMsgDelayTime
• GenMsgNrOfRepetition
• GenMsgFastOnStart

as I said, I spent quite a bit of time on the phone with Vector support, several hours...

also the version 1 and 2 I am referring to is version 1 and 2 of the CAN specification, v1 being 11 bit frame id's and v2 being 29 bit frame id's

[kdschlosser]

It is really complex because of how nested things can be dealing with CAN and also how they are referenced.

CAN-BUS is the hardware layer, the specifications for CAN-BUS cover a lot of things like the timings and frequencies, speed, what we are interested in is the arrangement of the 1's and 0's and how many 1's and 0's make a packet or "frame". this is how the information gets sent down the wire at the hardware level. the 2 pieces we use are the frame id and the payload, the payload being the data being sent.

Then yo have higher level protocols that ride on top of the CAN-BUS specifications. these protocols define what is contained in the frame id and the payload portion of the frame. so when we refer to OBDII this is a higher level protocol CAN-BUS is one of the many physical transport methods. the OBDII specification dictates what must be in the frame id and the payload portions. all vehicle communication methods have an id of some sort and also a payload. the higher level protocols simply tell us what to put and where to put it. the lower level or hardare protocols dictate how to transmit that information down the wire.

GM Parameter ID's, OBDII, J1939 are all high level protocols. Almost every vehicle manufacturer has their own high level protocol. and all vehicles made after 2008 MY (Model Year) have to have available the OBDII protocol for emissions related events taking place. There are concessions in the ODB protocol that do allow for a vehicle manufacturer to add additional commands, in most cases these commands are something that has to be purchased at a high cost, GM charges 50,000.00 USD a year to have access to the documentation for these commands. and they change the commands between models and years all the time.

You can set up a man in the middle between a code scanner that supports "enhanced" OBDII codes and the vehicle so you can isolate what is being sent from the scan tool, there are applications like the Torque app that does more of a brute force attach and will systematically try every possible extended command. The problem with this attack is you do not know what the command is for so you have no insight as to how to decode the payload. where as the man in the middle you are instructing the code scanner to request something so you know what the command is for and gives insight on how to reverse engineer the payload in the response. Best to know someone that has one of the advanced SnapOn type code scanners when doing the man in the middle attack, you will have more available to you on those types of scan tools.

When I finish up with the can bus decoder I am writing it is going to really make it easy to reverse engineer the CAN networks in a vehicle. and when I mean easy, I mean it is going to do most of the work. most manufacturers use the same math over and over again when encoding various payloads. an example would be oil pressure, manifold air pressure, fuel pump pressure (low) and air intake temperature, ambient temperature; these things will be grouped together and use the same "factor" or equation to decode them. It makes it far easier. by identifying the frame and tagging it as a specific "type" then the program will apply a generic factor for that type to it, this factor can be changed by the user. I have been working on an algorithm that will attempt to zero in on high speed frames for speed, rpm, temperatures those kinds of things automatically. at the very least it will be able to narrow down what frames could possibly carry that kind of data.

[collin80]

Yes, it is certainly important to support J1939 and GMLAN with DBC messages otherwise they will not be interpreted properly. I didn't do that but I was lucky to have other users add support to my CAN reverse engineering tool.

If you're making a reverse engineering tool (Welcome! We have chips and salsa) and thinking of automated signal searching then you might want to follow Brent Stone: https://github.com/brent-stone

I'm glad to hear that someone from Vector actually gave someone the time of day. They seem rather standoffish to me. It would be a very useful thing to the community if you are planning to open source your CAN decoder. The current state of DBC loading is rather hit or miss.

[kdschlosser]

@collin80

I have used SavvyCAN it is a good program 👍, It didn't work with my interface so i had to write the data to a CSV file. but it did help me to get a better understanding on CAN. I used your application when I first started messing about with CAN 4-5 months ago.

It doesn't appear like your application connects to interfaces via an interfaces dynamic library files It seems like it is more geared to connecting via an intermediate piece of software like SocketCAN or to a micro controller based solution over a serial connection. If you ever decide to expand onto this portion of your app let me know, I have written some code in Python that can pretty easily be ported to C/CPP. It would make things a whole lot easier if you ever decide you want to expand the interface support in SavvyCAN.

The hardest part when working with an interfaces library files is dealing with the timing registers. I wrote a program that generates timing registers for any bitrate that gets fed into it, the sample point and SJW can optionally be fed in as well.

It supports interfaces that use chipsets from these manufacturers

• Renesas
• Texas Instruments
• Microchip
• Atmel
• Silabs
• NXP

The fsys (oscillator frequency) can be specified and also the margin of error. If the timing registers are not able to be calculated for the bitrate with the given fsys the program will find the closest bitrate matching the given sample point that timing registers can be generated for, as long as it is within the given margin of error.

If this is something you are interested in let me know and I will shoot it on over to ya.

I also have a list of 103 attributes used in DBC files along with descriptions, object types they get applied to. min, max and default values. data type... all of the information needed to add them to an application. I like SavvyCAN and I would have continued to use it if I didn't have to create CSV files and read the data that way. Using the CSV files turns off some of your applications functionality.

I do not know if these features are available because I was not able to connect SavvyCAN to my interface. If they exist you can ignore this. If they do not exist you might consider adding these features.

• An option to display the payload as binary data with user selectable bit and byte order.
• The ability to highlight bits in a frame and apply an offset and factor to it or an expression to encode/decode a payload with.
• Send a selected frame exactly as it has been received
• Send the frame with data that has been changed by the user.
• A user editable table where a user can enter in offsets and factors or equations and apply a name to them, and offer these as a choice to apply to a frame. and also as a choice when wanting to change the payload of a frame. Car manufacturers seem to use the same equations over and over again so having them saved and be able to select them by name would be a a really nice feature to have available.
• Moving applied factors and offsets and selected bits into the DBC editor

[kdschlosser]

I will finalize some of the changes I have made to cantools and create a repository for it this way you can take a look at it if you want. If it is something that interests you and you want to add it to an existing project that is not written in Python I can also write a script that can be run and be controlled either through a named pipe, a socket on the loopback connector, websockets, etc... this way there is no need to port the code. I can compile it into a single file executable for windows and your app could run the executable, on posix systems I believe Python comes installed and that could be used to run the script.

Oh! another mechanism that can be used for passing information is creating a couple of exported C functions where a callback function can be supplied, so when the Python program runs I would be able to pass callbacks to your application and when your application needs to do something it can request what it needs over that callback. I am pretty good at writing python bindings to C libraries, this would be the easiest solution requiring the least amount of code needed on either side. the only thing that would need to be done to make it cross platform would be knowing the OS in order to apply the proper extension for the library file to connect to, this would be on the python side and also how to run the python script. Nothing hard to deal with.

[collin80]

Yeah, I have someone currently adding support for dynamically linked libraries. Then it will be much easier to support a range of capture devices. And, a lot of the features you mentioned are in there, perhaps not always in easy to find places. I would be interested in looking at your changes to cantool. I've been wanting to add python support to SavvyCAN for a while and just haven't done it. That'd be a very good addition for a number of reasons. This OpenDBC repo is quite a nice thing for the community and I appreciate the effort to make the DBC files here more compliant and proper.

[kdschlosser]

using either cantools or my version of cantools would help to locate problems in the current DBC files. I will be adding the ability to merge 2 DBC files together. it will be able to handle object collisions when I do add that functionality to it.

This is what I believe to be the easiest way to go about adding say my version of the DBC parser too your application.

You would need to create an exported C function that I can pass a callback to. I have the ability to wrap a python function so it will look like a C function. Your application will not know the difference. You need to create 2 structures that you would pass to the callback when you are requesting it to do something. the first structure would be the "comand" structure. this would contain the information to instruct my code what to do and it would also contain any data that may be needed to carry out that command. the second structure is what I would fill with any information that would need to be passed back to your application.

My program I can compile into an executable so it can be used on a Windows machine without the need to have python installed. When your app starts up it would run the executable and the executable would then make the connection using that exported C function. It would work similiar on posix systems except the python script would be there instead of an executable. Python comes preinstalled on most posix systems so you would just run python passing the script name as an argument to it. When your program shuts down it would send an instruction for my program to exit. no need to mess around with terminating processes and such.

The same kind of a mechanism can be used to add interfaces as well.

I did make a repo for my version of the DBC file parser
https://github.com/kdschlosser/vector_dbc

It is still a test version and there is some more work that I am going to do to it. and I am sure there is probably a few bugs kicking about in it.

[kdschlosser]

I am going to move this conversation over to your repo. I will open an issue there so we can continue the discussion.