Reverse Engineering the BMW ZF 8HP CAN Bus

Earlier this year I dragged home a ZF 8HP70X and a spare broken 8HP45 from an F10 530xd and 520d. At first the plan was just to bolt it up to my E39 with the M57 and see if a cheap DIY adapter plate was possible. Honestly, the plates you can buy now aren’t that expensive, so I don’t think it’s something worth spending time on. Pretty quickly I started turning my attention to the controller that would be needed to run the gearbox.

First Assumptions

I thought it’s going to be easy. The gearbox needs just a few wires:

• +12 V battery
  
• Ground
• Ignition signal
• Two PT-CAN buses
  
  

Sounds simple. Unfortunately, it’s not.

If you put this gearbox in another chassis, outside of the donor car, you run into walls. The transmission control unit (EGS/TCU) is basically deaf and blind if it doesn’t see the rest of the car talking to it on CAN. Without those messages, the TCU won’t even wake up properly.

First Logs

I connected to PT-CAN1 and PT-CAN2 and took some logs. Both run at 500 kbps.

My initial expectation was maybe 10–15 messages. What I got was around 60–70 IDs across both buses. Some are doubled for redundancy, but each bus has a few unique ones. It was a mess on the screen.

At the time all I had were a handful of Arduino Nanos and MCP2515 CAN bus modules, so using SavvyCAN wasn’t really an option. That meant I had to write my own Python program to filter selected IDs and try to make some sense of the network. First though, I wanted to see how the shifter works.

The shifter itself sends 3 IDs:

• 0x55E, 0x65E, 0x197

It looks for the following IDs from the TCU:

• 0x3FD, 0x1A6, 0x202

I wrote the following code for the Arduino Nano and MPC2515:

#include 
#include 

// ————— Hardware & CAN setup —————
const int SPI_CS = 10;
MCP_CAN CAN(SPI_CS);

const uint32_t GWS_ID       = 0x197;
const uint32_t EGS_ID       = 0x3FD;
const uint32_t HEARTBEAT_ID = 0x55E;
const uint32_t DUMMY1_ID    = 0x1A6;
const uint32_t DUMMY2_ID    = 0x0B6;

// ————— Gear bytes —————
#define GEAR_P   0x20
#define GEAR_R   0x40
#define GEAR_N   0x60  
#define GEAR_D   0x80
#define GEAR_MS  0x81

// ————— State & Timing —————
uint8_t  confirmedGear = GEAR_P;  // start locked in Park
uint8_t  lastRaw       = 0xFF;
uint8_t  pendingRaw    = 0xFF;
bool     parkLatched   = true;    // we begin in Park latched
unsigned long tHoldStart = 0;

unsigned long tEGS   = 0;
unsigned long tHB    = 0;
unsigned long tDummy = 0;

bool     gws_awake   = false;
uint8_t  counterEGS  = 0;

// ————— CRC‑8 (BMW) —————
uint8_t crc8_bmw(const uint8_t* data, size_t len) {
  uint8_t crc = 0;
  while (len--) {
    crc ^= *data++;
    for (uint8_t i = 0; i < 8; i++)
      crc = (crc & 0x80) ? (crc << 1) ^ 0x1D : (crc << 1);
  }
  return crc ^ 0x70;
}

// ————— Map raw selector to gear byte —————
uint8_t mapRawToGear(uint8_t raw) {
  switch (raw) {
    case 0x3F: return GEAR_P;
    case 0x2F: return GEAR_R;
    case 0x4F: return GEAR_D;
    case 0x1F: return GEAR_N;
    case 0x7F:
    case 0x6F:
    case 0x5F: return GEAR_MS;
    default:   return 0x00;  // includes 0x0F (Neutral)
  }
}

// ————— Send 0x3FD confirmation —————
void sendEGS() {
  if ((counterEGS & 0x0F) == 0x0F) counterEGS++;
  uint8_t msg[8] = {0, counterEGS, confirmedGear, 0,0,0,0,0};
  msg[0] = crc8_bmw(&msg[1], 4);
  CAN.sendMsgBuf(EGS_ID, 0, 8, msg);
  counterEGS++;
}

// ————— Send 0x55E heartbeat —————
void sendHeartbeat() {
  uint8_t hb[8] = {0,0,0,0,1,0,0,0x5E};
  CAN.sendMsgBuf(HEARTBEAT_ID, 0, 8, hb);
}

// ————— Send dummy frames —————
void sendDummies() {
  uint8_t a[8] = {0,0,0xE0,0,0x20,0,0,0};
  uint8_t b[8] = {0,0xC0,0,0,0,0,0,0};
  CAN.sendMsgBuf(DUMMY1_ID, 0, 8, a);
  CAN.sendMsgBuf(DUMMY2_ID, 0, 8, b);
}

void setup(){
  Serial.begin(115200);
  while (CAN.begin(MCP_ANY, CAN_500KBPS, MCP_8MHZ) != CAN_OK) {
    Serial.println("CAN init failed; retrying...");
    delay(100);
  }
  CAN.setMode(MCP_NORMAL);
  Serial.println("🔧 CAN ready — starting in PARK (latched)");
  tEGS   = tHB   = tDummy = millis();
}

void loop(){
  unsigned long now = millis();

  // 1) Listen for GWS (0x197)
  if (CAN_MSGAVAIL == CAN.checkReceive()){
    uint32_t id; uint8_t len, buf[8];
    CAN.readMsgBuf(&id, &len, buf);
    if (id == GWS_ID && len >= 4){
      uint8_t raw     = buf[2];
      bool    parkBtn = (buf[3] == 0xD5);
      uint8_t gear    = mapRawToGear(raw);

      if (!gws_awake){
        gws_awake = true;
        lastRaw   = raw;
        Serial.print("GWS awake, raw=0x"); Serial.println(raw, HEX);
      } else {
        // — Park button always immediate, one-shot —
        if (parkBtn){
          if (confirmedGear != GEAR_P){
            confirmedGear = GEAR_P;
            Serial.println("Park button — latched PARK");
          }
          parkLatched   = true;
          pendingRaw    = 0xFF;
        }
        else {
          // As soon as lever moves, clear park-latched
          if (raw != lastRaw){
            parkLatched = false;
          }

          // Only if parkLatched==false do we allow detent holds
          if (!parkLatched && gear){
            if (raw != pendingRaw){
              pendingRaw   = raw;
              tHoldStart   = now;
              Serial.print("Holding detent raw=0x"); Serial.println(raw, HEX);
            }
            else if (now - tHoldStart >= 500 && gear != confirmedGear){
              confirmedGear = gear;
              Serial.print("Held 500 ms — latched 0x"); Serial.println(gear, HEX);
              pendingRaw = 0xFF;
            }
          }
        }

        lastRaw = raw;
      }
    }
  }

  // 2) Send EGS every ~50 ms
  if (gws_awake && now - tEGS >= 50){
    sendEGS();
    tEGS = now;
  }
  // 3) Heartbeat every ~640 ms
  if (now - tHB >= 640){
    sendHeartbeat();
    tHB = now;
  }
  // 4) Dummy frames every ~100 ms
  if (now - tDummy >= 100){
    sendDummies();
    tDummy = now;
  }
}

The result:

TCU Is... Buzzing?

After playing with the shifter for a day or two I gathered enough courage to start working on the TCU. After giving it power, ground and WUP it started… buzzing. Also, the ampmeter on my power supply immediately jumped to around 2 A, which was concerning. I turned everything off and started looking for the issue. I didn’t have to look far: after plugging the TCU back into the car, the noises were gone, so I suspected it was related to missing messages on PT-CAN1 and PT-CAN2. After more digging I determined the most important message for the TCU is 0x0A5 on PT-CAN2. Sending only this message on PT-CAN2 is enough to stop the buzzing and let the TCU initialize properly. Coincidentally, this message also carries RPM and torque.

It seems like a lot of messages on this bus use a CRC-8/SAE-J1850 checksum. Byte 0 is the checksum and byte 1 looks like a counter. On this particular message it goes from 0x10 to 0x1F. If the checksum is incorrect, the TCU goes back to “sleep” and then wakes up when it detects a correct one. I spent a week diagnosing random clicking from the TCU that turned out to be this exact issue. The ID must be sent every 10 ms.

Diagnostics

For diagnostics I ended up writing a small program that runs on an Arduino Nano with an MCP2515. It’s nothing complicated—just a text menu over serial—but it lets me read VIN, immo status, adaptations, and clear fault codes directly from the gearbox. The reason I went this route is simple: ISTA will not communicate with the EGS when the transmission is out of the car. The FRM module acts as a gateway between ISTA and the TCU, and without it, ISTA never reaches the box. On the bench that makes OEM tools almost useless. With my little Arduino I can at least talk straight to the TCU and see what’s going on. I also tried to deal with the differential ratio, but that’s been inconsistent and more trouble than it’s worth, so I’ve parked that part for now. For day-to-day work, having this tool is a huge improvement—it saves me from constantly moving the gearbox back into the donor car just to check for errors, and it gives me exactly the bits of information I need to keep moving.

Code: 

/*
  
  │  Menu keys (single-char, press then Enter not required):                 │
  │    1 = Read DTCs                                                         │
  │    2 = Read Info  (VIN, Immo, F101: HWEL/BTLD/SWFL1/SWFL2 + Diff Ratio)  │
  │    3 = Read Adaptations (A–E: pressures, fill times, update counters)    │
  │    c = Clear DTCs (UDS $14 0xFFFFFF)                                     │
  │    r = Sniff UDS $31 RoutineControl Start RIDs for 10s                   │
  │    a = Run EGS Adaptation Reset routine (requires RID)                   │
  │    d = Run Diff-Ratio teach/reset routine (requires RID)                 │
  │    v = Toggle verbose                                                    │
  │    s = Show session (light probe)                                        │
  │                                                                          │
  │  Line commands (type then Enter):                                        │
  │    R?       → show saved diff ratio (EEPROM)                              │
  │    R3.15    → set + save diff ratio (EEPROM cache for Info screen)       │
  │                                                                          │
 

#include SPI.h>
#include "mcp_can.h"
#include EEPROM.h
#include math.h
#include string.h
#include stdio.h
// ─────────────────────────────────────────────────────────────────────────────
// Compile-time configuration
// ─────────────────────────────────────────────────────────────────────────────
#define CAN_CS    10
#define CAN_INT    2
#define CLOCKSET MCP_8MHZ     // set to MCP_16MHZ if your MCP2515 board is 16 MHz
#define CAN_BPS  CAN_500KBPS  // PT-CAN2 500 kbit/s

// Verbose logging toggle (can be changed at runtime with 'v')
static bool g_verbose = false;   // default OFF; press 'v' to toggle

// EEPROM slot for diff ratio cache
static const uint16_t DIFF_EE_OFF = 0;     // 5 bytes used (4 data + 1 checksum)

// ─────────────────────────────────────────────────────────────────────────────
// Types (defined BEFORE any function that mentions them to beat auto-prototypes)
// ─────────────────────────────────────────────────────────────────────────────
struct FItem {
  uint16_t did;
  uint8_t  a, b, c;
  bool     found;
};

// Forward declare any functions that take FItem to be extra safe
static void printF101Line(const char* tag, const FItem &it);
static void harvestF101_scan(uint8_t *buf, int len,
                             FItem *HWEL, FItem *BTLD, FItem *SW1, FItem *SW2);

// ─────────────────────────────────────────────────────────────────────────────
// MCP2515 driver
// ─────────────────────────────────────────────────────────────────────────────
MCP_CAN CAN(CAN_CS);

// ─────────────────────────────────────────────────────────────────────────────
// UDS / ISO-TP constants
// ─────────────────────────────────────────────────────────────────────────────
const uint8_t  EGS_TGT        = 0x18;            // extended address target byte
const uint32_t PRIMARY_REQ_ID = 0x6F1;           // Tester → EGS
const uint32_t RESP_ID        = 0x618;           // EGS → Tester (F-series EGS)
const uint16_t ISO_TP_TIMEOUT = 2500;            // ms
const uint8_t  FC_BLOCK_SIZE  = 0x08;            // normal FC pacing
const uint8_t  FC_STMIN       = 0x0A;            // 10 ms

static uint32_t g_last_req_id = PRIMARY_REQ_ID;

// ─────────────────────────────────────────────────────────────────────────────
// RoutineControl ($31) RID placeholders (fill after sniffing)
// ─────────────────────────────────────────────────────────────────────────────
// CAUTION: Set these to the RIDs you sniff from ISTA/xHP. Leaving 0x0000
// prevents accidental routine execution.
static uint16_t RID_RESET_ADAPT   = 0x0000; // e.g. "Reset EGS adaptations"
static uint16_t RID_RESET_DIFF    = 0x0000; // e.g. "Teach/Reset rear axle ratio"

// Optional small params (often none). Leave empty unless your sniff shows payload.
static uint8_t  RID_PARAM_BUF[6]  = {0,0,0,0,0,0};
static uint8_t  RID_PARAM_LEN     = 0;      // set >0 only if you captured params

// ─────────────────────────────────────────────────────────────────────────────
static void hexdump(const uint8_t* p, int n) {
  for (int i=0;i outmax || L > avail) return -4;
      memcpy(out, &b[2], L);
      return L;
    }

    // First Frame
    if (pci == 0x10) {
      int total = ((b[1] & 0x0F) << 8) | b[2];
      int avail = (int)len - 3; if (avail <= 0) return -4;
      int copied = avail; if (copied > total) copied = total;
      if (copied > outmax) return -4;
      memcpy(out, &b[3], copied);

      // Flow Control
      uint8_t fc[8] = {
        EGS_TGT, 0x30,
        quiet_fc ? 0x00 : FC_BLOCK_SIZE,
        quiet_fc ? 0x00 : FC_STMIN,
        0,0,0,0
      };
      if (!sendFrame(g_last_req_id, fc, 8)) return -5;

      while (copied < total && (int32_t)(millis() - deadline) < 0) {
        if (!tryReadOnce(id, b, len)) continue;
        if (id != RESP_ID) continue;
        if ((b[1] & 0xF0) != 0x20) continue; // Consecutive Frame only
        int a = (int)len - 2; if (a <= 0) continue;
        int take = a; if (take > (total - copied)) take = total - copied;
        if (copied + take > outmax) return -4;
        memcpy(out + copied, &b[2], take);
        copied += take;
      }
      return (copied >= total) ? total : -6;
    }
  }
  return -1; // timeout
}

// ─────────────────────────────────────────────────────────────────────────────
// UDS helpers
// ─────────────────────────────────────────────────────────────────────────────
static bool udsSendSF(uint32_t req_id, const uint8_t *p, uint8_t l) {
  if (l > 7) return false;
  uint8_t d[8] = {0};
  d[0] = EGS_TGT; d[1] = l;
  memcpy(&d[2], p, l);
  g_last_req_id = req_id;
  return sendFrame(req_id, d, 8);
}

static int udsRequest(const uint8_t *p, uint8_t l, uint8_t *r, int rm, bool quiet_fc=false) {
  flushRx(10);
  if (!udsSendSF(PRIMARY_REQ_ID, p, l)) return -10;

  int n = isotpReceive(r, rm, ISO_TP_TIMEOUT, quiet_fc);

  // Handle Response Pending (0x7F 0xXX 0x78)
  uint32_t t0 = millis();
  while (n >= 3 && r[0]==0x7F && r[2]==0x78 && (millis()-t0) < 3000) {
    if (g_verbose) { Serial.println(F("... response pending, waiting ...")); }
    n = isotpReceive(r, rm, ISO_TP_TIMEOUT, quiet_fc);
  }
  return n;
}

static bool changeSession(uint8_t sub) {
  uint8_t req[] = {0x10, sub}, r[16];
  int n = udsRequest(req, 2, r, sizeof(r));
  if (n>=2 && r[0]==0x50 && r[1]==sub) {
    if (g_verbose) { Serial.print(F("Session: ")); Serial.println(sessionName(sub)); }
    delay(40);
    return true;
  }
  if (g_verbose) {
    Serial.print(F("Session change to ")); Serial.print(sessionName(sub)); Serial.println(F(" failed."));
  }
  return false;
}

static bool diagStart() {
  if (!changeSession(0x03)) return false; // ExtendedDiagnostic
  uint8_t tp[] = {0x3E,0x00}, rr[8];
  udsRequest(tp,2,rr,8);
  delay(20);
  return true;
}

static void testerPresentOnce() {
  uint8_t req[] = {0x3E,0x00}, r[8];
  udsRequest(req,2,r,8);
}

// ─────────────────────────────────────────────────────────────────────────────
// Byte helpers
// ─────────────────────────────────────────────────────────────────────────────
static int16_t  be16s(const uint8_t *p){ return (int16_t)((p[0]<<8)|p[1]); }
static uint16_t be16u(const uint8_t *p){ return (uint16_t)((p[0]<<8)|p[1]); }

// ─────────────────────────────────────────────────────────────────────────────
// VIN / Immo
// ─────────────────────────────────────────────────────────────────────────────
static void readVIN() {
  uint8_t req[]={0x22,0xF1,0x90}, r[96];
  for(int i=0;i<2;i++){
    int n=udsRequest(req,3,r,96);
    if(n>=20 && r[0]==0x62 && r[1]==0xF1 && r[2]==0x90){
      char vin[18]; int j=0;
      for(int k=3; k='0'&&c<='9')||(c>='A'&&c<='Z')) vin[j++]=(char)c;
      }
      vin[j]=0;
      Serial.print(F("VIN: ")); Serial.println(vin);
      return;
    }
    delay(30);
  }
  Serial.println(F("VIN: read failed."));
}

static void readImmo() {
  uint8_t req[]={0x22,0xC0,0x00}, r[48];
  int n=udsRequest(req,3,r,48);
  if(n>=4 && r[0]==0x62){
    uint8_t st=r[3];
    Serial.print(F("Immo: "));
    if(st==0x00) Serial.println(F("UNLOCKED"));
    else if(st==0x01) Serial.println(F("LOCKED"));
    else { Serial.print(F("state=0x")); Serial.println(st,HEX); }
  } else {
    Serial.println(F("Immo info: not available."));
  }
}

// ─────────────────────────────────────────────────────────────────────────────
// Diff ratio: EEPROM cache + probe candidates
// ─────────────────────────────────────────────────────────────────────────────
static void saveDiffRatio(float r){
  union { float f; uint8_t b[4]; } u; u.f = r;
  uint8_t sum=0; for(int i=0;i<4;i++) sum+=u.b[i];
  for(int i=0;i<4;i++) EEPROM.update(DIFF_EE_OFF+i, u.b[i]);
  EEPROM.update(DIFF_EE_OFF+4, sum);
}
static bool loadDiffRatio(float &r){
  union { float f; uint8_t b[4]; } u;
  uint8_t sum=0, chk=EEPROM.read(DIFF_EE_OFF+4);
  for(int i=0;i<4;i++){ u.b[i]=EEPROM.read(DIFF_EE_OFF+i); sum+=u.b[i]; }
  if (sum==chk && isfinite(u.f) && u.f>2.0 && u.f<5.5) { r=u.f; return true; }
  return false;
}

// Try to read diff ratio from a few candidate DIDs (EGS/related)
static bool readDiffRatioProbe(float &out){
  struct Cand { uint16_t did; } cands[] = {
    {0xF187}, {0xF18C}, {0xF1A2}, {0xF1B1} // plausible vendor DIDs
  };
  uint8_t r[32];
  for (uint8_t i=0;i>8), (uint8_t)cands[i].did};
    int n = udsRequest(req,3,r,sizeof(r));
    if (n>=3 && r[0]==0x62 && ((r[1]<<8)|r[2])==cands[i].did){
      int L = n-3;
      auto ok=[&](float v){ return isfinite(v) && v>=2.2f && v<=4.5f; };
      if (L==1){ float v=r[3]/10.0f; if (ok(v)){ out=v; return true; } }
      if (L>=2){
        uint16_t u16=(uint16_t)((r[3]<<8)|r[4]);
        float v1=u16/1000.0f, v2=u16/100.0f;
        if (ok(v1)){ out=v1; return true; }
        if (ok(v2)){ out=v2; return true; }
      }
      if (L>=4){
        union{ uint32_t u; float f; } u; u.u = ((uint32_t)r[3]<<24)|((uint32_t)r[4]<<16)|((uint32_t)r[5]<<8)|r[6];
        if (ok(u.f)){ out=u.f; return true; }
      }
    }
  }
  return false;
}

static void printDiffRatio(){
  float v;
  if (readDiffRatioProbe(v)) {
    Serial.print(F("Diff ratio: ")); Serial.println(v,3);
    saveDiffRatio(v); // cache for next time
    return;
  }
  if (loadDiffRatio(v)) {
    Serial.print(F("Diff ratio (saved): ")); Serial.println(v,3);
    return;
  }
  Serial.println(F("Diff ratio: (unknown)  — type R3.15 then Enter to set"));
}

// ─────────────────────────────────────────────────────────────────────────────
// F101 aggregate parsing (robust scan for the four items)
// Supports: DID,a,b,c  OR  DID,0x03,a,b,c
// ─────────────────────────────────────────────────────────────────────────────
static void harvestF101_scan(uint8_t *buf, int len,
                             FItem *HWEL, FItem *BTLD, FItem *SW1, FItem *SW2)
{
  if (!(len>=6 && buf[0]==0x62 && buf[1]==0xF1 && buf[2]==0x01)) return;

  // Known DIDs (pri + aliases)
  const uint16_t DID_HWEL = 0x022A;
  const uint16_t DID_BTLD = 0x0C7C;
  const uint16_t DID_SW1P = 0x0A81;
  const uint16_t DID_SW1A = 0x0C7D; // alias
  const uint16_t DID_SW2P = 0x1CDA;
  const uint16_t DID_SW2A = 0x27F0; // alias

  // linear scan; at each index try both shapes
  for (int i=3; i+4found) { *HWEL = {did,a,b,c,true}; used=true; }
      if (did==DID_BTLD && !BTLD->found) { *BTLD = {did,a,b,c,true}; used=true; }
      if ((did==DID_SW1P || did==DID_SW1A) && !SW1->found) { *SW1 = {did,a,b,c,true}; used=true; }
      if ((did==DID_SW2P || did==DID_SW2A) && !SW2->found) { *SW2 = {did,a,b,c,true}; used=true; }
      if (used) { i += 4; continue; }
    }

    // try len=0x03 between DID and data
    if (i+5 < len && buf[i+2]==0x03) {
      uint8_t a = buf[i+3], b = buf[i+4], c = buf[i+5];
      bool used = false;
      if (did==DID_HWEL && !HWEL->found) { *HWEL = {did,a,b,c,true}; used=true; }
      if (did==DID_BTLD && !BTLD->found) { *BTLD = {did,a,b,c,true}; used=true; }
      if ((did==DID_SW1P || did==DID_SW1A) && !SW1->found) { *SW1 = {did,a,b,c,true}; used=true; }
      if ((did==DID_SW2P || did==DID_SW2A) && !SW2->found) { *SW2 = {did,a,b,c,true}; used=true; }
      if (used) { i += 5; continue; }
    }
  }
}

static void printF101Line(const char* tag, const FItem &it) {
  if (!it.found) return;
  char ver[16];
  snprintf(ver, sizeof(ver), "%03u.%03u.%03u", it.a, it.b, it.c);
  Serial.print(F("  ")); Serial.print(tag);
  Serial.print(F(" (DID 0x")); Serial.print(it.did, HEX); Serial.print(F("): "));
  Serial.println(ver);
}

// Fetch F101 aggregate with retries across sessions + FC styles
static void readF101() {
  FItem HWEL={0,0,0,0,false}, BTLD={0,0,0,0,false}, SW1={0,0,0,0,false}, SW2={0,0,0,0,false};

  uint8_t req[] = {0x22,0xF1,0x01};
  uint8_t r[384];
  const uint8_t sessions[] = {0x01, 0x03}; // Default then Extended

  for (uint8_t s=0; s=4 && r[0]==0x62 && r[1]==0xF1 && r[2]==0x01)) {
      if (g_verbose) Serial.println(F("F101: retry with quiet FC"));
      n = udsRequest(req,3,r,sizeof(r), /*quiet_fc=*/true);
    }
    if (n > 0) {
      harvestF101_scan(r,n,&HWEL,&BTLD,&SW1,&SW2);
      if (HWEL.found && BTLD.found && SW1.found && SW2.found) break;
    }
    delay(60);
  }

  Serial.println(F("Identification:"));
  if (!(HWEL.found||BTLD.found||SW1.found||SW2.found)) {
    Serial.println(F("  (no F101 entries parsed — dumping first 64 bytes)"));
    hexdump(r, 64);
  }
  printF101Line("HWEL",   HWEL);
  printF101Line("BTLD",   BTLD);
  printF101Line("SWFL #1",SW1);
  printF101Line("SWFL #2",SW2);
}

// ─────────────────────────────────────────────────────────────────────────────
// $14 ClearDiagnosticInformation helpers
// ─────────────────────────────────────────────────────────────────────────────
static bool clearAllDTCs(uint8_t *resp, int rmax) {
  // groupOfDTC = 0xFFFFFF = all records (UDS $14)
  uint8_t req[] = {0x14, 0xFF, 0xFF, 0xFF};
  int n = udsRequest(req, sizeof(req), resp, rmax);
  // Positive response is 0x54
  return (n >= 1 && resp[0] == 0x54);
}

static void doClearDTCs() {
  if (!diagStart()) { Serial.println(F("Session fail.")); return; }
  testerPresentOnce();
  uint8_t r[16];
  if (clearAllDTCs(r, sizeof(r))) {
    Serial.println(F("DTCs cleared."));
  } else {
    Serial.print(F("Clear DTCs failed. "));
    if (r[0]==0x7F && r[1]==0x14) {
      Serial.print(F("NR=0x")); Serial.print(r[2],HEX);
      Serial.print(F(" (")); Serial.print(nrToText(r[2])); Serial.println(F(")"));
    } else {
      Serial.println(F("(no positive response)"));
    }
  }
}

// ─────────────────────────────────────────────────────────────────────────────
// $31 RoutineControl (generic) + wrappers + sniffer
// ─────────────────────────────────────────────────────────────────────────────
static bool udsRoutineCtrl(uint8_t type, uint16_t rid,
                           const uint8_t *params, uint8_t plen,
                           uint8_t *resp, int rmax) {
  // type: 0x01=start, 0x02=stop, 0x03=result
  if (plen > 6) plen = 6; // keep SF safe
  uint8_t buf[4 + 6] = {0x31, type, (uint8_t)(rid>>8), (uint8_t)rid};
  memcpy(&buf[4], params, plen);
  int n = udsRequest(buf, 4 + plen, resp, rmax);
  return (n >= 4 && resp[0] == 0x71 && resp[1] == type &&
          ((resp[2] << 8) | resp[3]) == rid);
}

static bool udsRoutineStart(uint16_t rid, const uint8_t *p, uint8_t l, uint8_t *r, int rm) {
  return udsRoutineCtrl(0x01, rid, p, l, r, rm);
}
static bool udsRoutineResult(uint16_t rid, uint8_t *r, int rm) {
  uint8_t dummy[1] = {0};
  return udsRoutineCtrl(0x03, rid, dummy, 0, r, rm);
}

// Sniff for $31 Start requests for 10 seconds; prints any RID it sees
static void sniffRoutineRIDs(uint16_t ms = 10000) {
  Serial.println(F("Sniffing for RoutineControl ($31) requests for 10s..."));
  uint32_t end = millis() + ms;
  while ((int32_t)(millis() - end) < 0) {
    if (!digitalRead(CAN_INT)) {
      uint32_t id; uint8_t len, b[8];
      if (CAN.readMsgBuf(&id, &len, b) == CAN_OK) {
        // We’re interested in tester->EGS requests (0x6F1) carrying $31
        if (id == PRIMARY_REQ_ID && len >= 6 && b[0] == EGS_TGT) {
          // SF: b[1]=len, b[2]=SID=0x31
          if (((b[1] & 0xF0) == 0x00) && b[2] == 0x31 && len >= 6) {
            uint16_t rid = (uint16_t)((b[3] << 8) | b[4]);
            Serial.print(F("  Saw $31 Start RID=0x")); Serial.println(rid, HEX);
          }
          // FF path skipped (rare for these routines; add if needed)
        }
      }
    } else {
      delay(1);
    }
  }
  Serial.println(F("Sniff done."));
}

// run EGS adaptation reset routine (after setting RID_RESET_ADAPT)
static void doResetAdaptations() {
  if (RID_RESET_ADAPT == 0x0000) {
    Serial.println(F("RID_RESET_ADAPT not set. Run sniff first, then set RID in code."));
    return;
  }
  if (!diagStart()) { Serial.println(F("Session fail.")); return; }
  testerPresentOnce();
  uint8_t r[32];
  if (udsRoutineStart(RID_RESET_ADAPT, RID_PARAM_BUF, RID_PARAM_LEN, r, sizeof(r))) {
    Serial.print(F("Started routine 0x")); Serial.println(RID_RESET_ADAPT, HEX);
    delay(500);
    if (udsRoutineResult(RID_RESET_ADAPT, r, sizeof(r))) {
      Serial.println(F("Adaptation reset routine: result OK."));
    } else {
      Serial.println(F("Adaptation reset routine: no result/NR."));
    }
  } else {
    Serial.println(F("Adaptation reset: NR/denied (security/session?)."));
  }
}

// run diff ratio teach/reset routine (after setting RID_RESET_DIFF)
static void doResetDiffRatio() {
  if (RID_RESET_DIFF == 0x0000) {
    Serial.println(F("RID_RESET_DIFF not set. Run sniff first, then set RID in code."));
    return;
  }
  if (!diagStart()) { Serial.println(F("Session fail.")); return; }
  testerPresentOnce();
  uint8_t r[32];
  if (udsRoutineStart(RID_RESET_DIFF, RID_PARAM_BUF, RID_PARAM_LEN, r, sizeof(r))) {
    Serial.print(F("Started routine 0x")); Serial.println(RID_RESET_DIFF, HEX);
    delay(500);
    if (udsRoutineResult(RID_RESET_DIFF, r, sizeof(r))) {
      Serial.println(F("Diff ratio routine: result OK."));
    } else {
      Serial.println(F("Diff ratio routine: no result/NR."));
    }
  } else {
    Serial.println(F("Diff ratio routine: NR/denied (security/session?)."));
  }
}

// ─────────────────────────────────────────────────────────────────────────────
// Read exactly 5 words (10 bytes) from a DID (used by adaptations)
// ─────────────────────────────────────────────────────────────────────────────
static bool readDidFiveWords(uint16_t did, uint8_t out10[10]) {
  uint8_t req[] = {0x22,(uint8_t)(did>>8),(uint8_t)did}, r[56];
  for (int attempt=0; attempt<3; ++attempt) {
    int n = udsRequest(req,3,r,56);
    if (n >= 13 && r[0]==0x62 && ((r[1]<<8)|r[2])==did) {
      memcpy(out10, &r[3], 10);
      // guard: some stacks add 0x18 0x00 tail in last word sporadically
      uint16_t last = (uint16_t)((out10[8]<<8)|out10[9]);
      if (last == 0x1800) { delay(20); continue; }
      return true;
    }
    if (n>=3 && r[0]==0x7F && r[1]==0x22) {
      Serial.print(F("  DID 0x")); Serial.print(did,HEX);
      Serial.print(F(" negative response: 0x"));
      Serial.print(r[2],HEX); Serial.print(F(" ("));
      Serial.print(nrToText(r[2])); Serial.println(F(")"));
    }
    delay(20);
  }
  return false;
}

// ─────────────────────────────────────────────────────────────────────────────
// Adaptations block (A–E): 0x413A (mbar), 0x4140 (ms), 0x4131 (count)
// ─────────────────────────────────────────────────────────────────────────────
static void readAdaptations() {
  if (!diagStart()) { Serial.println(F("Failed to enter diagnostic session. Aborting.")); return; }

  struct Item { uint16_t did; const char* name; bool isSigned; const char* unit; };
  const Item items[] = {
    {0x413A,"Pressure corrections (A–E)",true ,"mbar"},
    {0x4140,"Fill times (A–E)"           ,false,"ms"},
    {0x4131,"Update counters (A–E)"      ,false,"x"}
  };

  for (int idx=0; idx<3; ++idx) {
    testerPresentOnce();
    delay(30);

    uint8_t d[10];
    if (!readDidFiveWords(items[idx].did, d)) {
      Serial.print(F("DID 0x")); Serial.print(items[idx].did,HEX); Serial.println(F(": read failed."));
      continue;
    }

    Serial.print(F("  ")); Serial.println(items[idx].name);
    const char lbl[5] = {'A','B','C','D','E'};
    for (int i=0;i<5;i++){
      long v = items[idx].isSigned ? (long)be16s(&d[2*i]) : (long)be16u(&d[2*i]);
      Serial.print(F("    Clutch ")); Serial.print(lbl[i]); Serial.print(F(": "));
      Serial.print(v); Serial.print(F(" ")); Serial.println(items[idx].unit);
    }
  }
}

// ─────────────────────────────────────────────────────────────────────────────
// DTCs (UDS 0x19 0x02 0x0C – snapshot of stored DTCs)
// ─────────────────────────────────────────────────────────────────────────────
static void readDTCs() {
  if (!diagStart()) { Serial.println(F("Failed to enter session. Aborting.")); return; }
  testerPresentOnce();

  uint8_t req[]={0x19,0x02,0x0C}, r[240];
  int n=udsRequest(req,3,r,240);
  if(n>=2 && r[0]==0x59 && r[1]==0x02){
    Serial.println(F("DTCs:"));
    int pos=3; bool any=false;
    while(pos+3=2 && r[0]==0x50) {
    Serial.print(F("  Reported: ")); Serial.println(sessionName(r[1]));
  } else {
    Serial.println(F("  (cannot probe directly; rely on last changeSession())"));
  }
}

// parse "R3.15" to set ratio & save; or "R?" to show current/EEPROM
static void handleRatioCommand(const String& line){
  if (line.equalsIgnoreCase("R?")){
    float v; if (loadDiffRatio(v)) { Serial.print(F("Saved diff ratio: ")); Serial.println(v,3); }
    else Serial.println(F("No saved diff ratio."));
    return;
  }
  if (line.length()>1 && (line[0]=='R' || line[0]=='r')){
    float v = line.substring(1).toFloat();
    if (v>2.0 && v<5.5){ saveDiffRatio(v); Serial.print(F("Saved diff ratio: ")); Serial.println(v,3); }
    else Serial.println(F("Out of range. Use e.g. R3.15"));
  }
}

// ─────────────────────────────────────────────────────────────────────────────
// setup / loop
// ─────────────────────────────────────────────────────────────────────────────
static String g_line;

void setup() {
  Serial.begin(115200);
  while(!Serial) {}
  pinMode(CAN_INT, INPUT);

  // Some MCP2515 clones print their own banners; we just init cleanly
  if (CAN.begin(MCP_ANY, CAN_BPS, CLOCKSET) == CAN_OK) {
    CAN.setMode(MCP_NORMAL);
    Serial.println(F("CAN @500k ready"));
  } else {
    Serial.println(F("CAN init FAILED"));
  }

  Serial.println(F("\nBMW F10 8HP EGS tool"));
  Serial.print(F("Requests on 0x")); Serial.print(PRIMARY_REQ_ID,HEX);
  Serial.println(F(" → expect replies on 0x618"));
  Serial.println(F("Enter: 1=DTCs  2=Info  3=Adaptations  c=clear DTCs  r=sniff RIDs  a=reset EGS adapt  d=reset diff ratio  v=verbose  s=session"));
  Serial.println(F("Or line cmd: R?  /  R3.15  (then Enter)"));
}

void loop() {
  while (Serial.available()) {
    char ch = Serial.read();

    // Handle newline for line commands (like R3.15)
    if (ch=='\n' || ch=='\r') {
      if (g_line.length()) {
        handleRatioCommand(g_line);
        g_line = "";
      }
      continue;
    }

    // If it's a single-key menu and no pending line text, act immediately
    if (g_line.length()==0) {
      if (ch=='1') { Serial.println(F("\n--- Read DTCs ---"));         readDTCs();        continue; }
      if (ch=='2') { Serial.println(F("\n--- Read Info ---"));         readInfo();        continue; }
      if (ch=='3') { Serial.println(F("\n--- Read Adaptations ---"));  readAdaptations(); continue; }
      if (ch=='c') { Serial.println(F("\n--- Clear DTCs ---"));        doClearDTCs();     continue; }
      if (ch=='r') { Serial.println(F("\n--- Sniff Routine RIDs (10s) ---")); sniffRoutineRIDs(); continue; }
      if (ch=='a') { Serial.println(F("\n--- Reset EGS Adaptations (Routine) ---")); doResetAdaptations(); continue; }
      if (ch=='d') { Serial.println(F("\n--- Reset Diff Ratio (Routine) ---"));      doResetDiffRatio();  continue; }
      if (ch=='v') { g_verbose = !g_verbose; Serial.print(F("Verbose: ")); Serial.println(g_verbose?F("ON"):F("OFF")); continue; }
      if (ch=='s') { Serial.println(F("\n--- Session probe ---"));     showSession();     continue; }
    }

    // Otherwise accumulate into line buffer (for R?/R3.15 etc.)
    g_line += ch;
  }
}

/*  ────────────────────────────────────────────────────────────────────────────
    IMPLEMENTATION NOTES (for future you)

    1) F101 only for identification
       Your EGS answered 0x7F 0x22 0x31 (Request Out Of Range) to direct DIDs.
       Many BMW EGS variants expose HWEL/BTLD/SWFL only inside the F101 blob.

    2) Parser strategy
       Some stacks include a length byte (always 0x03 here) between DID and data.
       Others put three version bytes right after the DID.
       We scan the entire payload and accept either pattern for known DIDs:
         HWEL  0x022A
         BTLD  0x0C7C
         SWFL1 0x0A81 or 0x0C7D
         SWFL2 0x1CDA or 0x27F0

    3) ISO-TP pacing
       If the EGS is finicky, quiet FC (BS=0, STmin=0) avoids chokes on long
       payloads. We try normal FC first, then quiet FC automatically.

    4) Auto-prototype gotchas
       We define struct FItem BEFORE any functions that mention it and also
       declare those functions 'static'. This prevents the Arduino builder
       from inserting prototypes that reference an unknown type.

    5) Diff Ratio
       Not part of F101. We probe a few plausible OEM DIDs and attempt simple
       decoders (u8/10, u16/100..1000, be-float). If none hits a sane range
       (2.2–4.5), we display "(unknown)". You can type R3.15 (example) once;
       it’s stored in EEPROM and displayed on subsequent Info reads.

    6) Clearing DTCs
       Uses UDS $14 with group 0xFFFFFF (all DTCs). Positive response is 0x54.

    7) Routines ($31)
       'r' sniffs any StartRoutine (0x31 0x01) SF requests to 0x6F1 for 10s
       and prints the RID (and you can add payload logging if needed).
       Copy the RID(s) you see into RID_RESET_ADAPT / RID_RESET_DIFF above,
       re-upload, then 'a' / 'd' will replay those routines.

    8) Security
       RoutineControl may require security access on some firmware. If you get
       NR=0x33/0x35, you’ll need to implement $27 security (beyond this scope).

    9) Verbose
       Default OFF (cleaner). Press 'v' to toggle logs on the 

Other Messages

There are many more IDs for now i have gathered this info: 

This is enough for now. Due to a lack of time the project is parked here. If you want to dig deeper, please go ahead and share your findings in the comments. There’s a lack of F-series CAN bus logs on the internet—many have been taken down for one reason or another. I have nothing against the controllers on the market, but I think it’s important to have a community-based open-source controller too.