Newer BBBs sometimes throw mmc init error (-19) with older U-Boot

Greetings,

We’re in the process of shifting to a newer U-Boot build, but currently are still loading an older 2015.1 U-Boot on our BBBs. We noticed recently that some BBBs from a recent order were slow to fully boot up to Debian. Connecting directly to the serial UART of the BBB, we saw the following error immediately after power is applied:

U-Boot SPL 2015.01-00001-gb2412df (Jan 29 2015 - 15:01:06)
spl: mmc init failed: err - -19
### ERROR ### Please RESET the board ###

Following that last line of output, the BBB will sit there for ~50 seconds, and then appears to reset (maybe a watchdog kicking the CPU or board?) and it tries to boot again. Sometimes this error is hit on the subsequent reset (or even hit on multiple subsequent resets; I’ve see it happen 4 times in a row before the U-Boot boot succeeds). And maybe ~20% of the time we do not see this error at all following initial power-on.

We noted that the BBB boards we see this on have newer Kingston MT32 eMMC parts, whereas the boards we’ve been using for years without this error have MK27 eMMC parts. Some digging into our version of U-Boot source code indicates that the -19 error in mmc init is a timeout error. We’re still examining source differences to see if there is some eMMC-related handling in the newer U-Boot that ours is missing, but we did casually note that the mmc init routines in our older version of U-Boot call hang() on some errors (which triggers that ### ERROR ### line of output and the board going idle) where the newer version of U-Boot (2019.4) currently shipping on BBBs we recently received from DigiKey does not appear to call hang() in similar circumstances.

Once the BBB has successfully booted Linux (it will eventually succeed, but might hit a few of those -19 U-Boot errors along the way), there do not appear to be any mmc errors or unusual messages in the kernel dmesg logs, nor anything “above” that (e.g. filesystem errors).

Was curious if anyone else has encountered this scenario or had any thoughts/ideas. TIA!

Oh this smells like mmc timing issues… You dig into the am335x driver, but 2015 is around when everything was starting to hit mainline u-boot.. My v2019.04 branch here is the closest in compatibility to the old old u-boot: Files · v2019.04-bbb.io-am335x · BeagleBoard.org / u-boot · GitLab

After that branch, u-boot started moving more and more away from board files to device-tree’s .. So for a long time dhcp boot in u-boot was broken.. Our next big stable branch drops even more compatibility: Files · v2022.04-bbb.io-am335x-am57xx · BeagleBoard.org / u-boot · GitLab

My v2019.04-bbb.io-am335x branch is pretty safe to drop into any old/ancient BBB u-boot..

Regards,

Appreciate the thoughts/insights and suggestions, Robert. TY!

Just wanted to circle back and share an interesting datapoint…

I’m only seeing the described error above (spl: mmc init failed: err - -19) with GHI BBBs that have the MT32 eMMC part. Related to my recent post about a SEEED Studio BBB I had bought last month from DigiKey that has a MT32 eMMC part on it, the older 2015.1 U-Boot code we’re using does NOT encounter any mmc init errors at boot time, it just boots up “as usual” for us into Linux, no issues. This was counter to my expectations and just wanted to share here in case anyone else bumps into a similar situation.

UPDATE to my above comment: I recently had the opportunity to test our older 2015.1 U-Boot code across five (5) of the newer SEEED BBBs which have the MT 32 eMMC parts (date code 202410), and they show the same spl: mmc init failed: err - -19 errors at power-on as the GHI BBBs also using the MT 32 parts do when booting 2015.1 U-Boot. It’s unclear to me why my first testing on my MT 32 SEEED BBB did not encounter those errors, rather booted up fine twenty times in a row from a fully powered off state without error. I’ll be spending some time this weekend looking into the branch you mentioned above, @RobertCNelson (thank you for that!).

Sayya @RobertCNelson, a penny for your thoughts if you have a moment…

Per the above conversation, we are working to update our very old U-Boot (2015.01) with updated MMC code/timings from your 2019.04 branch, but we’re encountering quite a few compile-time errors (not a surprise, with 4 years delta between the two branches, but still quite a few changed data structures, undefined things, etc.). We had also tried to manually review the MMC code in your 2019.04 branch and compare it to our 2015.01 branch, then incorporating noticed changes in mmc_init() (and the functions it calls, as well) found in your newer branch into our older code, but so far haven’t made the change that fixes the -19 “TIMEOUT” boot issue for us.

Was curious if you had any thoughts/guidance on our approach?

Was also curious if you thought the 2019.04 U-Boot could successfully boot our old kernel (3.8.13-bone86.2b) and Debian (7.8)?

T{IA,YVM}

How old/new of system are you building u-boot? it does not need to be the same version as native..

Regards,

For building, we’ve been using an Ubuntu 14.04 x64 VM (via cross compiling) for our 2015.01 U-Boot, and using an Ubuntu 18.04 x64 VM for building (via cross compiling) your 2019.04 U-Boot branch. Both of these builds fine and install OK onto our BeagleBone Black boards. The compilation errors I mentioned occur when I (perhaps too naively) copied the drivers/mmc/* files from your 2019.04 into the 2015.01 drivers/mmc directory.

Oh my Gosh! Yeah, no bid on that!

Regards,

Hah, fair enough. :slight_smile: Seems I had misunderstood the gist of your earlier comment further above:

Was the suggestion there that fully moving off our 2015.01 to your v2019.04-bbb.io-am335x branch would likely be a reasonable path?

Yeah my advice is to upgrade directly to the v2019.04-bbb.io-am335x branch, as that has the most compatibly with any version of u-boot we shipped on the BBB before v2019.04… The later branch (v2022.04-bbb.io-am335x-am57xx) you lose dhcp/eth support inside u-boot..

Regards,

Understood, TYVM!

Heyya @RobertCNelson, a belated quick note to say TYVM for the suggestions+pointers! This worked like a champ for us. :slight_smile:

Cheers!