Losing USB on latest firmware, any tips for debugging?

General
1

I keep losing my connection to the serial port on my MacBook Pro i9 2019. Once that happens I can’t reconnect, and eventually the device disappears (and also the drive disappears with the usual popup scolding me for not unmounting it first.)

Sometimes it happens right away, sometimes it works for a while.

HiZ> cu: Got hangup signal

Sometimes I am able to reconnect but I receive no output and it eventually hangs up again. When it’s hung like that, pressing the One Button sometimes makes it produce a little bit of output and immediately hang up.

It seems like it started after I upgraded from ci-buspirate5-main-6f1e888.zip (April 13 2024 )to ci-buspirate5-main-a5d0e8a.zip (May 12 2024). (And now running a debug build on b1e8b90 that still has the problem.)

I haven’t used OpenOCD before, but I had been wanting to mess with it for some other projects anyway. (And I also wanted to debug the issue where I can’t connect after my Mac sleeps.)

I follow the instructions at OpenOCD/RPi Debug Probe Setup

I don’t have the debug probe, so I put it on a regular Pi Pico. I used the milled pins I ordered with the BP (and some female jumper cables I had laying around that I got from Adafruit a while back) to connect to the debug probe on the back.

I see some references in the code to a serial debug console but I haven’t figure out how to connect to that yet.

For whatever reason, I thought OpenOCD was a little bit more magical and it would send me errors or something. Seems like for the most part, it’s just a way to connect gdb.

Which is still pretty awesome, but I quickly remembered that I need to ask gdb the right questions to get useful answers. So I started reading through the code. And then I ran out of time.

I read through the code a little bit more before work this morning. I saw some references to IRQs, so one thing I’ll check tonight is that the IRQ doesn’t get stuck disabled somehow.

Any other tips for what to check would be helpful.

I did pull down the code and configured / built with

cmake -S . -B build -DPICO_SDK_FETCH_FROM_GIT=TRUE -DCMAKE_BUILD_TYPE=Debug
cmake --build ./build --target bus_pirate5_rev10

(I also setup VsCode for cmake. I might try to clean up my config and contribute that back.)

Also, I tried to get more logs from the Mac side. I was able to get logs using the Mac built-in Console utility. Does anyone know of a better way to do that? It crashed at least once, and it’s a bit of a firehose. (Or maybe I don’t know how to use it correctly).

I haven’t finished going through those logs yet, but one relevant piece is:

error   16:33:37.748986-0500    kernel  AppleUSBIORequest: AppleUSBIORequest::complete: device 29 (Bus Pirate 5@14300000) endpoint 0x05: status 0xe0005000 (pipe stalled): 0 bytes transferred
error   16:33:37.749803-0500    kernel  AppleUSBIORequest: AppleUSBIORequest::complete: device 29 (Bus Pirate 5@14300000) endpoint 0x05: status 0xe0005000 (pipe stalled): 0 bytes transferred
error   16:33:37.750632-0500    kernel  AppleUSBIORequest: AppleUSBIORequest::complete: device 29 (Bus Pirate 5@14300000) endpoint 0x05: status 0xe0005000 (pipe stalled): 0 bytes transferred
2

Answering a couple of my own Mac questions:

1 Like
3

Thank you so much for the detailed debug info. If you revert to the ci-buspirate5-main-6f1e888.zip firmware, does the issue still occur?

There have been 3 updates to the USB ID descriptors in the past few weeks, I wonder if this is related? Not certain, just a thought.

4

I haven’t tried reverting yet, but that’s a good call out.

The problem was happening for me most of the time Sunday afternoon. So I was unplugging it and replugging it in a lot.

It didn’t happen at all Monday night, but I didn’t try to make it happen, so I only plugged in it once.

So I’m not certain how consistent / inconsistent the problem is yet. I’ll try power cycling it more next time, if it’s more consistently breaking I’ll try reverting. (If it’s not consistent enough, I’m not sure reverting will prove anything…)

I’ve been doing web development as my day job lately, but before that I used to debug pthread deadlocks in some C programs. (On servers, not embedded or anything.) So in some ways this is all new to me, but in other ways it’s kind of familiar. At least I have some experience with inconsistent bugs.

5

I like to call such things, “Heisenbugs”.

1 Like