Drastic changes to USB MSC storage interface

General
1

Gathering all the threads from various places about the USB storage issues.

I’m pretty sure there are several more. I will note one bug effecting me since the flurry of updates to the MSC switching:

  • The first time I connect to Windows the serial port won’t open.

A big chunk of the issues are under Linux distros. I am totally unqualified to debug that, but a bunch of people with a lot of experience have tried and it seems like things are still buggy with the report of SPI flash command 0 length files.

Here is what I propose (or rather will dictate from above with a git push):

USB
USB CDC (serial port) Closed Open
USB MSC (disk) Enabled Disabled

The USB MSD drive will no longer be visible when the serial port is open. This should help with the errors caused under Linux when the drive goes from Read/Write to Read Only. It will also stop the unexpected by users behavior that the device connects/disconnects to refresh the file list on the host OS.

A big bummer here is that the current setup works great for me under Windows. If I dump a flash to a file I can just drag it off the drive and move on with my life. Now there will be extra steps.

I will try to add a config option for a degraded mode compatible with whatever Linux irregularity causes this issue. Assuming this change does actually fix it under Linux, which is yet to be seen.

More drastic

If this still isn’t a solution, then I will use the button to force to USB MSC mode, otherwise the drive will not be visible.

MTP

MTP is not an option unless someone volunteers to write an MTP driver for TinyUSB and then support users through all this.

1 Like
2

I would appreciate any feedback from the folks who contributed to the previous solution. Did it end up fixing the issues for you?

3

@ian - I have been running 089e906 (2024-11-12T13:33:18Z) and I haven’t had any issues. I’m on Ubuntu 22.04.5, I use minicom for serial I/O to the BP, and typically use CLI for moving/copying files to/from the storage.

For reference, I started the Bus Pirate 5 resetting itself, causing storage corruption post.

Matt

2 Likes
4

Thanks for the info Matt, I really glad to hear it is now working for you!

2 Likes
5
Comprehensive Issue Summary -- Light Background Reading

As you know, I’ve written up a comprehensive overview of how storage works. It also includes a link to the FAT32 file system specification.

I am an expert from my work in the Windows low-level storage stack for ~10 years, from my detailed knowledge of the FAT and exFAT specifications, and from my work improving ghostfat (UF2 bootloader’s read-only FAT generator), …

For anyone not familiar with why the solutions here might result in corruption, please familiarize yourself with those resources. If there are unclear scenarios, please ask in that forum thread, to allow this thread to be focused on Ian’s work.

Hi Ian,

I agree MTP is not a short-term option.

Informed Decision

I want to ensure that you make an informed decision. I will support your decision fully.

The above proposal would significantly improve the situation of when multiple initiators might write to the NAND. At the same time, it does still have scenarios where multi-initiators could cause corruption. Off the cuff, if the host caches any modification to the media, and the firmware writes to the media (while the serial port is closed), there will be a mismatch between the host cache and NAND FS. I agree that this is an edge case.

As the user base grows, even a 99% solution will still have the occasional user impact. User education here can be helpful to mitigate. For example, document that one should not have long-running logging running when disconnecting the serial port?

While users would likely only hit corruption once, this does raise a risk of reputational damage. This is mitigated by the user base tending to be more technically savvy and the marketing of the device as a hacker tool. Thus, it’s not a technical question, but one of your risk aversion.

Dynamically Changing USB Descriptors

TinyUSB supports multiple device configurations by dynamically changing USB descriptors.

Why is this interesting? It would entirely avoid the problems associated with reporting the proper Sense/ASC/ASCQ errors. The MSC would either be exposed to the host, or simply not … No emulated missing media, no fake media that is the cause of corruption.

Architecturally, this would appear to simplify changing between possible solutions: current solution, the above proposed solution, the modal switching option, or any other solution later implemented.

Adafruit found a way to dynamically add interfaces, and Hathach says this works, by then forcing re-enumeration. So, this is definitely something that TinyUSB itself supports.

Q: Would you consider moving to dynamically selecting which USB configuration is enabled?

1 Like
6

maybe add a “mode” that blocks the buspirate and goes to a sort of “lock screen” to allow temporarily control from host to the storage without closing the serial line. leaving that screen detaches the device again and allows using the files from the BP side again

1 Like
7

Thank you everyone for the feedback via forum and PM.

Ok, this is very interesting.

Can we have our cake and eat it too? I like options :slight_smile: How about:

  1. Configuration switching using the button at startup (default, safe option?)
  2. Active mode (current setup)

This way we keep the current setup that seems to work for the vast majority, while having a (hopefully) bullet proof alternative when it doesn’t. I’m haunted by a nightmare scenario of endless debugging for different combinations of PC hardware and operating systems holding back actual progress on the firmware.

2 Likes