Your steps and logging of what’s happening is excellent … thank you! Unfortunately, this could not exclude my worst fears of a difficult-to-track bug. 
Linux still mounting the media
Did I understand correctly that your check script only logged open handles under /media/matty? (e.g., why not /dev/sdb1?)
From the kernel log:
[32659.577168] FAT-fs (sdb1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
This strongly suggests that the FAT file system was mounted, and the Linux system processed enough of the volume to know that the file system was FAT, and that the dirty bit was set.
Odd capacity ...
Later, the following interesting sequence occurs:
| Sectors | bps | WP | timestamp |
|---|---|---|---|
47824 |
2048 |
yes |
32688.939668 |
0 |
32691.329673 |
||
47824 |
2048 |
no |
32691.339663 |
0 |
2048 |
yes |
32692.054562 |
16 |
512 |
??? |
32692.061164 |
47824 |
2048 |
no |
32692.251649 |
The troubling aspect of the above is the reporting of 16x 512 byte sectors. I don’t recall anyplace where the BP5 should report the NAND as being 8k.
There are also a number of commands failing with Invalid Command Operation Code. However, the kernel log didn’t log what the actual CDB (command descriptor block … at least that’s what we called it) was.
I can think of two private binaries that, if created, might verify my suspicion that the Linux host is still mounting (and writing to) the media.
- First private binary that ALWAYS makes the exposed FAT volume read-only for the host. With this private binary, you could perform the same steps as above, only the host would never be able to write to the media.
- If first private binary does NOT prevent the corruption, then the issue is 100% on the device side, which in a way is good.
- If first private binary prevents corruption, then it’s the difficult situation to track down. Continue to needing the 2nd…
- Second private binary could log all storage CDBs incoming (16 bytes each), timestamp, and the Sense/ASC/ASCQ reported (<16 bytes each). Sort of a micro bus analyzer. Maybe also log writes that were initiated by the firmware to see how they interleave (if at all). Maybe also log first 256 bytes of data when writing to the boot sector (it’s a fixed location). Need to think about the format to be used here.
a. If this private binary shows no write commands coming from the host, then the issue is 100% on the device side, which in its own way is good.
b. Otherwise, some deep decoding of that log to understand who’s writing a corrupt root directory sector, and then walking backwards to why it would be written, is needed.
If other folks have better ideas, let me know. Otherwise, I will create a branch for you to sync/build from.