LED mode updates and syntax overhaul

[ian]

This week I’m going through each mode to make sure everything is cleaned up and there’s a good firmware available when hardware starts to arrive.

This weekend I went through the LED mode. It supports the common one and two wire LED protocols.

I noticed an issue with the single wire protocol. It’s time based, and anything more than 50uS between frames is considered a ‘reset’. The firmware currently takes about 160uS to process and output commands typed in the terminal. This results in the first LED being addressable when individual commands are used.

For example:
0x00ff00 0xff0000
would ideally turn the first LED red and the second led green. Instead, the first LED takes both commands and ends up green.

However:
0x00ff00:32
Sending the command with a 32 repeat lights 32 LEDs, this is because the repeat code has less overhead.

So, I made one major change in the code to make way for a future rework of everything:

1. To ensure the speed of repeat, the repeat code now sends the commands, and then later fills the USB output ring buffer with display data.
2. Because of this, we now need an array to hold the incoming data bytes so we can process them to the display later. This isn’t an issue for LED mode (no incoming data), but it will be an issue for SPI (which is the next mode I’ll rework).
3. We’ll need a way to detect errors (UART frame, I2C no ACK, etc), and flag where it happened

I’m thinking through a sane way to restructure all this, and I think it may be the right time to move to a sort of compiler/bytecode type thing.

1. Bus syntax is compiled to a simple bytecode that is then run full speed.
2. Incoming data is stored in a second array
3. When the transaction is finished (or the buffer is full) a post process function pushes all the output to the terminal

I did this previously when we had the FPGA/STM32 version, it’s doable but it makes writing modes much less flexible.

There are a couple things that could make it easier

1. We allow flexible mixing of bus syntax (read, write, etc) and menu commands. This has always been kind of a pain and really dictates several aspects of how we structure the parser. It really only makes sense in terms of the power command and maybe MSB/LSB switching. I would propose that we make menu commands more like a proper command line program execution. For example

/m

for mode? This would free up the possibility to have more extensive commands for things that previously would have been scripts:

/dump 24lc1024 > dump.bin

That would also give us a way to run scripts from the SD card. Or record macros, etc.

/run script.txt

This also frees up <> from being user macros, as I want to use them for a load/dump syntax command that reads/writes from the SD card. for example:

[ 0x03 0x00 0x00 >:2048]

would dump 2048 bytes from an EEPROM to a file on the sd card.

To summarize:

1. Bus syntax is processed to byte code which is then executed at full speed without gaps in output. The terminal display is processed afterwards.
2. Bus Pirate configuration commands would be limited to one per line, and could no longer be mixed with syntax. Probably prefixed with / or similar. Then we could do something like:

/W (setup/configure the power supply with menu)

W [0x03 0x00 0x00 r:16] w (turn on PSU with config settings, read from eeprom, turn power off)

This separates the configuration of the bus pirate from the syntax that controls the peripherals, which seems like an improvement overall.

In the photo I’m dogfooding all our stuff while I work through each mode. The AUX cable connects to a logic analyzer without trying to get an additional test hook on the pin connections. The milled breadboard pins connect the probe cable to the LED strip without fuss.

[ian]

Quick thoughts on the syntax update:

1. Bus Syntax will use an indicator such as / to tell the bus pirate we want to send something to the bus
2. Configuration commands (w/W/p/P, etc) and os-like commands (ls/cd/mkdir/rm/etc) will be one-per-line with configuration options following. Similar to every other command line I’ve used. Things that are currently macros like (1) can be changed to script-like commands with options.
3. Scripts will be ANSI text files (like .bat or .sh) that the bus pirate executes from the SD card. This functionality come later.

When the mode cleanup is done 1 &2 will be my next priority. Then I’ll work on the compile to bytecode stuff.

[grymoire]

This sounds great! It’s like a high level language and an ASM language, right? This is a great way to really understand the system.

It would be terrific if there was a way to save, restore and replay entire setups, with comments stored as well.

[ian]

Thanks for stopping by!

@grymoire

It would be terrific if there was a way to save, restore and replay entire setups, with comments stored as well.

Yes, definitely. Now that we have such a big chip, so much flash and an SD card I think we’ll be able to add really cool stuff. It’s night/day compared to BP v3 on a tiny PIC chip.

I’m struggling through updating the UART mode at the moment, and its really apparent that preprocessor ->bytecode->postprocessor is the way to go. Unfortunately I need to get UART and I2C workable in the current form, but immediately after that I’m excited to start tearing stuff apart.

[ian]

Today I’m going to make a huge mess of things. This is the “before” view of the command [0xff 0x00 0xaa 0x55]. First I’ll tear up the command line processor to make a kind of syntax compiler. Then, I’ll work up a simple statemachine to process the compiled syntax. Finally, a post-processor that shoves the results into the UI output buffer. I’m guessing this takes a few days, but I hope to have an ‘after’ photo of some sort later today.

[ian]

From 232us down to 16us, that’s only 12us if you subtract the 4us clock low period. That’s pretty tight, it might be made a bit more efficient. More likely it will get slightly worse as the statemachine grows more complex. It’s not as good as the FPGA version we did a few years ago, but this looks much better than before and eliminates the issue with timing in the 1 wire LED library.

Just a little test to see how it goes, it’ll be several days work to implement it.

[ian]

Today I’m doing the very unglamours work of splitting the syntax parsing from command parsing. To make it a bit more fun I started adding commands to work with the SD card. This is the ‘ls’ command that lists the files on the card.

Eventually this will lead to saving sessions to disk, but even more fun, running scripts. Some thoughts on scripts:

1. One line per action, like a batch file
2. # or // to denote comments.
3. Comments will be displayed in the terminal, so it can be used as a tutorial.

[ian]

Wow! So happy with the progress. I moved all the configuration commands into the new command system. Anything that has arguments (such as = 0xff to convert number formats) will need to be updated to accept arguments instead of parsing their own. This is a HUGH improvement over trying to parse out the various values in each function.

Also got the linux commands all working. It needs to be made pretty and in color, but it does indeed work. cat to print a file shown in the screenshot.

This is all around a huge improvement:

1. Separation of configuration commands (speed doesn’t matter) from bus syntax (speed matters)
2. Clean and consistent passing of optional arguments to config commands
3. Less overhead for bus syntax which has fewer argument options.

One thing I noted is that exfat was enabled and it shouldn’t be (patents, also limitations in fatfs to track directory). Now that it’s disabled and I’m using a FAT32 SD card, windows discovers it very slowly, often in the middle of doing something else, so we’ll need to add a spin lock and a way to mitigate that. I’m not sure if this is because I’m using a 64GB card formatted to fat32, I’ll find a smaller one and give it a try.

Anyone want to work on an in terminal editor like nano? VT100 has a “second display” command that brings up a new screen while preserving the original content. I assume this is how it is done.

[ian]

Digesting todays progress:

• Totally shifted my perspective from v3’s single letter commands with options, to programs with parameters. I’m going to lean into that and add longer form names. For example G activates PWMs on a pin, I’ll add a longer form pwm command that can be passed parameters instead of using the menu:

pwm 12mhz 50% 7

• Tomorrow I’m going to make a opt arg struct that describes the arguments, parser type, range limits, and help text for the command. This will move 99% of parsing out of the lower functions and standardize it.

• Syntax is still a slightly special case. Tomorrow I’ll get that prototyped. I’m thinking the protocol functions can be made non-blocking, with any needed blocking checks happening in functions where it’s really needed. For example no need to block before sending data to the line (assuming buffer space) but CS changes for example do need to block until the buffer is empty or the timing is off. That will get us even tighter times between bytes.

[ian]

It’s a glorious mess, but everything is coming together well. I think the basic prototype of how the new system goes together is almost there. Tomorrow I’ll build out the syntax compiler and run bits, then monday and tuesday will probably be implementing the changes and scrubbing up the billion consequences of such a drastic change. Very happy with it. 2us between data writes, but a lot of that is evidently built into the SPI peripheral, it can’t really get any faster on the code side.

[ian]

All the commands have been reworked to use the new opt args command line system. It seems pretty solid. After some more cleanup I’ll post a “preview” firmware here if anyone wants to try it out.

• ls/cd/mkdir/rm/cat - new linux like commands for working with the SD card
• All existing (configuration) commands are now integrated into the new command line parser
• All commands have a -h help option (though help needs to be written)
• Commands with a ‘.’ option (g/G,f/F,v/V) now use a space. f.1 becomes f 1 to measure the frequency on IO 1

To do list from here:

• Continue with the new syntax compile/run/report system. It is currently pretty bare bones
• Add multiple parsers (and backup options menu when user enters no or wrong options) to the main processing, so all parsing and menu options happen in a standard way in the main parsing function (and not in individual utility functions). This will enable full command line syntax like ‘pwm 12mhz 50% 7’.
• Write help for commands
• Update main ? help menu with new commands/syntax