Command line update discussion (tab completion, hierarchical / mode-sensitive, ...)

Continuing the discussion from OTP whitelabel options for RP2350 boards:

A few items for discussion:

  • Command reorganization
    • Hierarchical command definition
    • Common parsing of command line parameter types
    • Simpler extension with new sub-commands
    • Tab-completion … in long-term managable form (no per-command work)
Example using (global) OTP commands

Given the desire to add the following permitted commands:

otp dir [entryTypeId]
otp dump 3xbytes (startRow) [rowCount] [--show-votes]
otp dump n_of_m (startRow) (rowCount) N [--show-votes]
otp dump raw (startRow) [rowCount] [-q|--quiet]
otp dump ecc (startRow) [rowCount] [-q|--quiet]
otp dump auto (startRow) [rowCount] [-q|--quiet]

At root level, would add otp as a string to the global (1st-level) table, which points to an OTP-specific 2nd-level table (same structure as the 1st-level table). This OTP entry would indicate the command(s) is globally available (should be parsed regardless of mode).

The 2nd-level OTP table would have two entries: One for dir (otp dir if parsing the whole hierarchy), which points to a command structure with options.
One for dump, which points to a 3rd-level table (again, same structure as 1st/2nd levels).

The 3rd-level otp dump table would have five entries, each pointing to a command structure with options for the corresponding variant.

A command structure would support listing mandatory and optional parameters, provide help for each parameter, examples for the command, etc.

Example for a specific mode

Given the desire to add the following permitted commands:

mode irplank [--use_prior_values]
mode irplank [--show_prior_values]
mode irplank [--tx_hertz N] [--tx_hertz N] ...

This indicates that there’s a mode supporting buspirate scripting.

Adding a mode would NOT modify the global command table, but only modify the mode 2nd-level table, by add an entry for the command irplank (with its unique options to configure that mode).

Note that doing this would remove the overhead of having all the prompting for settings. The help text for each option would define what those options do. The --use_prior_values and --show_prior_values can be common code for all modes, reducing complexity of adding things further.

Common Parsing reduces per-addition overhead

Some default parsing becomes possible for each hierarchy. For the mode, the display / setting / saving of parameters can become automatic (with opt-out).

For all commands, a -h / --help switch can parse the same structures and display the help that was defined … right next to its implementation.

Optional parameter? Parameter with optional / required arguments? Flags to permit arguments to be integer / hex / binary / bpscript token(s) / etc.? 90% declarative, and thus self-documenting.

As another example, for modes, it becomes possible to automatically document which bpscript tokens have meaning. Which modes ignore / implement:

  • protocol_start
  • protocol_start_alt
  • protocol_dath (“set data line high”)
  • protocol_write
  • protocol_read
  • protocol_datl
  • protocol_dath
  • protocol_dats
  • protocol_periodic
  • protocol_... etc.

The information is already in the configuration structure (note: I would have the configuration using nullptr to select a do-nothing option, adjusted to the default do-nothing function when the mode is entered). In other words, this can help to automatically document a substantial portion of the various modes, AND tab completion would become way more powerful by excluding options that the current mode doesn’t actually support.

Tab completion

Basically, processes each character of terminal input (not line-by-line). While at the CLI, it uses the command tables to discover the valid token(s) that might complete the current partial user-input-line. When tab key is pressed, if only one valid option to complete the partial line exists, it completes the entry, and adjusts the “current” table it looks at to that next-level table. If multiple entries could complete them, it displays those options (compactly).

While inputing bpscript, this would be a distinct tab completion process. First, it could be dependent on the mode’s configuration, such as to exclude from the list of allowed tokens those that the current mode does not support.

If this is pulled off, it will add a level of polish and shine to the CLI that reflects a much more mature (and expensive) product … with only a one-time-engineering cost.

I love the project. I see limitations and trouble coming for the CLI.

How can we make the CLI much more powerful and usable, while making it easier to add new commands / subcommands?

Should there be a way to mark a command / tree of commands as tied to a specific type of plank attached? (IR, smart card, etc. … each of which relies on hardware on the plank to do more)

[Edit – Add this para] Perhaps the structure listing the commands is universal, with a flag indicating if the command is available. e.g., OTP function availability would return false if running on RP2040. Why? Less #ifdef mess outside of the implementation (where it is unavoidable).

All the above is described in concept form … actual code comes after defining how it should work to be pleasant and user-friendly.

Oh, and that whole tab-completion thing has zero runtime overhead when not using tab. Winning!

3 Likes

Tab completion is amazing! Please make it happen if possible. Are you going to implement it with a state machine or something like that?

I have another proposal—let me know what you think. In many debuggers for exploiting, we do it like this:

Pressing ENTER without any input repeats the previous command!

This way, something that was previously useless now has a purpose. Do you think it’s a good idea, or does it seem weird or unnecessary?

Nah, it’s a bad idea. People copy-paste commands from the web, and if an accidental ENTER sneaks in, they could mess something up… Nah, forget the ENTER thing.

2 Likes

Are you interested in a specific cli package?

Edit: the up arrow does flip through the history as well.

How it happens is TBD… It depends on interest level, because such a sweeping change would likely require deep changes to how commands are defined … and maybe result in a less cryptic UI?

I tried to give a (too-brief) summary above in the “Tab Completion” details block … did you know that block is expandable / collapsible? (I didn’t when I first saw one…) Here’s my off-the-cuff attempt to describe what happens from starting state, where I will take some liberties by re-organizing how some commands work.

Command re-org for this example

io
    powersupply ( ON | off )
    pin (0..7) ( READ | low | high )
    pullup [--pinmask 0bXXXX'XXXX] ( ON | off )
    ...

cool ...

config
    language
    ansi
        colormode ( 16M | 256 | disabled )
        toolbar ( ENABLED | disabled )
    display
        mode ( DEFAULT | oscilliscope | scope )
        screensaver ( 5MIN | 10min | 15 min | disabled )
    pixels
        effect ...
        color ...
        brightness ...
    button
        ...

(storage | stor)
    ls
    cd
    ( md | mkdir )
    ( del | rm )
    ( rd | rmdir )
    format

macro
    load -f (filename) 
    ...

mode
    i2c [ -speed N ] [ -clockstretching ( OFF | on ) ]
    pixel [ -type ( WS2812 | SK6812 | Neopixel | APA102 | SK9822 | Onboard ) ]

// Commands valid only when in mode==I2C
i2c
    scan [-v | --verbose]
    device
        si7021
        ms5611
        tsl2561
    ...
Tables structure concept...

typedef int32_t parse_error_t;
typedef struct _command_t {

    // Critically, the token to be matched
    const char * token;

    // Points to the start of the next level's table
    // which allows tab-completion with low overhead?
    // if this is NULL, when it's a single command
    const struct _command_t * next_table;

    // Dynamic determination of whether (if it's a command)
    // it is currently usable ... e.g., for mode-specific commands
    bool (*usable)(void);

    // When "enter" is pressed and the token matches,
    // AND there's no table ... that means this function
    // should be called to parse the remainder (arguments, etc.)
    parse_error_t (*parser)(const char * cmd_remainder);

    // token-level help, first tab completion in command
    const char * one_line_help_string;
    // last tab shows this help, and now another tab w/o other input?
    // allow deeper help to be shown? (default is nullptr, which will
    // just show the above one-liner again)
    void (*detailed_help)(const char * cmd_remainder);

} command_t
LONG tab completion walkthrough

Matching for tab completion is based on space-separated tokens.
Each entry of the table lists one token that might be matched, and there’s a pointer to another table in the interesting cases. There’s also a function to determine if the function is usable, allowing to filter commands that are not actually usable … such as mode-specific commands when the BP is not currently in that mode.

“”

At the start, the command line is the empty string, and the user pressed <tab>. The common parser function is passed at least two things:

  1. the root table
  2. the string “” (the command line)

As a special case, when the remainder of the command line is empty, all tokens in the table are considered a potential match.

Thus, the common parser just iterates through the table, updates the terminal by showing all the tokens in that first-level table, and exits. This might look like:

HiZ> 
io     config    macro
cool   storage   mode      
HiZ> █

Note that the i2c command is not shown as an option, because the bus pirate is currently in HiZ mode.

“c”

The user adds one character, c, and hits tab again. The common parser function is passed:

  1. the root table
  2. the string “c”

First token is thus “c”, and no whitespace follows. It prefix-matches more than one entry. Therefore, it shows the options (again)

Thus, the common parser just iterates through the table, updates the terminal by showing all the tokens in that first-level table, and exits. This might look like:

HiZ> c
cool   config
HiZ> c █

“con”

The user adds two more characters, o and n, and hits tab again. The common parser function is passed:

  1. the root table
  2. the string “con”

First token is thus “con”, and there is no whitespace following it. The first token prefix-matches exactly one entry. Therefore, it completes the string for the user and adds a space character … this modifies the pending command line. This might look like:

HiZ> con
HiZ> config █

“config a”

The user adds one more characters, a, and hits tab again. The common parser function is passed:

  1. the root table
  2. the string “config a”

First token is thus “config”, and there are additional characters following. Thus the remainder is determined to be “a”. The first token fully matches exactly one entry. Therefore, tail-recursion calls itself with:

  1. table["config"]->next_table
  2. “a”

The same function executes, this time with a different table, and pointing deeper into the cmdline. A single prefix-match is found, so it auto-completes, adding "nsi " to the command line for the user. This might look like:

HiZ> config a
HiZ> config ansi █

“config ansi c”

  • common_parser( root, “config ansi c”, …)
  • common_parser( config_table, “ansi c”, …)
  • single prefix match, so completes…
HiZ> config ansi c
HiZ> config ansi colormode █

“config ansi colormode 1”

  • common_parser( root, “config ansi colormode 1”, …)
  • common_parser( config_table, “ansi colormode 1”, …)
  • Token for “colormode” has nullptr for next_table, which means it’s an actual command.
  • Given memory constraints, likely would just print a list of options / single-line help for all remaining arguments?
  • Tab completion for the parameters is a distinct problem … and can be added later to make it even better.

So… there are multiple distinct problems, of which I am only addressing the first in this post:

  1. Command hierarchy tab-completion
  2. Once at a command, various options with various complexity
    • tab-completion showing short help
    • repeated tab-completion showing larger help
    • tab-completion having input-responsive tab completion (e.g., only remaining valid options)
  3. tab completion for buspirate scripts ( [ / 0b1.32 \ ] )

Having the hierarchical command structures (something like my example’s concept) improves discoverability and usability, allows common and consistent help parsing, reduces name collisions, etc.

No, don’t care about specific packages. Care about how it feels / works, and then can try to find if any package can work with reasonable results, or just write one or more parts ourselves. BusPirate input is complex already … essentially having three parts that require vastly different tab-completion handling:

  1. Hierarchical command tree … see my overlong example above.
  2. Command parameters (optional, repeated, etc. … so many options … tab-completion here may be tricky to do more than show generic help string. c.f. Powershell’s tab completion on parameters)
  3. buspirate script … which probably can have some great improvements and even show per-mode unique tab-completion results (at least after there are 2+ tokens)
1 Like

One potential starting point for this:

cliparser library by Henry Kwok, which has a BSD 3-clause license (which seems compatible).

The biggest change will be creating the structures for each command, and the options that each command supports.

Those structures would support tab completion not only for the commands, but each command’s options.

For globalization, the help text would need to be updated to get the localized strings. However, the commands themselves would not be localized, so that’s a smaller change.

Some commands may get more verbose command lines as a result. For example, instead of mode i2c 400, it might be:
mode i2c --speed_kHz 400. Beneficially, such a change would increase stability for scripts over time, as each option being prefixed with an explicit indicator of what it refers to allows the list of supported options to expand / grow without affecting existing scripts / option ordering.

1 Like

I am on board with updating the command line parser. It is a mid term goal on my todo, but first I need to complete the docs update and build the all encompassing binary access mode.

1 Like

Hey all,

I got a BusPirate 6 a few weeks ago as I do a lot of reverse engineering work on ISP networking equipment that we integrate our custom firmware onto at my company. I’ve fallen in love with it. In fact I’ve ordered a few more. Since then I have been lurking on the forum.

In my time with the device. One thing that is a little clunky to me and took a little bit to get used to has been the input handling and some of the general cli complexity. So over the past weekend I set up a dev environment and have been making some tweaks and changes to it and in general just going through and familiarizing myself with the code.

While lurking on the forums I saw a suggestion for microrl (micro read line, its Apache-2.0 license wasn’t sure how y’all felt about that) library to act as the cli input handler. I decided to take a crack at it just for fun as possible throw away code. I also think moving to a call back model would be a better/cleaner approach in the long run. I managed to get the general input and callback code working, along with tab completion, and integrated into the core 0 state machine. I haven’t made it past further down stream prompts as it would require a larger rewrite of the underlying command and prompt structure like @henrygab is saying and in fact cliparser could build the parse trees and then the microrl library can play hand off with ui_process back and forth with the commands.

So rather then continue down a path for what could be throw away code. I was hoping to see where y’all were landing on this topic and also wanted to introduce myself and jump in and say that I would love to help with this project and you all have done an amazing job building one of the coolest tools I have seen in quite a while!

2 Likes

Welcome @crgorect, I’m so glad you find the Bus Pirate useful!

Yes, yes, yes! Momentum is gathering to update the command line interface and arguments parser. Any license that is BSD/MIT compatible is acceptable.

The current setup is zombie code from the original Bus Pirate v3.x. It was only intended to process single character commands, and parse hex/dec/bin values. With a dual core ARM and plenty of RAM/FLASH we can do much better.

I’m got a hefty todo list, so this has been a mid-term priority for me. If other people are interested in contributing that is fantastic and I will assist as much as I can.

I’d really like to have a look at what you’ve done and see how it all fits together.

Issues

A laundry list of issues with the current CLI, that might be corrected in a new CLI:

  • The basic command line needs to handle program like commands and pass input to the syntax processor. These should be kept in the history
  • Some things such as prompts need cli like function (type value, backspace/left/right/insert, hit enter, x to exit), but don’t belong in the history. Currently this is a mix and handled inconsistently. Some times using the prompt blows out the CLI history.
  • The Bus Pirate has pretty extensive scripting support, but it is poorly served by the way we have to jam values into the CLI. Pressing up after a script runs can result in non stop loop.
  • It is not self documenting. There are secret “orphaned” global commands that are not obvious until they are manually added to help. Mode commands however are added to the help automatically.

Style

I’m not glued to any specific style, but I rather not be boxed into a “tightly verbose” command line.

- -double-dash-long-flag

Double dash is currently not supported. It is great for scripting, but I don’t want it to be the only option.

-d

I like obvious short flags. If you use the UI a lot it becomes second nature, and I try to use similar flags for the same feature in all the commands.

m i2c
W 3.3

vs

m --mode i2c
W --volts 3.3

Positional arguments seem to be out of vogue, but I find them super useful to quickly get into a mode for testing. This is a hill I will die on because I test and debug the firmware almost every day and I need this for speed :slight_smile:

Menu vs command arguments

This idea has been simmering for a while. Currently you see a config menu when you enter a mode, which takes the settings and dumps you into the mode.

Instead, the configuration menus could output the command line + arguments, place them in the CLI, and run them. The advantage being that the user can copy and paste the command to reuse in scripts, and the mode change would also be a proper part of the up arrow history.

1 Like

What about positional argument support for the mandatory parameters? (while still enabling verbose versions, which is used for enhanced stability with scripting ?

e.g., you can write your shorthand, while scripts can be written with the long (but stable) description you don’t like for interactive use?

1 Like

Sure, of course :slight_smile:

m i2c
W 3.3

This sequence (with various modes) is something I do dozens or more times a day. It’s a reflex at this point.

1 Like