Command line update discussion (tab completion, hierarchical / mode-sensitive, ...)

Development
1

Continuing the discussion from OTP whitelabel options for RP2350 boards:

A few items for discussion:

  • Command reorganization
    • Hierarchical command definition
    • Common parsing of command line parameter types
    • Simpler extension with new sub-commands
    • Tab-completion … in long-term managable form (no per-command work)
Example using (global) OTP commands

Given the desire to add the following permitted commands:

otp dir [entryTypeId]
otp dump 3xbytes (startRow) [rowCount] [--show-votes]
otp dump n_of_m (startRow) (rowCount) N [--show-votes]
otp dump raw (startRow) [rowCount] [-q|--quiet]
otp dump ecc (startRow) [rowCount] [-q|--quiet]
otp dump auto (startRow) [rowCount] [-q|--quiet]

At root level, would add otp as a string to the global (1st-level) table, which points to an OTP-specific 2nd-level table (same structure as the 1st-level table). This OTP entry would indicate the command(s) is globally available (should be parsed regardless of mode).

The 2nd-level OTP table would have two entries: One for dir (otp dir if parsing the whole hierarchy), which points to a command structure with options.
One for dump, which points to a 3rd-level table (again, same structure as 1st/2nd levels).

The 3rd-level otp dump table would have five entries, each pointing to a command structure with options for the corresponding variant.

A command structure would support listing mandatory and optional parameters, provide help for each parameter, examples for the command, etc.

Example for a specific mode

Given the desire to add the following permitted commands:

mode irplank [--use_prior_values]
mode irplank [--show_prior_values]
mode irplank [--tx_hertz N] [--tx_hertz N] ...

This indicates that there’s a mode supporting buspirate scripting.

Adding a mode would NOT modify the global command table, but only modify the mode 2nd-level table, by add an entry for the command irplank (with its unique options to configure that mode).

Note that doing this would remove the overhead of having all the prompting for settings. The help text for each option would define what those options do. The --use_prior_values and --show_prior_values can be common code for all modes, reducing complexity of adding things further.

Common Parsing reduces per-addition overhead

Some default parsing becomes possible for each hierarchy. For the mode, the display / setting / saving of parameters can become automatic (with opt-out).

For all commands, a -h / --help switch can parse the same structures and display the help that was defined … right next to its implementation.

Optional parameter? Parameter with optional / required arguments? Flags to permit arguments to be integer / hex / binary / bpscript token(s) / etc.? 90% declarative, and thus self-documenting.

As another example, for modes, it becomes possible to automatically document which bpscript tokens have meaning. Which modes ignore / implement:

  • protocol_start
  • protocol_start_alt
  • protocol_dath (“set data line high”)
  • protocol_write
  • protocol_read
  • protocol_datl
  • protocol_dath
  • protocol_dats
  • protocol_periodic
  • protocol_... etc.

The information is already in the configuration structure (note: I would have the configuration using nullptr to select a do-nothing option, adjusted to the default do-nothing function when the mode is entered). In other words, this can help to automatically document a substantial portion of the various modes, AND tab completion would become way more powerful by excluding options that the current mode doesn’t actually support.

Tab completion

Basically, processes each character of terminal input (not line-by-line). While at the CLI, it uses the command tables to discover the valid token(s) that might complete the current partial user-input-line. When tab key is pressed, if only one valid option to complete the partial line exists, it completes the entry, and adjusts the “current” table it looks at to that next-level table. If multiple entries could complete them, it displays those options (compactly).

While inputing bpscript, this would be a distinct tab completion process. First, it could be dependent on the mode’s configuration, such as to exclude from the list of allowed tokens those that the current mode does not support.

If this is pulled off, it will add a level of polish and shine to the CLI that reflects a much more mature (and expensive) product … with only a one-time-engineering cost.

I love the project. I see limitations and trouble coming for the CLI.

How can we make the CLI much more powerful and usable, while making it easier to add new commands / subcommands?

Should there be a way to mark a command / tree of commands as tied to a specific type of plank attached? (IR, smart card, etc. … each of which relies on hardware on the plank to do more)

[Edit – Add this para] Perhaps the structure listing the commands is universal, with a flag indicating if the command is available. e.g., OTP function availability would return false if running on RP2040. Why? Less #ifdef mess outside of the implementation (where it is unavoidable).

All the above is described in concept form … actual code comes after defining how it should work to be pleasant and user-friendly.

Oh, and that whole tab-completion thing has zero runtime overhead when not using tab. Winning!

3 Likes
2

Tab completion is amazing! Please make it happen if possible. Are you going to implement it with a state machine or something like that?

I have another proposal—let me know what you think. In many debuggers for exploiting, we do it like this:

Pressing ENTER without any input repeats the previous command!

This way, something that was previously useless now has a purpose. Do you think it’s a good idea, or does it seem weird or unnecessary?

Nah, it’s a bad idea. People copy-paste commands from the web, and if an accidental ENTER sneaks in, they could mess something up… Nah, forget the ENTER thing.

2 Likes
3

Are you interested in a specific cli package?

Edit: the up arrow does flip through the history as well.

4

How it happens is TBD… It depends on interest level, because such a sweeping change would likely require deep changes to how commands are defined … and maybe result in a less cryptic UI?

I tried to give a (too-brief) summary above in the “Tab Completion” details block … did you know that block is expandable / collapsible? (I didn’t when I first saw one…) Here’s my off-the-cuff attempt to describe what happens from starting state, where I will take some liberties by re-organizing how some commands work.

Command re-org for this example

io
    powersupply ( ON | off )
    pin (0..7) ( READ | low | high )
    pullup [--pinmask 0bXXXX'XXXX] ( ON | off )
    ...

cool ...

config
    language
    ansi
        colormode ( 16M | 256 | disabled )
        toolbar ( ENABLED | disabled )
    display
        mode ( DEFAULT | oscilliscope | scope )
        screensaver ( 5MIN | 10min | 15 min | disabled )
    pixels
        effect ...
        color ...
        brightness ...
    button
        ...

(storage | stor)
    ls
    cd
    ( md | mkdir )
    ( del | rm )
    ( rd | rmdir )
    format

macro
    load -f (filename) 
    ...

mode
    i2c [ -speed N ] [ -clockstretching ( OFF | on ) ]
    pixel [ -type ( WS2812 | SK6812 | Neopixel | APA102 | SK9822 | Onboard ) ]

// Commands valid only when in mode==I2C
i2c
    scan [-v | --verbose]
    device
        si7021
        ms5611
        tsl2561
    ...
Tables structure concept...

typedef int32_t parse_error_t;
typedef struct _command_t {

    // Critically, the token to be matched
    const char * token;

    // Points to the start of the next level's table
    // which allows tab-completion with low overhead?
    // if this is NULL, when it's a single command
    const struct _command_t * next_table;

    // Dynamic determination of whether (if it's a command)
    // it is currently usable ... e.g., for mode-specific commands
    bool (*usable)(void);

    // When "enter" is pressed and the token matches,
    // AND there's no table ... that means this function
    // should be called to parse the remainder (arguments, etc.)
    parse_error_t (*parser)(const char * cmd_remainder);

    // token-level help, first tab completion in command
    const char * one_line_help_string;
    // last tab shows this help, and now another tab w/o other input?
    // allow deeper help to be shown? (default is nullptr, which will
    // just show the above one-liner again)
    void (*detailed_help)(const char * cmd_remainder);

} command_t
LONG tab completion walkthrough

Matching for tab completion is based on space-separated tokens.
Each entry of the table lists one token that might be matched, and there’s a pointer to another table in the interesting cases. There’s also a function to determine if the function is usable, allowing to filter commands that are not actually usable … such as mode-specific commands when the BP is not currently in that mode.

“”

At the start, the command line is the empty string, and the user pressed <tab>. The common parser function is passed at least two things:

  1. the root table
  2. the string “” (the command line)

As a special case, when the remainder of the command line is empty, all tokens in the table are considered a potential match.

Thus, the common parser just iterates through the table, updates the terminal by showing all the tokens in that first-level table, and exits. This might look like:

HiZ> 
io     config    macro
cool   storage   mode      
HiZ> █

Note that the i2c command is not shown as an option, because the bus pirate is currently in HiZ mode.

“c”

The user adds one character, c, and hits tab again. The common parser function is passed:

  1. the root table
  2. the string “c”

First token is thus “c”, and no whitespace follows. It prefix-matches more than one entry. Therefore, it shows the options (again)

Thus, the common parser just iterates through the table, updates the terminal by showing all the tokens in that first-level table, and exits. This might look like:

HiZ> c
cool   config
HiZ> c █

“con”

The user adds two more characters, o and n, and hits tab again. The common parser function is passed:

  1. the root table
  2. the string “con”

First token is thus “con”, and there is no whitespace following it. The first token prefix-matches exactly one entry. Therefore, it completes the string for the user and adds a space character … this modifies the pending command line. This might look like:

HiZ> con
HiZ> config █

“config a”

The user adds one more characters, a, and hits tab again. The common parser function is passed:

  1. the root table
  2. the string “config a”

First token is thus “config”, and there are additional characters following. Thus the remainder is determined to be “a”. The first token fully matches exactly one entry. Therefore, tail-recursion calls itself with:

  1. table["config"]->next_table
  2. “a”

The same function executes, this time with a different table, and pointing deeper into the cmdline. A single prefix-match is found, so it auto-completes, adding "nsi " to the command line for the user. This might look like:

HiZ> config a
HiZ> config ansi █

“config ansi c”

  • common_parser( root, “config ansi c”, …)
  • common_parser( config_table, “ansi c”, …)
  • single prefix match, so completes…
HiZ> config ansi c
HiZ> config ansi colormode █

“config ansi colormode 1”

  • common_parser( root, “config ansi colormode 1”, …)
  • common_parser( config_table, “ansi colormode 1”, …)
  • Token for “colormode” has nullptr for next_table, which means it’s an actual command.
  • Given memory constraints, likely would just print a list of options / single-line help for all remaining arguments?
  • Tab completion for the parameters is a distinct problem … and can be added later to make it even better.

So… there are multiple distinct problems, of which I am only addressing the first in this post:

  1. Command hierarchy tab-completion
  2. Once at a command, various options with various complexity
    • tab-completion showing short help
    • repeated tab-completion showing larger help
    • tab-completion having input-responsive tab completion (e.g., only remaining valid options)
  3. tab completion for buspirate scripts ( [ / 0b1.32 \ ] )

Having the hierarchical command structures (something like my example’s concept) improves discoverability and usability, allows common and consistent help parsing, reduces name collisions, etc.

5

No, don’t care about specific packages. Care about how it feels / works, and then can try to find if any package can work with reasonable results, or just write one or more parts ourselves. BusPirate input is complex already … essentially having three parts that require vastly different tab-completion handling:

  1. Hierarchical command tree … see my overlong example above.
  2. Command parameters (optional, repeated, etc. … so many options … tab-completion here may be tricky to do more than show generic help string. c.f. Powershell’s tab completion on parameters)
  3. buspirate script … which probably can have some great improvements and even show per-mode unique tab-completion results (at least after there are 2+ tokens)
1 Like
6

One potential starting point for this:

cliparser library by Henry Kwok, which has a BSD 3-clause license (which seems compatible).

The biggest change will be creating the structures for each command, and the options that each command supports.

Those structures would support tab completion not only for the commands, but each command’s options.

For globalization, the help text would need to be updated to get the localized strings. However, the commands themselves would not be localized, so that’s a smaller change.

Some commands may get more verbose command lines as a result. For example, instead of mode i2c 400, it might be:
mode i2c --speed_kHz 400. Beneficially, such a change would increase stability for scripts over time, as each option being prefixed with an explicit indicator of what it refers to allows the list of supported options to expand / grow without affecting existing scripts / option ordering.

1 Like
7

I am on board with updating the command line parser. It is a mid term goal on my todo, but first I need to complete the docs update and build the all encompassing binary access mode.

1 Like
8

Hey all,

I got a BusPirate 6 a few weeks ago as I do a lot of reverse engineering work on ISP networking equipment that we integrate our custom firmware onto at my company. I’ve fallen in love with it. In fact I’ve ordered a few more. Since then I have been lurking on the forum.

In my time with the device. One thing that is a little clunky to me and took a little bit to get used to has been the input handling and some of the general cli complexity. So over the past weekend I set up a dev environment and have been making some tweaks and changes to it and in general just going through and familiarizing myself with the code.

While lurking on the forums I saw a suggestion for microrl (micro read line, its Apache-2.0 license wasn’t sure how y’all felt about that) library to act as the cli input handler. I decided to take a crack at it just for fun as possible throw away code. I also think moving to a call back model would be a better/cleaner approach in the long run. I managed to get the general input and callback code working, along with tab completion, and integrated into the core 0 state machine. I haven’t made it past further down stream prompts as it would require a larger rewrite of the underlying command and prompt structure like @henrygab is saying and in fact cliparser could build the parse trees and then the microrl library can play hand off with ui_process back and forth with the commands.

So rather then continue down a path for what could be throw away code. I was hoping to see where y’all were landing on this topic and also wanted to introduce myself and jump in and say that I would love to help with this project and you all have done an amazing job building one of the coolest tools I have seen in quite a while!

2 Likes
9

Welcome @crgorect, I’m so glad you find the Bus Pirate useful!

Yes, yes, yes! Momentum is gathering to update the command line interface and arguments parser. Any license that is BSD/MIT compatible is acceptable.

The current setup is zombie code from the original Bus Pirate v3.x. It was only intended to process single character commands, and parse hex/dec/bin values. With a dual core ARM and plenty of RAM/FLASH we can do much better.

I’m got a hefty todo list, so this has been a mid-term priority for me. If other people are interested in contributing that is fantastic and I will assist as much as I can.

I’d really like to have a look at what you’ve done and see how it all fits together.

Issues

A laundry list of issues with the current CLI, that might be corrected in a new CLI:

  • The basic command line needs to handle program like commands and pass input to the syntax processor. These should be kept in the history
  • Some things such as prompts need cli like function (type value, backspace/left/right/insert, hit enter, x to exit), but don’t belong in the history. Currently this is a mix and handled inconsistently. Some times using the prompt blows out the CLI history.
  • The Bus Pirate has pretty extensive scripting support, but it is poorly served by the way we have to jam values into the CLI. Pressing up after a script runs can result in non stop loop.
  • It is not self documenting. There are secret “orphaned” global commands that are not obvious until they are manually added to help. Mode commands however are added to the help automatically.

Style

I’m not glued to any specific style, but I rather not be boxed into a “tightly verbose” command line.

- -double-dash-long-flag

Double dash is currently not supported. It is great for scripting, but I don’t want it to be the only option.

-d

I like obvious short flags. If you use the UI a lot it becomes second nature, and I try to use similar flags for the same feature in all the commands.

m i2c
W 3.3

vs

m --mode i2c
W --volts 3.3

Positional arguments seem to be out of vogue, but I find them super useful to quickly get into a mode for testing. This is a hill I will die on because I test and debug the firmware almost every day and I need this for speed :slight_smile:

Menu vs command arguments

This idea has been simmering for a while. Currently you see a config menu when you enter a mode, which takes the settings and dumps you into the mode.

Instead, the configuration menus could output the command line + arguments, place them in the CLI, and run them. The advantage being that the user can copy and paste the command to reuse in scripts, and the mode change would also be a proper part of the up arrow history.

1 Like
10

What about positional argument support for the mandatory parameters? (while still enabling verbose versions, which is used for enhanced stability with scripting ?

e.g., you can write your shorthand, while scripts can be written with the long (but stable) description you don’t like for interactive use?

1 Like
11

Sure, of course :slight_smile:

m i2c
W 3.3

This sequence (with various modes) is something I do dozens or more times a day. It’s a reflex at this point.

1 Like
12

Ok I hope you don’t mind, but I kind of turned this into an architecture document that hopefully can be more easily picked apart. I am looking for feedback on this and I can edit and update so that I make sure I am on the same page with you all. I purposefully left out any libraries as I just want to make sure that we have the end goal of how this works in mind. Also as far as code I was playing around with I will get back into my github account fork the firmware and throw the code up on a branch for you to take a look at. It’s very rough though.

Bus Pirate CLI: Proposed Architecture (Revised)

1. Introduction

This document proposes a modernized command line interface (CLI) architecture for the Bus Pirate. It consolidates feedback from the Bus Pirate community, including:

  1. A desire for hierarchical commands (with “modes” and subcommands).
  2. Improved tab-completion and help systems.
  3. Clean scripting support (without multi-step wizards).
  4. An optional interactive prompt for config if no arguments are given.
  5. Short, positional, and compact commands for everyday “power user” usage.
  6. Long-form or explicit flags for clarity and scripting.

The goal is a flexible yet intuitive CLI that can be used daily for quick debugging, as well as in scripts or advanced testing.

2. Requirements

  1. Usability
  • Provide a fast experience for advanced users (short commands) while allowing a more verbose style for discoverability (long commands).
  • Retain the familiar “modes” and subcommands from previous Bus Pirate usage, but avoid confusion.
  1. Discoverability & Documentation
  • Offer robust help or ? at each level so users can see available commands and parameters.
  • Support both positional arguments (e.g., m i2c, W 3.3) and flag-based arguments (e.g., mode i2c --speed=400kHz).
  1. Scripting
  • Allow direct single-line or multi-line scripts without forcing multi-step wizards.
  • Preserve advanced features, e.g., bridging, in a script-friendly manner.
  1. Mode Handling
  • When in a specific mode (e.g., UART), show relevant commands and subcommands.
  • Consider whether “global” commands remain accessible in a sub-mode or are disallowed (two approaches).
  1. Bridging
  • Provide a pass-through “bridge” with a clear exit condition (hardware button or unique key).
  • Ensure the user can pass control signals like Ctrl-C to the downstream device when needed.
  1. Short vs. Long Commands
  • Short: e.g., m i2c, W 3.3, a 0, d:4.
  • Long: e.g., mode i2c --speed=400000, write-supply --voltage=3.3, etc.
  • Both need to be recognized. Power users can rely on short forms, while new/scripting users can use verbose flags.

3. Layered Architecture

3.1 User Interaction Layer

  • Captures keyboard input, handles history, line editing, and tab-completion.
  • On <Enter>, the entire line is passed to the parser.

3.2 Command Parsing Layer

  • Tokenizes the input. Supports short positional forms (e.g., m i2c 400kHz) and long flag forms (e.g., mode i2c --speed=400kHz).
  • Matches tokens against a hierarchical command registry with aliases and short command names.
  • Considers the current mode to decide which commands or subcommands are valid.
  • If recognized, dispatches a handler function with parsed arguments.
  • If unrecognized, prints usage/help or goes to an optional interactive fallback.

3.3 Command Execution Layer

  • Implements the logic for each command or subcommand (e.g., bridging, scanning I2C, enabling power rails).
  • Manages transitions to specialized states (like bridging) that override normal CLI input.
  • Handles errors or missing arguments (prompting if desired).

4. “Modes” and Sub-CLIs

4.1 Mode Switching

  • Global command m <target> or mode <target>:
    • Short usage example:
HiZ> m i2c
  • Verbose usage example:
HiZ> mode i2c --speed=400kHz
  • Either approach sets hardware config (bus speed, etc.) and switches the user into that mode’s prompt if interactive usage is desired (e.g., I2C>).

4.2 Mode-Specific Commands

  • Each mode has its own subcommands (e.g., bridge, gps, scan).
  • Short examples:
    • bridge => starts bridging in UART mode
    • scan => scans I2C addresses
  • Long examples:
    • bridge --escape=ctrl-]
    • scan --verbose

4.3 Short vs. Long/Flag Styles

  • Short, Positional:
    • m i2c 400kHz sets mode to i2c at 400kHz.
    • W 3.3 sets power supply to 3.3V.
    • a 0 sets a pin to 0 (low).
  • Long, Flag-Based:
    • mode i2c --speed=400kHz
    • write-supply --voltage=3.3
    • pin --index=a --value=low
  • The parser can interpret either style by referencing the command table definitions, each param possibly having aliases (e.g., -b for --baud).

5. Handling Global Commands Inside a Mode

Two major options:

5.1 Approach A: Strict Sub-CLI (No Global Commands in Mode)

  • Once the user does m i2c and the prompt changes to I2C>, only I2C-specific commands are available.
  • The user must type exit (or Ctrl-C) to return to HiZ> for global commands.
  • Pros: Clear separation, simpler local help.
  • Cons: If you want to do ls or macro while in I2C>, you must exit first.

5.2 Approach B: Unified CLI (Global Always Accessible)

  • While at I2C>, you can still do global commands like ls, script, or even m spi.
  • Pros: Very flexible; fewer steps to jump between tasks.
  • Cons: Potential confusion or accidental mode switching.

5.3 Recommended “Hybrid” Variation

  • Let the user see mode-specific commands plus a whitelist of essential global commands (e.g., help, exit, script).
  • This prevents random or accidental mode changes while retaining critical operations in sub-mode.

6. Scripting and One-Shot Usage

6.1 Single-Line Mode Switch

  • Short:
HiZ> m uart 115200 8n1

(Sets UART at 115200 baud, 8-bit, no parity, 1 stop.)

  • Long:
HiZ> mode uart --speed=115200 --parity=none --stop=1 --bits=8

Either way, no interactive wizard. Perfect for scripting or advanced users.

6.2 Multi-Line Scripts

m uart 115200
W 3.3
bridge

Or:

mode uart --speed=115200
write-supply --voltage=3.3
bridge

After bridging, user exits via hardware button or special key. The script or CLI then continues.

6.3 Interactive vs. One-Shot

  • If typed interactively, the user might drop into UART> after m uart 115200.
  • If run in a script, it might remain in HiZ> after the command finishes, depending on how we define the default.
  • Consistency can be configured: e.g., always switch the prompt in interactive sessions, but not in scripts.

7. Bridging and SIGINT

7.1 Bridging Sub-State

  • bridge command creates a pass-through loop between USB and the bus lines (UART, I2C, etc.).
  • Short usage: bridge
  • Long usage: bridge --escape=ctrl-]

7.2 Exiting the Bridge

  • Hardware Button: Always an option to break bridging.
  • Alternate Key: If Ctrl-C must be passed downstream, choose Ctrl-] or another sequence for the Bus Pirate to break bridging.

7.3 Return to Sub-CLI

  • Once bridging ends, the user returns to UART> (or I2C>, etc.).
  • They can type more commands or exit to go back to HiZ>.

8. Signals & Error Handling

  1. SIGINT / Ctrl-C
  • In normal CLI mode, cancels prompts or scripts.
  • In bridging mode, is forwarded downstream unless you define a different policy.
  1. Error Cases
  • Unrecognized command => short usage or help.
  • Missing parameters => error + usage, or optional interactive prompt if desired.
  1. Recovery
  • A forced “hard exit” can reset the CLI to HiZ> if needed.

9. Short & Long Command Definitions

Below is a simplified example of how both short and long forms can be defined in code:

typedef struct cmd_def {
    const char *name;         // e.g. "mode"
    const char *alias;        // e.g. "m" for short form
    bool (*is_available)(void);
    cmd_func_t handler;       
    cmd_param_def_t *params;  // optional param list
    cmd_def_t *subcommands;   // for hierarchical approach
    const char *help_text;
} cmd_def_t;

// Example: "mode" with a short alias "m"
static cmd_param_def_t mode_params[] = {
    // For short usage, we might define a positional param (like "i2c")
    { .position = 1, .type=PARAM_STRING, .required=false },
    // Possibly a second positional param for speed, etc.
    // Or we also define a named param for the long form: "--speed"
    { 0 }
};

static cmd_def_t global_commands[] = {
    {
        .name = "mode", 
        .alias = "m", 
        .handler = mode_handler,
        .params = mode_params,
        .help_text = "Change to a protocol mode, short usage: m i2c"
    },
    // ...
    { 0 }
};

This design accommodates short aliases (like m) and also named arguments (--speed) in the same command definition.

10. Conclusion & Next Steps

This architecture supports:

  1. Hierarchical “modes” with sub-CLI prompts (e.g., UART>).
  2. Short or positional commands for rapid daily usage.
  3. Long or flag-based commands for clarity and scripting.
  4. Bridging with a hardware or alternate key exit.
  5. A choice between strict sub-CLI or unified approach for global commands in a mode.
  6. An optional interactive wizard if the user types m with no arguments, but never forced.

Discussion Points

  1. Strict vs. Unified: Decide how many “global” commands remain accessible in a mode.
  2. Interactive Defaults: Decide if m i2c should always drop you into an I2C> prompt, or if it’s one-shot for scripting.
  3. Short Param Parsing: Clarify how many positional arguments are supported, e.g., m uart 115200 8n1.
  4. Exit Mechanisms: Decide if exit is mandatory or if switching modes directly is allowed.
  5. Key Sequence for Bridging Exit: Standardize on hardware button only, or add a software “escape key.” My preference is both as the button is convenient, but in my case when I travel I have a serial server set up and won’t have physical access to the device until I am back. so having an option to ctrl-c or ctrl-[ or something else would be really convenient

Just some open questions on the design.

2 Likes
13

Wow, that’s a great writeup.

Discourse is not great for such long text, especially with edits expected. Even a git depot just for some markdown would work better, and allow comments (e.g., make a pull request and everyone can make comments with context).

Even so, here’s some initial thoughts:

1. Introduction

Not sure how to convey this, or if it’s understood just from the term.

The purpose of the hierarchical command concept is to reduce contention, and make it easier to add related commands in one area without affecting everything else. e.g., once the first-level command of “pixels” exists, sub-commands only have to be unique relative to other sub-commands under “pixels”. and any one of those might itself result in a tree of options. (Even global commands are just root-level tokens with no subcommands…)

A reasonably good example of a command structure (although not targeting an IOT-class device) is the Proxmark3 client. That project has stood the test of time … with over 20k commits now, and the client CLI remains usable and extensible. It uses cliparser (caution: its own source is GPL, so do not copy/paste), and thus we can benefit from their hard-earned (painful?) learnings.

2.2 Discoverability & Documentation

  • -h / --help as a command option should always show the help for that branch of the command, even if other options are on the line of input.
  • flag-based arguments may occur in any order
  • NON-GOAL: positional argument stability … instead, they are allowed to change over time (scripts should use flag-based long form). This is a necessary evil to allow commands to change over time.

2.4 Mode Handling

I believe global commands must remain accessible.

2.5 Bridging

Is this too specific to a single mode? It’s not clear to me what this is intending to describe?

3.1 UI Layer

Yes to all. Add:

  • On , the entire line is passed to a tab-completion handler.
    • The tab completion handler substantially overlaps with the command parsing layer …
    • If the tab can add at least one character to the line, it will do so (including adding a space after a fully matched token).
    • Else, if there are multiple matching tokens, the list of potential matches is returned from the tab-completion handler, so that the UI can do the necessary thing (depending on ANSI / TTY / etc.)
    • Tab completion will NOT consider a flag-based option that already exists on the command line a valid completion.
    • Tab completion consider the current mode for listing options.

3.3 Command Execution Layer

Um … this may be too wordy. The handler function does whatever it wants. When it’s done, control reverts to the UI layer?

5 Global commands and Modes

Global commands today remain available. I don’t expect that
to change. I do expect those global commands to need to remain available.

General

Commands should not be “verb-object” (e.g., write supply). Instead, the commands should be progressively more specific. Thus, instead the command might be power with two initial subcommands:

  • “set” with option flag “–voltage” that accepts the floating-point value
  • “get” which returns the current setting

6.3 Interactive vs. One-Shot

A script should act identically to user typing.
If a script wants to return to HiZ mode, it should end with m HiZ.
To do otherwise removes power from the scripts, such as preventing chaining of small scripts into more powerful ones.

Regardless, this also seems out-of-scope for this document. Tangentially related, but not core to the CLI.

7. Bridging etc.

This seems to be out-of-scope. Each command handler decides how to handle input while it’s executing. When it’s done executing, either new commands change what it does, or not. Similarly, the binding of a button to a specific function (exiting a mode, rebooting the BP, etc.) is out-of-scope.

8. Signals & error handling

A decision should be made as to whether to exclude Ctrl-C from the potential inputs a command may receive when it’s executing. I’d rather command handlers opt-in to this. Similarly, the command input layer (UI layer) can define its own handling of Ctrl-C.

Error cases – At the UI layer, if no command handler is found, then show help from the deepest matched subcommand. If a command handler was found, the command handler provides any error message (e.g., incompatible combination of flags). Users can then universally simply append --help for default help from that deepst-matching subcommand.

9. Short & Long Command Definitions

This seems out-of-scope? the specific libraries / structures used to reach the goals seems a bit of cart-before-the-horse. Critical when implementing a prototype, but the specific data structures don’t matter yet.

BTW, what you show seems quite close to what cliparser does.

10. Discussion Points

  1. Unless they are pulled out later, presume global commands are global.
  2. Scripts should behave identically to the user typing.
  3. I recommend minimizing the commands that support positional arguments.
  4. Can you help me understand what you mean by “Decide if exit is mandatory or if switching modes directly is allowed”? Are you talking about going directly from (for example) m i2c to m spi? If so, I’ve no strong feeling, but lean towards allowing direct switching (which is current state).
  5. This is outside the scope here, except to say that each handler, while executing, can do whatever it wants with input (including the buttons). We can recommend that long-running commands check for input and handle Ctrl-C to terminate early, but I do not recommend preventing a long-running command from doing otherwise.

Concluding

Good writeup, and I hope it helps you (and others). And I recommend it be shared in a different collaboration medium, given its length. Even my few thoughts are hard to correlate and would have been much shorter as comments to a markdown file on github.

1 Like
14

Excellent write up indeed!

  • Strict vs. Unified: Decide how many “global” commands remain accessible in a mode.

In general global commands should always be available. The exception is global commands what are not allowed in HiZ mode (anything that changes pins or power).

  • Interactive Defaults: Decide if m i2c should always drop you into an I2C> prompt, or if it’s one-shot for scripting.

It seems more consistent with the history of the UI and also for clarity if scripting was handled just like the normal CLI. Especially when the script is used in tutorial mode, where the user gets a walk through of various commands and features.

Also note that some modes (LED, INFRARED) have submodes below the main protocol mode.

  • Short Param Parsing: Clarify how many positional arguments are supported, e.g., m uart 115200 8n1.

SPI and UART seem to have the most configuration options at the moment.

  • Exit Mechanisms: Decide if exit is mandatory or if switching modes directly is allowed.

Switching between modes directly should be allowed.

  • Key Sequence for Bridging Exit: Standardize on hardware button only, or add a software “escape key.” My preference is both as the button is convenient, but in my case when I travel I have a serial server set up and won’t have physical access to the device until I am back. so having an option to ctrl-c or ctrl-[ or something else would be really convenient

For this specific command (UART bridge) button should be an option. An optional ctrl sequence to exit is fine, but it should remain optional so that if bridged data coincidentally contains the quit character sequence it doesn’t drop user out of the bridge unexpectedly.

There’s a lot more I need to digest, I’ll come back with some more thoughts and questions soon.

15

Whew. A lot to absorb there.

On the interactivity/scripting note.
There is a difference between something run when controlled by a process (api) and something run my a human, when conditions exist that could be a mistake.

Example: Warning that the voltage isn’t turned on, etc, If a new user is trying something, there may be a prompt like (are you sure?) Or when there is a missing argument, or tab completion. These things are great for humans and a pain for someone trying to automate things.

If it’s an automated process, you want "helpfull features: turned off. Perhaps warnings can be piped to STDERR. I would expect this would be using an API or else using some terminal-based scripting mechanism.

I’ve used simple echo/printf shell scripts piped to a terminal device, and I’ve also used the expect programming language when interacting with a program that expected terminal inputs and responses to prompts.
Some people want a quick and dirty way to run a BP script without having to write a C program. A Python-based API would definitely be better…

Some shell scripts detect if the input is a terminal or a pipe, and react differently. I’m not sure how the automated/human dichotomy is managed.

16

The goal is to provide a command-line API that avoids the prompting.

Prompting is evil, both from firmware perspective and for scripting / API interactino. However, tab-completion makes it possible to effectively do the prompting via tab-completion and error output.

Stretch goals:

Option-specific help via tab completion

For tab-completion, if the prior token was a flag, and there is no current token, tab completion could provide the option-specific help text, if it’s listed in the structures.

Below, I’ll use the ▄ to indicate the current cursor input position.

No current token example:
m i2c --speed ▄ … notice the trailing space

Example of the above:
HiZ> mode i2c -speed ▄<TAB> → 

HiZ> mode i2c -speed 
Provide the i2c bus speed, in kHz units (integer, `khz` optional)
HiZ> mode i2c -speed ▄

Mid-line tab completion

Support for tab-completion in the middle of a line that is being input. (some libraries already support this … but not listed above, so thought I’d make it explicit)

EOL

1 Like