ProtoBuf for controlling BP5

General
1

[reserving to link to later posts]

1 Like
2

Continuing the discussion from Programmatic control of BP5:

7 thoughts for anyone thinking of taking this on

Current thoughts for anyone that might take this up:

  1. First, I recommend waiting until the major parsing update for commands / options is done. It’s expected that will result in a well-defined structure with explicit listing of all the options for commands, etc. … which is 95% of the work. Converting one of those commands to a protobuf message, testing, and incrementally add additional messages then becomes much easier.

  2. Consider use of nanopb … which uses the ZLIB license and seems to be actively developed. There’s also a list of third-party options for various languages, in case a better option appears.

  3. All fields in the .proto should have explicit presence. Generally, this means every field should be explicitly optional or repeated.

  4. Check out the great examples

  5. CAUTION: nanopb’s options include fixed-length options, but there are caveates! See Nanopb: Basic concepts for additional details.
    CRITICALLY, do NOT use fixed_count on any non-LEAF messages (messages that do not contain other messages). See second Note in the decoder implementation details:

Note: The decoder only keeps track of one fixed_count repeated field at a time. Usually this it not an issue because all elements of a repeated field occur end-to-end. Interleaved array elements of several fixed_count repeated fields would be a valid protobuf message, but would get rejected by nanopb decoder with error "wrong size for fixed count field" .

  1. Read about versioning best practices before working on this. Non-exhaustive examples:

  2. Start small … get the infrastructure in place with some test requests / responses.

Entirely untested off-the-cuff concept

// BusPirate.proto file
syntax = "proto3"
import "nanopb.proto" // custom nanopb options directly inline

package com.buspirate.protobuf;

message BP5_test_request {
    optional bool ignored = 1 [default = 0];
}
message BP5_test_response1 {
    optional uint64 ticks = 1 [default = 0]; // monotonically increasing
}
message BP5_test_response2 {
    optional uint32 major = 1 [deafult = 0]; // "version core"
    optional uint32 minor = 2 [default = 0]; // "version core"
    optional uint32 patch = 3 [default = 0]; // "version core"
    repeated string prerelease = 4; // after version core, optional hyphen followed by a series of dot-separated identifiers [0-9A-Za-z-], this would hold that array of identifiers
    repeated string metadata = 5; // version core and optional prerelease, optional plus sign followed by a series of dot-separated identifiers [0-9A-Za-z-], this would hold that array of identifiers
};

message BP5_request {
    // Only one request at a time
    oneof request {
        // don't use 1-15 for test cases!
        bool ignored = 19000;
        BP5_test_request test1 = 19001;
        BP5_test_request test2 = 19002;
    }
}
3
Protobuf does NOT provide framing...

Protobuf does not, itself, provide a framing for the transfer of the messages. In other words, how to know the start and end of a transmission?

This would be implicit in a UDP packet (so long as the message fits in one UDP packet).

A serial port, however … that’s trickier.

Nanopb documentation notes that solutions for serial ports exist, pointing to HLDC as a protocol to wrap the protobuf messages.

:anguished_face:

Thus, it seems a pre-requisite to using protobuf will be a way to frame messages sent over serial … or similar. There may be solid drop-in solutions for the firmware side. But whatever framing is used … need to consider which languages have readily available framing wrappers.

Maybe just base64 encode the whole thing, and send as a null-terminated string … since there’s essentially universal support?

4

base64 is easy, but it has quite the overhead on the wire. And I think the effective wirespeed is something relevant for some protocols where you have large consecutive chunks of data, like for example when flashing firmware, reading out SD cards or similar things.

I think with protobuf and a framing layer you don’t have fully ready-to-use support in any language. So some coding will always be required.

So I’d say aiming for a better framing to keep the overhead in check is justified.

I suggest to use Consistent Overhead Byte Stuffing (COBS) for framing: Consistent Overhead Byte Stuffing - Wikipedia

This isn’t difficult to implement on top of Protobuf.

1 Like
5
COBS could work for framing

Yes, COBS is simple and relatively low overhead, and should be on the short list for consideration as the framing method when communicating over a serial port.

If the serial connection connects in the middle of communcating, it doesn’t re-synchronize? Does the caller have to send a 255-byte sequence of zeros to ensure synchronization?

6

Let’s assume zero is the delimiter byte. Then zero is only used as delimiter and for nothing else. So if the receiver sees just one zero in the received datastream it can re-synchronize.

If there are zeros in the payload they are encoded according to the COBS logic.

1 Like