BPIO2 binary mode

[ian]

Change binmode to BPIO2. Currently sump is active. Use the binmode command in the terminal.

[bz2]

Thank you! That did the trick!

[ian]

Let me know how it goes. You’re the first to try the Python module

[bz2]

So far, so good using I2C!

[ian]

There is a “core dump” doc here, but I’m thinking about how I want to structure all this.

git repo

Separate git repo with

• .fbs tables master file
• tooling for every language available
• modules/libraries for working with the tooling (python example)

Probably a top level readme with an overview of the flatbuffer tables.

For each module/library I imagine there should be a readme.md with some basic instructions.

docs

I’m struggling with the order and what to put where. I prefer all the docs to be on the docs site, but know that it is always nice to have something in the git repo to get started quickly.

• Non technical description of the options in the flatbuffer tables
  –Version
  – Mode
  – psu
  – pullup
  – io
  – disk
  – led
  – max (packet size)

I’m not sure how to handle this section. What I want to convey is “the BPIO2 interface offers these status and configuration options”, without getting into the weeds about flatbuffers.

• Python interface
  – Install
  – example.py simple get status
  – psu config
  – pull-ups
  – io
  – led
  –print string
  – hardware
  – Changing modes, mode configuration
  – Read/write data

These would be actual code examples using the python module.

• Generating flatbuffer tooling
  – Link to repo
  – How to generate

This is pretty much complete in the draft docs.

• Explanation of the actual flat buffer tables, request/response

This would be the technical bit.

Alternately I could see starting with the python module as a explanation of the options, something like this:

• Introduction with TOC
• Python example
• List of available table options
• Flatbuffer tooling generation
• Overview of actual fbs file

Will try to draft something and see how I like it.

[bz2]

@ian

There is a “core dump

Did you intend to post a localhost link?

[ian]

Thanks! Sorry, I copied from my local dev server. Link updated and also here is the right link:

BPIO2 Flat Buffer Interface

I spent a few hours preparing to write the docs. I hope to push the Python module bit tomorrow.

If anyone is familiar with the languages covered by flatcc, I’d welcome help adding the base flat buffer code to the new repo. I will push a work in progress tomorrow.

[ian]

A rough first draft of docs for the Python example library and initial check in for the BPIO2 flatbuffer repo.

Here’s the current outline. I feel like the tables are a bit bulky and inconsistent, but the python examples are pretty comprehensive. Any suggestions on how to display functions, variables, etc? I haven’t documented a lot of high level software before.

[ian]

Wrote a python script to automate the generation of all the flat buffer tooling for every language. All tooling is now in the repo along with the language-specific include files. It also generates a readme file automatically.

Broke apart the Python examples into individual scripts with help, options, etc.

Reorganized the Python folder structure to be a bit tidier.

Wrote a Python script that extracts the help from each Python example script and automatically generate a readme file.

[ian]

Updated the flat buffer tables to a final v2.0:

• firmware_ and hardware_ fields in the StatusResponse table changed to version_firmware and version_hardware to be consistent with the rest of the fields
• Added mode_max_packet_size:uint32;, mode_max_write:uint32; mode_max_read:uint32; so the Bus Pirate can report the maximum size COBS encoded packet, data read/write. This is where the 6+ is going to start pulling away from the 5 - it has 256K extra ram to play with.
• Removed pullx fields, the implementation needs more thought
• Added hardware_selftest field to configuration request.
• Request and response packet version incremented to 2.0. Lower version will be rejected by the Bus Pirate.

Changed how the COBS buffer was implemented and got a read speed increase of ~30KB/s!

New flat buffer tooling and .fbs file pushed to the new BPIO repo.

An updated firmware should arrive shortly.

I’ll continue with the docs.

[ian]

Everything from here down is complete. FlatBuffers schema is documented with python examples.

There is a new “spaghetti code” minimal implementation of BPIO2 in python, which is also the code used to illustrate the flatbuffers schema in the docs.

Several things changed today though, so now I need to go back and update the Python library docs again. And a heavy proof read.

Almost there!

ETA: have an hour while my Sarsae chills, so I finished the updates to the Python library docs as well. Still in dire need of a complete proof read, will do that first thing tomorrow.

TODO:

• Proof read
• Update python library & examples docs (mostly done)
• Document the flatbuffer schema
• Document how to directly use the Python flatbuffer tooling
• Final updates to flatbuffers: increment version, remove pull-x, add max packet size, max data size to status info.
• Split examples.py into individual scripts with help and options
• Generate tooling for all available languages, upload to repo
• Upload all language flat buffer support files to repo
• Some basic readme.md files in the repo

[robjwells]

Thanks for this Ian.

I believe we should remove the default version numbers from the flatbuffer schema for RequestPacket. If it’s not explicitly set, it’s the default value, and the default value is whatever the receiver’s schema says it should be.

So a v1 client can send a packet without a version number, and a v2 receiver that checks the version will see “2”. Eg:

❯ diff -u v1/schema_v1.fbs v2/schema_v2.fbs
--- v1/schema_v1.fbs
+++ v2/schema_v2.fbs
@@ -1,3 +1,3 @@
 table T {
-    version: uint8 = 1;
+    version: uint8 = 2;
 }

❯ v1/target/debug/v1
Wrote flatbuffer_from_v1.bin without version number.
❯ v2/target/debug/v2
T { version: 2 }

[ian]

Excellent point! I did not think about that, you’re totally right.

I assume it should it be included in packets going each direction?

I can see a scenario where the bus pirate doesn’t need to send the version because any request with the wrong version will be rejected.

On the other hand the minor version indicates addition or missing features, so good for the host to know the exact capabilities. Perhaps this could be tucked into the status request. My brain is a bit muddy from the big push today, I’ll think it over a bit more.

[robjwells]

Probably for correctness the ResponsePacket shouldn’t have defaulted version numbers either, but it’s only the Bus Pirate firmware that needs to get this right since it’s the only place a ResponsePacket is constructed.

[ian]

I’ve updated the docs, demos, etc to remove the default value and to explicitly set it from the scripts.

union RequestPacketContents { StatusRequest, ConfigurationRequest, DataRequest}

table RequestPacket {
  version_major:uint8;
  version_minor:uint8;
  contents:RequestPacketContents;
}

union ResponsePacketContents { ErrorResponse, StatusResponse, ConfigurationResponse, DataResponse}

table ResponsePacket{
  version_major:uint8;
  version_minor:uint8;
  contents:ResponsePacketContents;
}

Current blocking item is the major/minor version in the Response packet. Can it be eliminated, or is it useful?

What is the purpose of the major version?

• If the Request major version is incorrect, will the bus pirate be able to validate the packet to check the major version? Probably not, because the changes are breaking by nature of incrementing the major version.
• Assuming the Bus Pirate responds with an error if the Request packet major version doesn’t match. If the major versions are not compatible, can the host decode the Response and error message? Probably not, because a major version increment is by nature breaking.

What is the purpose of the minor version?

• In a Request packet the minor version tells the Bus pirate what the host can support, but the nature of flatbuffers is backwards compatibility and ignoring things that are unsupported. So I’m not sure how the Bus Pirate would behave differently based on minor version.
• In a Response packet it actually is kind of useful. It tells the host that some new fields might be missing, to alert the user to upgrade or reduce functionality. However is this needed in every single response packet? I’d argue no, it could be part of the StatusResponse.

What I suggest is to remove the major and minor versions from Request and Response tables, as I’m not sure they are actually useful and it reduces packet overhead.

Then add a version_flatbuffers field to the StatusResponse table.

I’m far from an expert though. I’m going to leave this here for feedback and not make any changes for now.

Will continue with proof reading the docs.

[robjwells]

Perhaps we could:

• Retain the major version in RequestPacket.
• “Freeze” the presence of ErrorResponse in ResponsePacket.

Ideally a mismatched client would make a request and always be able to decode an ErrorResponse that makes clear the client version is unsupported by the firmware.

The motivating scenario in my mind is:

• User buys a Bus Pirate with current firmware (or updates to current firmware).
• User downloads a BPIO2 CLI that is out-of-date.
• User gets a clear message that the CLI and firmware versions are incompatible, rather than some weird difficult-to-debug behaviour.

That said, I need to check what stability guarantees flatbuffers make, because this all goes out the window if the client can’t reliably decode an ErrorResponse received from newer firmware.

[ian]

@robjwells

That said, I need to check what stability guarantees flatbuffers make, because this all goes out the window if the client can’t reliably decode an ErrorResponse received from newer firmware.

That would be my ideal scenario too, but I am also unsure if that is feasible.

Finished the docs and pushed. Will do a final update when this bit is worked out.

[robjwells]

My understanding from reading the “evolution” docs is that we should be able to guarantee stability of the ErrorResponse by:

• Keeping the ResponsePacketContents position in the ResponsePacket table the same.
• Keeping the ErrorResponse position in the ResponsePacketContents enum the same.
• Keeping the error string at the same position in the ErrorResponse table.

Everything in flatbuffers is positional, it turns out. The danger comes when entirely removing (rather than deprecating) table fields, and re-ordering enum variants. Enum variants can be given an explicit discriminant (effectively forcing their “position”), but table fields cannot.

[ian]

@robjwells

• Retain the major version in RequestPacket.
• “Freeze” the presence of ErrorResponse in ResponsePacket.
• Keeping the ResponsePacketContents position in the ResponsePacket table the same.
• Keeping the ErrorResponse position in the ResponsePacketContents enum the same.
• Keeping the error string at the same position in the ErrorResponse table.

I agree, this should lock in everything properly if we evolve the schema in the future. New tables are no issue, otherwise everything needs to be added to the end. ErrorResponse is the first member of the ResponsePacketContents to assert it’s prominence.

@robjwells

• User buys a Bus Pirate with current firmware (or updates to current firmware).
• User downloads a BPIO2 CLI that is out-of-date.
• User gets a clear message that the CLI and firmware versions are incompatible, rather than some weird difficult-to-debug behaviour.

I need to game this out to get a better picture of the different scenarios, feel free to skip to the bottom…

Scenario: host application uses older flatbuffers schema than firmware schema, but they have been evolved correctly.

In this case the CLI should work if the schema was evolved correctly, it is unaware and not in need of the new fields/tables.

Scenario: host application uses newer flatbuffers schema than firmware, but they have been properly evolved.

In this case the

• host application could query the StatusResponse and get the current version_flatbuffer.
• If the version is < {feature requirement} the application tells the user to update or has reduced functionality.
• The Bus Pirate would not report an error because it is unaware of the new fields or tables.

We could force it to report an error by sending a evolution version field in the RequestPacket.

• If the evolution version > firmware, report an error.
• However, this will break applications that use a newer schema but would still be perfectly compatible with an older schema (e.g. doesn’t use new features).
• Instead the evolution version field could be a minimum version requirement,which the Bus Pirate detects and responds with an error

Scenario: Complete breaking rework of schema to BPIO3. BPIO3 application and BPIO2 firmware.

If we play it exactly right and keep the ResponsePacket and ErrorPacket consistent between 2 & 3, and

• the RequestPacket has a major version (2 vs 3)
• the BPIO2 firmware can still detect the major version mis-match
• send an BPIO2 error packet the BPIO3 host can decode.

Scenario: Complete breaking rework of schema to BPIO3. BPIO2 application and BPIO3 firmware.

Assuming the same limited compatibility between 2 & 3:

• Request packet has major version (3 vs 2)
• The BPIO3 firmware can still detect the major version mis-match
• Send a BPIO3 error packet the BPIO2 host can decode.

So then…

Some truths (?):

• It’s probably better that the Bus Pirate always report incompatibilities in “evolution versions”. Lazy software DEVs (me!) might not implement a flatbuffer_version query/check, but probably will (ha!, never!) at least report error strings.
• We may want to return more detailed error strings with instructions/a link to docs on compatibility issues we aren’t yet aware of.
• If at all possible, the Bus Pirate should be able to report major version mismatches because the host will be unable to detect them through normal queries.
• Mismatched “evolution versions” should not cause errors unless the missing functionality is actually needed.

1. RequestPacket.major_version is needed to detect mismatch BBIO2 vs BBIO3 in 2 scenarios
2. RequestPacket.minimum_evolution_version is needed for the Bus Pirate to detect if the required features are missing
3. StatusResponse.version_flatbuffers_evolution should be added so that applications can actually query and discover the current evolution version and adapt accordingly.

union RequestPacketContents { StatusRequest, ConfigurationRequest, DataRequest}

table RequestPacket {
  version_major:uint8; //BPIO2, BPIO3, etc
  version_minimum_flatbuffers_evolution:uint16; //application wants this version or above
  contents:RequestPacketContents;
}

union ResponsePacketContents { ErrorResponse, StatusResponse, ConfigurationResponse, DataResponse}

// remove major and minor version from response packet.
table ResponsePacket{
  contents:ResponsePacketContents;
}

• Request now specifies the major version and the minimum required evolution version
• Remove major/minor from response, this will be handled by error instead. Cons: host has no way to detect the actual major/evolution version, it will only get the error string.
• I feel inclined to increase the evolution version to in16 because we could blow through 256 evolutions by adding in things like logic analyzer, disk access, etc.

Of course we could keep major and evolution version in the Response packet so the host can read them out and do something useful. This could be enhanced a bit by only returning major/evolution version in Response packets containing Status or Error tables to improve speeds on Data and Configuration requests.

union ResponsePacketContents { StatusResponse, ConfigurationResponse, DataResponse}

table ResponsePacket{
 error:string;
  contents:ResponsePacketContents;
}

Since fields not populated are not sent, what impact would it have to move the error string into the Response packet?

table StatusResponse {
  error:string; // Error message if any.
  version_flatbuffers_evolution:uint16;

Adding an evolution version to the StatusResponse seems necessary.

Thoughts are still convoluted, if you made it this far congratulations

[robjwells]

@ian

Instead the evolution version field could be a minimum version requirement,which the Bus Pirate detects and responds with an error

To make sure I understand this, the intention is:

• Client says “I need features from evolution 123.”
• Bus Pirate returns error if firmware evolution version is <123. Proceeds if ≥ 123.

Correct?

@ian

Remove major/minor from response, this will be handled by error instead. Cons: host has no way to detect the actual major/evolution version, it will only get the error string.

This could be something the host queries with a StatusRequest. Though with the major version, it may be a case of “if you can parse a StatusResponse, our major versions match.”

@ian

I feel inclined to increase the evolution version to in16

I think so. u8 is ultimately a small range.

@ian

what impact would it have to move the error string into the Response packet?

The down side would be that you couldn’t include extra fields that would conceptually be better as part of ErrorResponse and deserve to be non-string types. I don’t know that’s a real concern, an error string is pretty flexible after all.

In practice, I think it’s an improvement. It’s simpler to check, and it mirrors the error field in the other response tables, which I treat as “if there’s an error string, the other response fields are not present or invalid”.