Long duration voltage/DIO logging

[Vynce]

I’m using binmode on a Bus Pirate 5 to log the ADC voltages on several pins of a PIC 16F in a battery charge controller to understand how it works. It’s been running for a couple of days so far and it’s working pretty well!

This is essentially all I’m doing:

with BPIOClient(args.port) as client:
    with args.logfile.open("w") as f:
        f.write("Timestamp,IO0,IO1,IO2,IO3,IO4,IO5,IO6,IO7\n")
        while True:
            try:
                adc_values = client.status_request(adc=True)["adc_mv"]
            except Exception:
                continue

            timestamp_ms = time.time_ns() // 1_000_000
            f.write(f"{timestamp_ms},{','.join(str(v) for v in adc_values)}\n")

I get ~50 samples/sec on my MacBook and ~38 samples/sec on a Raspberry Pi 3. I experimented with optimizing the loop (including removing the file writes), but I suspect the Pi is just spending more time in the flatbuffers/cobs code.

This brings a few thoughts/questions to mind:

1. How fast could the Bus Pirate send ADC sample packets? 40-50 samples/sec is working for my use case, but a bit faster may be useful to fill the gap between voltmeter and oscilloscope (?). The fastest signal I’ve seen on this PIC has been a 40 Hz PWM fan speed control.
2. A voltage logging mode might be useful (?). Like send a command that asks the BP to send ADC sample packets as fast as it can (or maybe at a requested rate?). This would avoid needing to send a request for each sample and the overhead that comes with that. Some kind of relative timestamp would need to be included with each sample.
3. I haven’t experimented with the DIO mode yet, but a similar digital input logging mode might be useful (?). Now that I’ve determined which signals are digital, logging the digital states would be more efficient. The BP could potentially only send a packet when one of the digital states changed.

Just some ideas . It’s already been very useful as is!

[ian]

Welcome, and thank you so much for trying out the new Python library!

When we brought up the flatbuffers binmode we found a lot of the lag to be in the USB CDC packet latency. Depending on tons of different factors, I believe we managed about 100 packets/sec (somewhere around there).

That would be consistent with your 50 samples/s (with Python taking more of a toll on the Pi 3) - one request packet, one response packet. Of course, it could also be a bug that has yet to be found.

@Vynce

How fast could the Bus Pirate send ADC sample packets? 40-50 samples/sec is working for my use case, but a bit faster may be useful to fill the gap between voltmeter and oscilloscope (?). The fastest signal I’ve seen on this PIC has been a 40 Hz PWM fan speed control.

A voltage logging mode might be useful (?). Like send a command that asks the BP to send ADC sample packets as fast as it can (or maybe at a requested rate?). This would avoid needing to send a request for each sample and the overhead that comes with that. Some kind of relative timestamp would need to be included with each sample.

Interesting question. If the samples are sent full in 64byte CDC packets, that’s 4 full samples a packet (64/8/2). If we estimate ~100 packets a second, that could potentially be 400 samples a second if the wind is blowing right. Any timestamp would eat into that, unless a lot of effort went into precise timing and sufficient buffer to handle USB lag.

A command to “just spray me with ADC samples until I say stop” would be an easy add though.

@Vynce

I haven’t experimented with the DIO mode yet, but a similar digital input logging mode might be useful (?). Now that I’ve determined which signals are digital, logging the digital states would be more efficient. The BP could potentially only send a packet when one of the digital states changed.

Yeah, totally. DIO isn’t yet integrated into the flatbuffers binmode, but I am also thinking along these lines.

        # Create the query vector BEFORE starting the StatusRequest table
        StatusRequest.StartQueryVector(builder, 1)
        builder.PrependUint8(StatusRequestTypes.StatusRequestTypes.All)
        #builder.PrependUint8(StatusRequestTypes.StatusRequestTypes.Version)
        query_vector = builder.EndVector()

This is probably adding a bunch of overhead too. The Python library grabs all the Status info with each query, but the Bus Pirate actually supports making individual queries as well.

       # Define status type mapping for cleaner code
        STATUS_TYPES = {
            'version': StatusRequestTypes.StatusRequestTypes.Version,
            'mode': StatusRequestTypes.StatusRequestTypes.Mode,
            'pullup': StatusRequestTypes.StatusRequestTypes.Pullup,
            'psu': StatusRequestTypes.StatusRequestTypes.PSU,
            'adc': StatusRequestTypes.StatusRequestTypes.ADC,
            'io': StatusRequestTypes.StatusRequestTypes.IO,
            'disk': StatusRequestTypes.StatusRequestTypes.Disk,
            'led': StatusRequestTypes.StatusRequestTypes.LED,
        }

        # Collect requested status types
        requested_types = []
        if not kwargs:
            # No specific types requested, request all
            requested_types = [StatusRequestTypes.StatusRequestTypes.All]
        else:
            # Collect only the requested types that are True
            for key, status_type in STATUS_TYPES.items():
                if kwargs.get(key, False):
                    requested_types.append(status_type)

        # Create the query vector
        StatusRequest.StartQueryVector(builder, len(requested_types))
        for status_type in requested_types: 
            builder.PrependUint8(status_type)
        query_vector = builder.EndVector()

Totally untested, but something like this could replace the above in bpio_client.py ~ line 302. This would only request the specific subset of status info. ADC will only return ADC info instead of everything.

If that is functional please let me know

[ian]

Another possibility, currently not supported by the Python client, is to queue multiple requests in a single USB packet. There are 64 bytes for the CDC packet, so 2 – 3 requests could be packed in at once.

[Vynce]

Thanks for the reply! My Bus Pirate is very busy logging data right now and I don’t want to disturb it, but I’ll try to experiment with your suggestions this weekend .

[Vynce]

I tried your suggested change. It needed some extra checking in the status_dict where it processes the response package since all fields are now optional.

With that, I get ~75 samples/sec on a Raspberry Pi 3 (~2X faster)!

I also tried making a dedicated adc_request() method and that manages ~81 samples/sec on a RPi 3.

    def adc_request(self, **kwargs):
        builder = flatbuffers.Builder(1024)

        StatusRequest.StartQueryVector(builder, 1)
        builder.PrependUint8(StatusRequestTypes.StatusRequestTypes.ADC)
        query_vector = builder.EndVector()

        StatusRequest.Start(builder)
        StatusRequest.AddQuery(builder, query_vector)
        status_request = StatusRequest.End(builder)
        resp_packet = self.send_request(builder, RequestPacketContents.RequestPacketContents.StatusRequest, status_request)

        if not resp_packet:
            return None

        status_resp = StatusResponse.StatusResponse()
        status_resp.Init(resp_packet.Contents().Bytes, resp_packet.Contents().Pos)
        return [status_resp.AdcMv(i) for i in range(status_resp.AdcMvLength())]

[ian]

Wow, so a decent improvement in both cases.