This E9 bug scares me a lot.
I think, I have seen it as well, when implementing a QSPI with PIO, where it changes to inputs, with no pull-down (or up) enabled.
Nothing seen anymore as received (providing VDD or GND for testing input pin, inputs floating most of the time in between). I had to power cycle the board.
Similar to this report:
https://github.com/micropython/micropython/issues/15718

My questions:
Does it affect ANY GPIO, e.g. also used via peripheral devices, such as SPI (via ALT, not using PIO or GPIO bit-banging)?
Can it happen as well, when SPI as peripheral device is used and MISO becomes floating (slave does not drive, in between MISO input becomes floating)?
Or: a SPI slave device, where SCLK as input is floating (most of the time, when no SPI master selected)?

Would an external pull-up (only a pull-up, not a pull-down) help?
(actually, even I might need a pull-down, e.g. as SPI slave with SCLK as input in SPI mode 0, would a pull-up help?)

These "messages" scare me a lot:
https://github.com/raspberrypi/pico-feedback/issues/401
https://hackaday.com/2024/09/04/the-wor ... situation/
which gives me the impression: does it happen any time, with any GPIO, even for pins used in functions (ALT) like SPI?
(or when programming interfaces like QSPI, MDIO, with a floating input signal, via PIO?)

Any suggestions, e.g. using pull-up enabled (or external one) on any GPIO? (even on SPI peripheral signals?)
Or is it at the end a "show stopper", so that we had to wait for a new spin (B0, C0 version) of this chip?

What to do?

BTW: is just E9 bug related to Bank 0 pins only (all the user GPIOs), not Bank 1 (the USB DP/DM and QSPI signals for external flash memory)?

tjaekel wrote: ↑

Fri Sep 06, 2024 1:45 am

BTW: is just E9 bug related to Bank 0 pins only (all the user GPIOs), not Bank 1 (the USB DP/DM and QSPI signals for external flash memory)?

My understanding is the E9 erratum as written in the documentation is incomplete. I'd expect that to be updated with a definitive description of what's affected within a week. It would be nice if those who know more could also comment here.

Still looking for a real product using I2C/SPI/PIO - this one is interesting DEFCON-32-Badge. But it looks like the source code is not shared on the github so we do not know if the actual code uses real SPI/I2C peripherals or bit-banging.

hippy wrote: ↑

Thu Sep 05, 2024 1:01 pm

bgolab wrote: ↑

Thu Sep 05, 2024 12:25 pm

I looked into DEFCON-32-Badge schematic to see real (I believe) working product and I2C / SPI / UART seems to be unaffected by E9 issue (assumed all available RP2350 chips behave same way...).

Though you would have to check the software to see if that is using actual on-chip peripherals or is bit-banging on the same pins as used by those peripherals in a manner which side-steps or mitigates the E9 issues.

tjaekel wrote: ↑

Fri Sep 06, 2024 1:45 am

BTW: is just E9 bug related to Bank 0 pins only (all the user GPIOs), not Bank 1 (the USB DP/DM and QSPI signals for external flash memory)?

Have you read the datasheet?

"This issue presents under the following conditions, for any GPIO 0 through 47"

"This issue does not affect the QSPI pads, which use a different pad macro without the faulty circuitry.
The USB PHY’s pull-down resistors are also unaffected"

Rockets are loud.
https://astro-pi.org

jdb wrote: ↑

Fri Sep 06, 2024 6:12 am

Have you read the datasheet?

SPI / I2C / PIO seems to be not mentioned here...
Should we assume the E9 issue has no impact on SPI / I2C / PIO peripherals?

jdb wrote: ↑

Fri Sep 06, 2024 6:12 am

tjaekel wrote: ↑

Fri Sep 06, 2024 1:45 am

BTW: is just E9 bug related to Bank 0 pins only (all the user GPIOs), not Bank 1 (the USB DP/DM and QSPI signals for external flash memory)?

Have you read the datasheet?

"This issue presents under the following conditions, for any GPIO 0 through 47"

"This issue does not affect the QSPI pads, which use a different pad macro without the faulty circuitry.
The USB PHY’s pull-down resistors are also unaffected"

This issue presents under the following conditions, for any GPIO 0 through 47:
1.Pull-down resistor is enabled in GPIO0.PDE
...

bgolab wrote: ↑

Fri Sep 06, 2024 6:23 am

jdb wrote: ↑

Fri Sep 06, 2024 6:12 am

Have you read the datasheet?

SPI / I2C / PIO seems to be not mentioned here...
Should we assume the E9 issue has no impact on SPI / I2C / PIO peripherals?

Why would it? The issue is with the pin, not the function driving the pin.

Rockets are loud.
https://astro-pi.org

jdb wrote: ↑

Fri Sep 06, 2024 6:27 am

Why would it? The issue is with the pin, not the function driving the pin.

I did not mean the SPI / I2C function per se. I meant any disturbance caused by E9 on, for example, the I2C bus itself...

tjaekel wrote: ↑

Fri Sep 06, 2024 1:33 am

This E9 bug scares me a lot.

What to do?

Use the RP2040 then.

Always interested in innovative audio startups needing help and investment. Look for InPoSe Ltd or Future Horizons on LinkedIn to find me (same avatar photograph)

tjaekel wrote: ↑

Fri Sep 06, 2024 1:33 am

Does it affect ANY GPIO, e.g. also used via peripheral devices, such as SPI (via ALT, not using PIO or GPIO bit-banging)?

Quite fresh report of some SPI related issues, not investigated, neither proved it has anything to do with E9:
https://github.com/raspberrypi/pico-feedback/issues/409

It has nothing to do with the pin erratum.

Rockets are loud.
https://astro-pi.org

bgolab wrote: ↑

Fri Sep 06, 2024 6:49 am

jdb wrote: ↑

Fri Sep 06, 2024 6:27 am

Why would it? The issue is with the pin, not the function driving the pin.

I did not mean the SPI / I2C function per se. I meant any disturbance caused by E9 on, for example, the I2C bus itself...

If one experiences latch-up, the pin can't be used as a reliable digital input without applying the workround, then I can't see how it could not also affect SPI, I2C, UART, PIO, etc, input.

That, plus latch-up being reported as happening even when the internal pull-down hasn't been enabled, seems to me to be the basis for all the 'worse than initially thought' reports.

I am sorry to put it this way, but have you ever tried to use it ?
I did, and it works.

hippy wrote: ↑

Fri Sep 06, 2024 12:34 pm

If one experiences latch-up, the pin can't be used as a reliable digital input without applying the workround, then I can't see how it could not also affect SPI, I2C, UART, PIO, etc, input.

That, plus latch-up being reported as happening even when the internal pull-down hasn't been enabled, seems to me to be the basis for all the 'worse than initially thought' reports.

And is not a real latch-up. I can see it on my scope.
A real one would need a power cycle to release it, not just a little over 200 uA of drive between 1-2 V, where it happens.
Most digital circuits are capable to drive an order of magnitude higher ... 2-12 mA.
The internal pull-up/down is not intended to drive signals, especially at high speeds.
Only with the internal ones, It takes around 1us just to transition from 0 to 2 V (or the other way around), with the pin floating freely (no external load). Add more inputs and longer traces (capacitive loads) and you will see that even the slowest I2C (the only one in the list which uses pull-up to drive signal) would struggle with it.

Okay, can somebody with a Pico2 try this.

Get the pin in a latched up state.
Set the SIO on that pin to be logic 0 and enable the output, then disable the output say 1uS later.

Has the latched up state gone ?

Obviously one cannot do that if being driven by an active high external gate, but if you are you won't see the problem in any case.

Always interested in innovative audio startups needing help and investment. Look for InPoSe Ltd or Future Horizons on LinkedIn to find me (same avatar photograph)

hippy wrote: ↑

Fri Sep 06, 2024 12:34 pm

bgolab wrote: ↑

Fri Sep 06, 2024 6:49 am

I did not mean the SPI / I2C function per se. I meant any disturbance caused by E9 on, for example, the I2C bus itself...

If one experiences latch-up, the pin can't be used as a reliable digital input without applying the workround, then I can't see how it could not also affect SPI, I2C, UART, PIO, etc, input.

That, plus latch-up being reported as happening even when the internal pull-down hasn't been enabled, seems to me to be the basis for all the 'worse than initially thought' reports.

The use of the term "latch-up" seems to have been the cause of much confusion. Latch-up in ICs traditionally refers to the extremely destructive state where the substrate gets reverse-biased and very large currents flow; the E9 issue is not remotely related to that kind of latch-up. In more general usage of the words, it might be taken to mean a state where the pin gets stuck high or low and can't be got out of that state - but that's not an accurate description of the E9 issue either.

In fact, the behaviour is that under some circumstances when it is supposed to be a high-impedance digital input, the pin sources a small amount of current. It can always be driven to the desired level by a low-impedance signal.

It now appears that this behaviour occurs under slightly more circumstances than described in the official erratum notice, but those circumstances seem now to be well understood in the community - it applies when the input buffer is enabled, not otherwise. Various other possibilities (such as it applying even with the input disabled and so affecting the ADC) that were suggested early on have since been shown not to be correct. The one remaining area that's not been totally characterised is the magnitude of that unintended current and so the necessary source impedance to ensure that the correct level is read.

None of this is to say that the issue as a whole is "serious" or "not serious" - which are in any case subjective terms and depend on the application - but given the information we have now it is possible to fairly easily tell whether a given application is likely to be affected.

This is a pin/pad problem, so clearly applies to any signal connected to the pin regardless of which function is selected inside the RP2350 (or indeed if no function at all is in use but the input buffer is enabled). What matters is the external circuit connected to the pin.

For I2C, it is extremely unlikely that any applications will be affected: I2C necessarily has a pull-up applied, and so the effect of the E9 behaviour is to slightly increase the strength of that pull-up. Applications using an external pull-up resistor typically use values in the region of 1K-4K, so the extra current will be small in relation to that already flowing in the external pull-up; applications relying on the internal pull-ups (which are weaker than desirable for I2C) will actually be improved by the erratum.

For UART, the vast majority of applications drive the input with a digital signal and so data transfer will not be affected. However, a relatively common 'trick' is to have the input pulled down so that it reads zero when unconnected, as a way of detecting whether a cable has been plugged in or not. Such applications may well fail to detect the cable unplugged, even though they work correctly when plugged in.

Similarly, in nearly all cases where SPI is used to interface with standard SPI peripheral chips, those chips will actively drive the data pin and so will not be affected. Some more unusual cases where the SPI interface on RP2040 has been used to interfaces something that isn't quite standard SPI could be affected. I read one report where someone was using a device where the output (input to RP2040) was not driven for the first few bits of the transfer; on RP2040 these bits happened to read as zero and so the software hadn't bothered to mask them off, on RP2350 they sometimes read as 1. Arguably that one is a case of software that wouldn't have reliably worked on RP2040 under all circumstances either, but appears to the user as a backwards-compatibility issue.

PIO applications are many and varied - for many such applications that are purely digital then they are unlikely to be affected, but some more inventive applications could well be affected. For any given application it should be easy to look at the circuit and say unaffected/problem/needs-investigation as applicable.

The datasheet and E9 have been updated : https://datasheets.raspberrypi.com/rp23 ... asheet.pdf

Okay, can somebody with a Pico2 try this.
Get the pin in a latched up state.
Set the SIO on that pin to be logic 0 and enable the output, then disable the output say 1uS later.
Has the latched up state gone ?

Yes, I did it on the first try and it seems to clear the "latch-up" in no time, with a slew rate definitely exceeding my scope.
That was my first though as a workaround in some scenarios. But I need try again by writing directly to the registers (without SDK) and with different timings and with different combinations (lie disabling the Schmidt trigger, sync bypass etc.).

Only at very low slew rate and relatively high impedance, it shows a deformation of the signal in the middle range of voltage, with rising edge accelerating a little bit in that region, and the falling is slowing down.

In some cases, I observed that if driven fast enough at start of the edge (rising 0->1 V, or falling 3->2 V), the latch effect tends to soften, if not disappear at all. Something like if the signal has enough initial momentum, it passes through the latching zone like nothing, but it needs more tests, take it as a guess.

So far, it looks like some kind of static bias in the input buffer, not related to the state of internal pull-up/down, though the internal pull-down helps a little, by lowering the latch voltage. It behaves more like a slightly unbalanced bidirectional soft level-shifter. More test to come, it takes time, but I like it.

The updated E9 is much better and reflects the findings that had been discussed here and GitHub. Sad that everything requiring high impedance won't work (Opinion: Adding extra hardware like buffers is not a real solution. Higher costs, less board space and more energy consumption).

dp11 wrote: ↑

Fri Sep 06, 2024 3:07 pm

The datasheet and E9 have been updated : https://datasheets.raspberrypi.com/rp23 ... asheet.pdf

Oh, that is a much better description, matching what I've seen (not only what I've heard), so far .
I would suggest to add a short note in the GPIO,/PADs sections, for users to be aware from the very beginning, before reaching the end of the document. In my case It was quite disturbing, to discover it hidden there. At that time, my first RP2340 (B) was on the way, arriving next day, in the morning. I am used to work during the night, but man, that night was one of the longest, just to see that is not such a big deal, at least for me. I though you guys, you are not even capable to produce a proper latch-up, which holds even without power, if not a catastrophe, at least a real disaster. No, you are not able.
Just kidding. I really like the new RP234X, as it is.

Anyway,
Thanks for the update !

gmx wrote: ↑

Fri Sep 06, 2024 3:12 pm

Okay, can somebody with a Pico2 try this.
Get the pin in a latched up state.
Set the SIO on that pin to be logic 0 and enable the output, then disable the output say 1uS later.
Has the latched up state gone ?

Yes, I did it on the first try and it seems to clear the "latch-up" in no time, with a slew rate definitely exceeding my scope.

Have you tried shorter and shorter times for the logic 0 ? 1 (overclocked :-) clock cycle even ?

Always interested in innovative audio startups needing help and investment. Look for InPoSe Ltd or Future Horizons on LinkedIn to find me (same avatar photograph)

bgolab wrote: ↑

Fri Sep 06, 2024 7:52 am

tjaekel wrote: ↑

Fri Sep 06, 2024 1:33 am

Does it affect ANY GPIO, e.g. also used via peripheral devices, such as SPI (via ALT, not using PIO or GPIO bit-banging)?

Quite fresh report of some SPI related issues, not investigated, neither proved it has anything to do with E9:
https://github.com/raspberrypi/pico-feedback/issues/409

jdb wrote: ↑

Fri Sep 06, 2024 10:11 am

It has nothing to do with the pin erratum.

(1) Now that the updated datasheet and E9 description is out, could you please elaborate a bit? (I wrote pico-feedback #409 by the way - note the EDIT at the end if you've missed it perhaps.)
(2) If you can conclusively and absolutely dismiss this as not E9 related, should we read into this statement of yours that you're instead aware of some other issue - possibly lingering SDK 2.0.0 gremlins? - affecting SPI, as experienced by me and seemingly others in this thread?
(3) Is there any official ETA yet on when a fixed "A3" silicon might be on the market?

(PS. For the record, my Pico-carrying product interfaces with 2 devices on SPI0, 4 devices on SPI1, and up to 10 devices on I2C, while also utilizing all remaining GPIOs for other purposes, and I've only seen these occasional glitches - then affecting all SPI1 devices by the way; not absolutely sure whether SPI0 was affected too but I believe so - since migrating my development rig to RP2350A0A2. Unfortunately, as I also noted in #409, it's hard to trace something that comes and goes, regardless of whether the underlying issue turns out to be HW and/or SDK/SW related. Of course, if/when it rears its ugly head again, I'll try to dig deeper... :| )

Thanks!

Who exactly has problems with SPI in this thread and how is it related to E9 (updated) ?

You are complaining about some performance issues in software, giving a rather vague description:
I've also experienced a few unexplainable glitches along the way, most prominently some weird occassional SPI performance glitches where I could see the calling loops drop by a factor 100x or so in performance (e.g. from ~70000 cycles per second down to ~700 cycles per second for one of said loops, i.e. not to standstill but very low performance).

It's a long way down to the hardware signaling. You have to investigate more, do a debugging, and if you suspect some hardware issues, take a trace of the signals and look for something weird.

BTW I've just tested again:

MikeDB wrote:

gmx wrote: ↑

Fri Sep 06, 2024 3:12 pm

Okay, can somebody with a Pico2 try this.
Get the pin in a latched up state.
Set the SIO on that pin to be logic 0 and enable the output, then disable the output say 1uS later.
Has the latched up state gone ?

Yes, I did it on the first try and it seems to clear the "latch-up" in no time, with a slew rate definitely exceeding my scope.

Have you tried shorter and shorter times for the logic 0 ? 1 (overclocked :-) clock cycle even ?

So, at the default 150 MHz processor clock, I put the pin in "latch" state by using the internal resistors, then directly from SIO enabled and disabled the output (with 0 on output register) for a very short time (6.66 ns).
It cleared the "latch" in less than 2.5 ns. at the minimum drive strength of 2mA, without any visible glitches or delays in the middle. It behaves the same with the internal pull-up enabled ("non-latched").

Therefore, most probably, it shouldn't be a problem driving the pin, at least not below 200 MHz.

UPDATE: Setting stronger drive 8-12 mA, brings the edge to an average of 1.5 ns, no glitches

gmx wrote: ↑

Sat Sep 07, 2024 5:19 am

MikeDB wrote: Have you tried shorter and shorter times for the logic 0 ? 1 (overclocked :-) clock cycle even ?

So, at the default 150 MHz processor clock, I put the pin in "latch" state by using the internal resistors, then directly from SIO enabled and disabled the output (with 0 on output register) for a very short time (6.66 ns).
It cleared the "latch" in less than 2.5 ns. at the minimum drive strength of 2mA, without any visible glitches or delays in the middle. It behaves the same with the internal pull-up enabled ("non-latched").

Therefore, most probably, it shouldn't be a problem driving the pin, at least not below 200 MHz.

UPDATE: Setting stronger drive 8-12 mA, brings the edge to an average of 1.5 ns, no glitches

This would seem to be the obvious software fix then - add some functions called 'Protected read' or something to the SDK so that anyone using non-logic gate driven inputs would use this. Shorting the output at 2.5mA for 1 clock cycle isn't going to damage anything.

Always interested in innovative audio startups needing help and investment. Look for InPoSe Ltd or Future Horizons on LinkedIn to find me (same avatar photograph)

gmx wrote: ↑

Sat Sep 07, 2024 5:19 am

Who exactly has problems with SPI in this thread and how is it related to E9 (updated) ? [...] You have to investigate more, do a debugging, and if you suspect some hardware issues, take a trace of the signals and look for something weird.

As others have already noted, not every individual/hobbyist/organization/maker/company/whatever will have access to the time/money/resources/equipment/knowledge/whatever to perform such low level debugging. What anyone can do, however, is to report on possible issues experienced that can not be explained by our own levels of insight into different areas (read: none of us are masters in all areas), in the hope of soliciting feedback and/or confirmation from others on said issues.

I may have misread the overall basis for this thread's discussion when I found and read it very late last night, thinking others were (seemingly, as I noted) somehow affected by or suspecting gremlins, and if so I apologize. In any case, though, there are still quite a few of us - including some apparent very frequent posters on this forum - that have asked the very valid (at least prior to the updated E9?) question of whether SPI/I2C/PIO/etc could in theory be implicitly affected by E9 and if so under which circumstances. If you're developing something for commercial purposes, sometimes your number one priority is to reduce risk and possibly related damages to your business. Given this, even if there turns out to be a SW workaround, you might choose to wait for a new silicon revision just to reduce that risk. In fact, I'm using a headered Pico2 for my product rather than brewing my own RP2xxx based design for this very reason - to reduce risk over the lifetime of the product by leveraging the official (and normally solid) product of a big company that's also promised to remain in production "until at least January 2040". And please, do not blindly suggest people go back to RP2040 if they're afraid of gremlins - the differences between it and the RP2350 are too significant to pass on, in more ways than one, for many of us and especially if you're forward-looking with regards to your product or business.

Finally, as we've seen over the past couple of weeks, even the folks with the most insight into the HW/SW sides of this issue had it right from day one, and we possibly still haven't gotten to the bottom of it (?) (nb. there may of course also be some degree of damage preventing/limiting factors in play here where careful wording might be necessary in order not to exacerbate the issue or risk to said parties). Anyway, with the updated E9 out, I thought it was fair to ask @jdb, as an (associate) official spokesperson of RPi on these forums, to elaborate further on the overall fears (FUD?) expressed in this thread.

Thanks,

BR//Karl @xoblite