New LED hardware driver: WLED Pixel Bus

[DedeHai]

This is a major update adding WLED Pixel Bus written from scratch to replace NeoPixelBus and it packs a lot of useful features. From a user perspective the main improvements are less memory use and dynamic adjustment of LED timing to eliminate flickering by fine-tuning the timing from -30% to + 30% in 10% steps (dropdown menu).

The new hardware driver structure allows for fully dynamic updates of LED outputs and LED timings. It also adds glitch free parallel SPI output support on the C3 as well as parallel bit-banging on all platforms including ESP8266.
All parallel outputs (except bit-bang) use ping-pong DMA buffers instead of fully pre-filled buffers - memory usage is optimized and DMA buffers are independent of number of LEDs. Also the calculation from LED colors to DMA buffers is highly speed optimized, making the driver blazing fast - I saw 30% FPS improvements in some cases.
The fully dynamic nature of the driver allows for a "Custom digital bus" in the LED settings to support virtually any LED bus type out there - specify timing, number of color channels, invert any color channel or set any color channel combination. The driver also supports inverting the output signal on any pin (ESP32 only).

The LED config UI was updated to fully support the new driver options.

A lot more testing is required for all different kind of LEDs on all platforms. I am sure there is a ton of bugs and kinks to iron out.

I would like to take this opportunity to thank @Makuna for his outstanding work on NeoPixelBus which served me well as a reference on hardware configurations and special LED types.

• saves over 10k of heap on ESP32 and about 3k on ESP8266
• increases available LED outputs on ESP32 C3 from 2 to 14 (2x RMT, 4x parallel SPI, 8x bit-bang)
• increases available LED outputs on ESP32 S2 with 8x bit-bang if I2S is not available
• increases available LED outputs on ESP8266 3 to 6 (1x UART, 1xI2S DMA, 4x bit-bang)
• allows over 1000 LEDs again on ESP8266 thanks to reduced RAM footprint, it does not use a global buffer but writes to LED buffers directly which is slower but uses less RAM.
• adds support for TM1815
• improved low-brightness color accuracy to TM1814, TM1815 and APA102 by utilizing LED current settings
• improved color resolution on 16bit LED types (UCS8903, UCS8904, SM16825)
• if needed I2S (ESP32 and ESP32 S2) and LCD (ESP32 S3) driver can output on 16 parallel pins changing a #define
• automatic calculation and optimized use of multiple RMT memory blocks, reducing ISR frequency when not using all RMT outputs and thus increases FPS

ESP32 flash and RAM usage comparison (bit bang disabled)

using NeoPixelBus
RAM: [== ] 24.9% (used 81576 bytes from 327680 bytes)
Flash: [======== ] 83.1% (used 1306493 bytes from 1572864 bytes)
Free Heap: (6 outputs, 256 each): 79.8k

using WLEDpixelBus
RAM: [== ] 24.9% (used 81680 bytes from 327680 bytes)
Flash: [======== ] 83.0% (used 1306053 bytes from 1572864 bytes)
Free Heap: (6 outputs, 256 each): 92.8k

Summary by CodeRabbit

• New Features
  • Added per-bus Bus Speed Factor (SF/bsf) settings.
  • Added Custom Bus support with channel mapping, inversion, custom timing, and per-channel colors.
  • Expanded supported output driver/type options (bit-bang, parallel SPI, RMT, I2S, LCD where available) and added new LED type options (e.g., TM1815 / custom bus).
• Bug Fixes
  • Improved ESP8266 direct-to-bus pixel handling and refined gamma/brightness behavior across realtime and particle rendering.
• Chores
  • Updated capability limits and the LED settings UI + config import/export for the new bus features.

[DedeHai]

@willmmiles I made a lot of updates and it is now in a pretty good shape. I compiled a TODO list in WLEDpixelBus.cpp with the important things that still should be updated / checked / fixed, all of them are improvements that do not need immediate fixing.
If you find the time, please have another look at it and check if you agree with the general approach of handling things WRT future compatibility. The drivers themselves are (or were at one point) tested. ESP8266 slipped my testing recently so I can't vouch for it not having issues, I focused on the V5 updates mostly.

[willmmiles]

OK! I should be able to get some time to do a detailed review on the weekend.

[willmmiles]

I'm finding it challenging to review this properly - I just keep exploding in a hundred nitpicks. It's hard to know where to start and I feel bad pushing suggestions in dribs and drabs.

Bigger things I've noticed so far:

• WLEDpixelBus.h has duplicate copies of code from colors.h. This component already isn't isolated from WLED internals (it's importing const.h for WLED's type mapping). If we want to maintain the conceit that this is an independent library, pull out colors.h to its own "library" that both WLED and WLEDpixelBus can depend on. (wled::colors ?? ) Otherwise we should use the definitions for colors.h directly. (It makes sense in its way: one task performed here is to map between WLED's internal color format to the output pixel format.)
• The split between BusDigital and PixelBus is costly -- calls like setPixelColor invoke two vtable lookups. (And a third dynamic dispatch for the color encoding -- switch and virtual are conceptually similar.) I don't have an immediate suggestion; this is mostly a performance tuning concern and we don't necessarily have to address it up front. From an API perspective, the key change would be moving to a factory function (eg replacing new BusDigital(...) to makeDigitalBus(...)) -- this would let the PixelBus system someday return a fully qualified Bus type with fewer dynamic dispatches.
• CustomBusConfig -- Good call to factor this separately from the common BusConfig, but directly embedding it in BusConfig gives up all the advantages. I suggest using a type-erased bus-specific-settings slot in the BusConfig level (eg. instead of CustomBusConfig custom, do std::shared_ptr<void> customBusSettings or the like), and then cast it to the bus-specific struct during construction. This lets each Bus implement its own custom settings struct and doesn't require sharing the type outside of the local bus implementation and local bus settings parser. (The next step will be to factor the config parser on a per-TYPE_ basis, leveraging getLEDTypes(), but that's a plan for another day.)

[DedeHai]

Thanks for taking a dive.

I'm finding it challenging to review this properly - I just keep exploding in a hundred nitpicks. It's hard to know where to start and I feel bad pushing suggestions in dribs and drabs.

Fully understand, its a major overhaul and a "diamond in the rough", I also did not start out with a full concept but kept adding and replacing things (as you can see in the PR having over 200 commits). Each time I read through the code I find things that could be done better or more clearly but let's do it in increments, its just too much code to have a clean overview and its easy to get lost in details.

Bigger things I've noticed so far:

• WLEDpixelBus.h has duplicate copies of code from colors.h. This component already isn't isolated from WLED internals (it's importing const.h for WLED's type mapping).

This was actually intentional - I did try to keep current code & structure intact as much as possible, also to not change things and loose backwards compatibility. I am not saying it's the way it has to be in the end.

If we want to maintain the conceit that this is an independent library, pull out colors.h to its own "library" that both WLED and WLEDpixelBus can depend on. (wled::colors ?? ) Otherwise we should use the definitions for colors.h directly. (It makes sense in its way: one task performed here is to map between WLED's internal color format to the output pixel format.)

I did not build it up to be an independent library - but ultimately that should be the goal. I did focus on getting the drivers to work and somehow integrate them and less on making it as clean - as I am sure you can tell ;)

• The split between BusDigital and PixelBus is costly -- calls like setPixelColor invoke two vtable lookups. (And a third dynamic dispatch for the color encoding -- switch and virtual are conceptually similar.) I don't have an immediate suggestion; this is mostly a performance tuning concern and we don't necessarily have to address it up front. From an API perspective, the key change would be moving to a factory function (eg replacing new BusDigital(...) to makeDigitalBus(...)) -- this would let the PixelBus system someday return a fully qualified Bus type with fewer dynamic dispatches.

Agreed, the point from above applies here as well: I kept the old structure to not get tangled up and keep my focus on the pixel-pushing driver. I am not sure on where to make the split i.e. should the WLEDpixelBus replace bus digital or be more of an independent library.
edit:
it is still (way) faster than NPB - apart from the lower level driver being faster I think the main speedup is from not doing as many color conversions, those are expensive, the color order lookup is also much faster.

• CustomBusConfig -- Good call to factor this separately from the common BusConfig, but directly embedding it in BusConfig gives up all the advantages. I suggest using a type-erased bus-specific-settings slot in the BusConfig level (eg. instead of CustomBusConfig custom, do std::shared_ptr<void> customBusSettings or the like), and then cast it to the bus-specific struct during construction. This lets each Bus implement its own custom settings struct and doesn't require sharing the type outside of the local bus implementation and local bus settings parser. (The next step will be to factor the config parser on a per-TYPE_ basis, leveraging getLEDTypes(), but that's a plan for another day.)

sounds like a good idea, I have no real grasp on that concept though - it's a bit above my every-day C++ knowledge but I am sure I can learn that abstraction level.
I at one point had custom bus config as it's own config but then also wanted to be able to modify "standard" configs, i.e. take any of the supported bus formats and reorder the output format. If I understand correctly, your suggestion gets the best of both worlds.

Maybe we should have a call to discuss some of this?