Hello,
Flamegraph profiling on stunnel has shown that most time is NOT spent in encryption/decryption, but in sending/receiving data.
Experimental setup: ``` CPU: 2x 18-core Intel Xeon Gold 6354 NIC: Intel E810-XXV, 100 Gbit/s Kernel: 6.14.8-300.fc42.x86_64 x86_64 Mem: 251.32 GiB OS: Fedora Linux 42 openssl version: OpenSSL 3.2.4 11 Feb 2025 (Library: OpenSSL 3.2.4 11 Feb 2025) stunnel: 5.75 from git ```
Network, encryption and memory copy speeds converted into Gbit/s on the above HW: ``` iperf3: 40Gbit/s openssl speed -evp aes-256-gcm: 72Gbit/s perf bench memcpy: 100Gbit/s ```
Under ideal conditions we should be able to achieve at most ~20.5Gbit/s, however I only measured ~7.6Gbit/s.
The scripts that I used to perform measurements are included in the attached patch 3.
See also related discussion on curl: https://github.com/curl/curl/pull/17548
Enabling readahead, avoiding excessive alloc/free of buffers, increasing buffer sizes, and improving buffer handling can all improve performance.
For now I'm sending just the first small patches as proofs of concept, each of them one-liners. I hope you can include these in some form in the next release.
Eventually these should probably be exposed as configuration flags in stunnel.conf, I can help implementing that, or I can leave implementing that to you if you prefer.
In fact readahead was enabled in previous versions of stunnel, and then disabled again, so I assume you ran into some bugs with it on certain protocols?
Increasing the buffer size is not a clear win (as with curl), due to the excessive use of memmove/memset, and that SSL_read/SSL_write only still processes 1 TLS record at a time. I have some further changes that can improve that, but the patches are larger and I haven't finished testing them for correctness yet. Please let me know if you'd want these as patches, or if you'd rather implement them yourself.
Edwin Török (3): openssl: enable readahead openssl: disable SSL_MODE_RELEASE_BUFFERS Benchmark scripts
src/ctx.c | 5 +- tests/benchmark/benchmark.sh | 120 +++++++++++++++++++++++ tests/benchmark/launch.sh | 185 +++++++++++++++++++++++++++++++++++ 3 files changed, 309 insertions(+), 1 deletion(-) create mode 100644 tests/benchmark/benchmark.sh create mode 100755 tests/benchmark/launch.sh
-- 2.43.5