Hi
I'm using the following s/w versions on RHEL3:
Stunnel: 4.04 (also experimenting with 4.22, no difference so far) Samba: 3.0.9-1.3E.10 Kernel: 2.4.21-32.0.1.EL
Stunnel is used to encrypt samba connections from linux clients because redhat 3 doesn't support NTLMv2 (cifs not standard and probably not an option). To list 5000 files (simply typing ls within a mounted directory on the client) it consistently takes around 20-23 seconds to return the data. Listing is almost instantaneous when using a windows client, or using a linux client without stunnel. As a side note, if I pipe the result to /dev/null it takes around 9 seconds (?). The network forwarding path from the samba client to samba server is: smbclient > localhost:924 > stunnel > remotehost:923 > stunnel > samba server (port 446)
Tcpdump shows that when using stunnel about 10500 packets are generated, minus stunnel it's more like 500. I'd expect some overhead related to SSL, but 21 times the traffic seems a little excessive. I've experimented with socket options such as TCP_NODELAY, SO_LINGER, SO_RCVLOWAT, SO_OOBINLINE, etc with no improvement at all. However, my understanding of these is pretty superficial so I'm not confident I've exhausted all options here (ie perhaps combining multiple settings at once).
I've got stunnel debug set to 7 on client and server. No errors and no logging at all except for the initial handshake when the mount is created. Including the tcpdump would probably be excessive at this stage. In summary, using stunnel the data gets transmitted in packets usually containing around 200 bytes, whereas without stunnel it's mostly 1408 byte packets.
Any suggestions?
Thanks
Paul
On Wed, 2008-04-02 at 14:12 +1100, Paul Kerin wrote:
I've got stunnel debug set to 7 on client and server. No errors and no logging at all except for the initial handshake when the mount is created. Including the tcpdump would probably be excessive at this stage. In summary, using stunnel the data gets transmitted in packets usually containing around 200 bytes, whereas without stunnel it's mostly 1408 byte packets.
Any suggestions?
Given that you are also experiencing data-expansion overall (despite getting smaller individual packets), I'm guessing that you are losing coalescing as data enters the SSL tunnel. I assume the majority of data is in the server-->client direction if you're doing an "ls", so what is probably happening is this;
1. in the unencrypted case, small amounts of data are being emitted from the samba server via socket writes, but it is coalescing within the TCP stack due to Nagle (TCP_NODELAY).
2. in the encrypted case, samba server output is being emitted locally into the SSL tunnel which produces delineated SSL records to be written out on to the network. Even ignoring the expansion of SSL records relative to small unencrypted inputs, there may also be sufficient additional processing overhead in stunnel that Nagle doesn't get a chance to coalesce these 200byte SSL records into larger TCP packets.
If my hunch is right, the individual writes from the samba server are probably quite small (perhaps one per file in the directory listing response) because even once they're encapsulated in SSL, the records are still only 200 bytes. Depending on the SSL/TLS cipher-suite you're using, that could be a significant expansion relative to the unencrypted data. Also, in the unencrypted case (no stunnel), I imagine that if you ran 10 copies of "openssl speed rsa1024" in a while(1) loop on the server, you may find that you start getting smaller packets from the server rather than the nice coalescing you're currently seeing. Ie. if you slow samba down a little, the Nagle algorithm may start to fail you in the same way you're already seeing with stunnel.
Also, certain combinations of SSL/TLS version and cipher-suite produce a lot of extra "chatter" over the wire. Consider playing with the versions and cipher-suites of your SSL tunnel to see if you can get a tunnel that gets you a better data rate. Eg. the "default" cipher-suites for openssl apps are often set to use ephemeral key-exchange cipher-suites because they have a theoretical security advantage ("perfect forward secrecy"), but they're also typically a lot bulkier (though this would be most observable in the initial handshake rather than in the data-plane). Also try SSLv3 rather than TLSv1, just for kicks...
What else? If the NAGLE issue is the key difference here, the localhost connection between the stunnel server and the samba server is probably too direct for samba's output to coalesce into larger packets the way it does when it goes unencrypted onto the network. You could possibly confirm this by moving the stunnel server to a different machine (even back on the same machine as the stunnel client, just for curiosity's sake) so that the samba output is still going unencrypted through a real network link before it reaches the SSL tunnel. If this is the issue, then you would observe that the SSL records within the encrypted tunnel get larger, there should be fewer of them, and the expansion ratio of data-size between unencrypted and encrypted network data should reduce significantly (because you're possibly encrypting ~1Kb per SSL record, rather than encrypting 10-50 bytes per SSL record, as may be the case currently).
Those are all the thoughts I have for now - let me know what you find and perhaps we can suggest something in consequence.
Cheers, Geoff
Thanks for the excellent response. It does seem to be a nagle/buffering issue of some sort. I set up stunnel as you suggested (so that all conns are localhost -> remotehost) and the problem disappeared. However, after again explicitly turning TCP_NODELAY off completely in samba and stunnel, restarting, etc, the problem remains in the localhost -> localhost configuration.
It'd be nice to fix it, but none of the socket options seem to change anything so will probably have to find a work around, such as keeping the number of files in the directory low or pushing harder to use cifs instead of stunnel.
Thanks again for your time.
Paul
On 4/3/08, Geoff Thorpe geoff@geoffthorpe.net wrote:
On Wed, 2008-04-02 at 14:12 +1100, Paul Kerin wrote:
I've got stunnel debug set to 7 on client and server. No errors and no logging at all except for the initial handshake when the mount is created. Including the tcpdump would probably be excessive at this stage. In summary, using stunnel the data gets transmitted in packets usually containing around 200 bytes, whereas without stunnel it's mostly 1408 byte packets.
Any suggestions?
Given that you are also experiencing data-expansion overall (despite getting smaller individual packets), I'm guessing that you are losing coalescing as data enters the SSL tunnel. I assume the majority of data is in the server-->client direction if you're doing an "ls", so what is probably happening is this;
- in the unencrypted case, small amounts of data are being emitted from
the samba server via socket writes, but it is coalescing within the TCP stack due to Nagle (TCP_NODELAY).
- in the encrypted case, samba server output is being emitted locally
into the SSL tunnel which produces delineated SSL records to be written out on to the network. Even ignoring the expansion of SSL records relative to small unencrypted inputs, there may also be sufficient additional processing overhead in stunnel that Nagle doesn't get a chance to coalesce these 200byte SSL records into larger TCP packets.
If my hunch is right, the individual writes from the samba server are probably quite small (perhaps one per file in the directory listing response) because even once they're encapsulated in SSL, the records are still only 200 bytes. Depending on the SSL/TLS cipher-suite you're using, that could be a significant expansion relative to the unencrypted data. Also, in the unencrypted case (no stunnel), I imagine that if you ran 10 copies of "openssl speed rsa1024" in a while(1) loop on the server, you may find that you start getting smaller packets from the server rather than the nice coalescing you're currently seeing. Ie. if you slow samba down a little, the Nagle algorithm may start to fail you in the same way you're already seeing with stunnel.
Also, certain combinations of SSL/TLS version and cipher-suite produce a lot of extra "chatter" over the wire. Consider playing with the versions and cipher-suites of your SSL tunnel to see if you can get a tunnel that gets you a better data rate. Eg. the "default" cipher-suites for openssl apps are often set to use ephemeral key-exchange cipher-suites because they have a theoretical security advantage ("perfect forward secrecy"), but they're also typically a lot bulkier (though this would be most observable in the initial handshake rather than in the data-plane). Also try SSLv3 rather than TLSv1, just for kicks...
What else? If the NAGLE issue is the key difference here, the localhost connection between the stunnel server and the samba server is probably too direct for samba's output to coalesce into larger packets the way it does when it goes unencrypted onto the network. You could possibly confirm this by moving the stunnel server to a different machine (even back on the same machine as the stunnel client, just for curiosity's sake) so that the samba output is still going unencrypted through a real network link before it reaches the SSL tunnel. If this is the issue, then you would observe that the SSL records within the encrypted tunnel get larger, there should be fewer of them, and the expansion ratio of data-size between unencrypted and encrypted network data should reduce significantly (because you're possibly encrypting ~1Kb per SSL record, rather than encrypting 10-50 bytes per SSL record, as may be the case currently).
Those are all the thoughts I have for now - let me know what you find and perhaps we can suggest something in consequence.
Cheers, Geoff
On Thu, 2008-04-03 at 14:47 +1100, PK wrote:
Thanks for the excellent response. It does seem to be a nagle/buffering issue of some sort. I set up stunnel as you suggested (so that all conns are localhost -> remotehost) and the problem disappeared. However, after again explicitly turning TCP_NODELAY off completely in samba and stunnel, restarting, etc, the problem remains in the localhost -> localhost configuration.
Yeah, I'm guessing here that stunnel itself is getting data directly from samba and forming outbound SSL records right away - so this is where you want the coalescing but you're not getting it. Then the fact you don't get buffering of the outbound SSL records into larger TCP packets is probably just bad luck, particularly as the most likely reason you're not fast enough to beat the ACK packets for Nagle is because you're too busy pulling in loads and loads of really small clear-text inputs and producing too many SSL records... (rinse and repeat). It's a "push-back" architecture problem that you can't fix in stunnel. Disabling TCP_NODELAY is unlikely to change anything as it should usually be off by default anyway (ie. Nagle turned on). The problem is that although Nagle is on, it isn't helping on your localhost connection between samba and stunnel.
Hmmm ... this may not work, but have you considered using an external interface address rather than loopback/localhost/127.0.0.1? Eg. rather than the stunnel server connecting to localhost:446, try having it connect to 192.168.0.1:446 (or whatever your NIC address is). If the reason you're getting no buffering is because the connection is too fast, this won't help unless the external interface is much slower than loopback (when bending packets back to itself). But if the reason was that buffering is inherently ignored on the loopback device, then you could get lucky if you use the external device address instead.
It'd be nice to fix it, but none of the socket options seem to change anything so will probably have to find a work around, such as keeping the number of files in the directory low or pushing harder to use cifs instead of stunnel.
If you get *really* stuck, you could always insert a nasty hack into the child-process routine within stunnel, eg. something like;
[...] static int buffering = 0; [...]
[...] /* find wherever we read the clear-text socket */ ret = read(sock); if (ret < 0) /* error-handling */ /* NASTY HACK - THIS IS THE NEW CODE*/ if (!buffering && ((ret > 0) && (ret < 512))) buffering = 5; if (buffering) { msleep(50); buffering--; } [...]
It's hideous, would never get accepted upstream (in any modified form), and would cause any purist to expunge generously into the nearest porcelain ... which is often true of practical solutions for people who have better things to do. ;-)
The idea is that if you get a small read, you're "too fast" and will probably continue to get small reads from the samba server (or samba client, on the stunnel client). With this trick, it'll start sleeping 1/20th of a second after each read for the next 5 reads. That means your reads are less frequent and should pick up more data each time - and after 5 such delays, we stop sleeping so that stunnel can process inbound data at full speed, hopefully the socket has accumulated a back-log of data (so you'll get maximally-sized chunks from your reads until you "catch up"). Eventually you'll catch up, get a small read, and go back into this throttling mechanism. This would be extremely unwise for latency-critical/interactive protocols, but could help significantly for "download or upload"-style protocols. Letting a back-log build up can improve your efficiency. If you're listing 5000 files and it's taking 23 seconds, this seems like a good candidate.
Anyway, might be worth trying if you run out of options ... please let me know how you get on.
Cheers, Geoff
Hallo,
Can you tell me what are the default timeout values listed below. Perhaps you can add them to the documentation.
;time to wait for expected data TIMEOUTbusy = seconds
;time to wait for close_notify (set to 0 for buggy MSIE) TIMEOUTclose = seconds
;time to wait to connect a remote host TIMEOUTconnect = seconds
;time to keep an idle connection TIMEOUTidle = seconds
Thanks and Best Regards Jan
-
Geoff Thorpe -
Jan.Haering＠rz-kiru.de -
Paul Kerin -
PK