[stunnel-users] Stunnel Dies at 3500 connections...

[bear at bears.org]

Hey!

I'm having a problem with a stunnel setup I have, and was wondering if 
anyone could help?

I have two machines, one is acting as a simulator of clients.  It uses the 
MINA library from apache to simulate a bunch of clients of the server. 
And the other acting as a server.  Without stunnel in place, the client 
and server chat with no trouble... even with as many as 10k simulated 
clients.  The protocol they communicate is just compressed binary data 
over standard TCP/IP.  20-50 bytes each direction per exchange.

The behavior changes once stunnel is in place.  The server behaves just 
fine, but the client acts very strange... in ways that make me think that 
stunnel is hitting some OS limit.  (The files ulimit is set waaaaaay 
higher than the number of clients)

On starting the client application, it begins making connections through 
the stunnel tunnel.  The first 1500 or so are fine, but after that the 
load on the machine begins to climb.  It gets to 5 or so around 2000.  If 
I only do 2000 clients, the load then drops down into the 0.5 range, and 
the system runs okay.  However, if I set the number of simulated clients 
higher... say 4000, the load continues to climb.  When it gets to 3500 or 
so, the load goes above 500.  The machine becomes unusable, and after a 
bit, stunnel dies with the message KILLED.

I tried changing how quickly the clients connect, but that seemed to make 
no difference.  Also, running two stunnels for the clients, and sending 
2000 to each, for instance, causes it to still break down around 3500 or 
so.  (Which is why I think it is an os limit)

Any suggestions on this?  Thanks!

Peace,
Gary

version
Stunnel 4.20

startup info
2007.09.24 12:09:50 LOG7[26537:3086939840]: RAND_status claims sufficient 
entropy for the PRNG
2007.09.24 12:09:50 LOG7[26537:3086939840]: PRNG seeded successfully
2007.09.24 12:09:50 LOG4[26537:3086939840]: Wrong permissions on 
hcs-key-and-cert-1024.pem
2007.09.24 12:09:50 LOG7[26537:3086939840]: Certificate: 
hcs-key-and-cert-1024.pem
2007.09.24 12:09:50 LOG7[26537:3086939840]: Certificate loaded
2007.09.24 12:09:50 LOG7[26537:3086939840]: Key file: 
hcs-key-and-cert-1024.pem
2007.09.24 12:09:50 LOG7[26537:3086939840]: Private key loaded
2007.09.24 12:09:50 LOG7[26537:3086939840]: SSL context initialized for 
service hcsuno
2007.09.24 12:09:50 LOG5[26537:3086939840]: stunnel 4.20 on 
i686-pc-linux-gnu with OpenSSL 0.9.8d 28 Sep 2006
2007.09.24 12:09:50 LOG5[26537:3086939840]: Threading:PTHREAD SSL:ENGINE 
Sockets:POLL,IPv4 Auth:LIBWRAP
2007.09.24 12:09:50 LOG6[26537:3086939840]: file ulimit = 80000 (can be 
changed with 'ulimit -n')
2007.09.24 12:09:50 LOG6[26537:3086939840]: poll() used - no FD_SETSIZE 
limit for file descriptors
2007.09.24 12:09:50 LOG5[26537:3086939840]: 39062 clients allowed
2007.09.24 12:09:50 LOG7[26537:3086939840]: FD 3 in non-blocking mode
2007.09.24 12:09:50 LOG7[26537:3086939840]: FD 4 in non-blocking mode
2007.09.24 12:09:50 LOG7[26537:3086939840]: FD 5 in non-blocking mode
2007.09.24 12:09:50 LOG7[26537:3086939840]: SO_REUSEADDR option set on 
accept socket
2007.09.24 12:09:50 LOG7[26537:3086939840]: hcsuno bound to 0.0.0.0:4090
2007.09.24 12:09:50 LOG7[26537:3086939840]: Created pid file 
/tmp/stunnel1.pid

uname -a
Linux testbed-client1 2.6.9-42.ELsmp #1 SMP Sat Aug 12 09:39:11 CDT 2006 
i686 i686 i386 GNU/Linux
libc 2.3.4

stunnel -version
stunnel 4.20 on i686-pc-linux-gnu with OpenSSL 0.9.8d 28 Sep 2006
Threading:PTHREAD SSL:ENGINE Sockets:POLL,IPv4 Auth:LIBWRAP

Global options
debug           = 5
pid             = /usr/local/var/run/stunnel/stunnel.pid
RNDbytes        = 64
RNDfile         = /dev/urandom
RNDoverwrite    = yes

Service-level options
cert            = /usr/local/etc/stunnel/stunnel.pem
ciphers         = ALL:!ADH:+RC4:@STRENGTH
key             = /usr/local/etc/stunnel/stunnel.pem
session         = 300 seconds
sslVersion      = SSLv3 for client, all for server
TIMEOUTbusy     = 300 seconds
TIMEOUTclose    = 60 seconds
TIMEOUTconnect  = 10 seconds
TIMEOUTidle     = 43200 seconds
verify          = none

gcc -v
Reading specs from /usr/lib/gcc/i386-redhat-linux/3.4.6/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
--infodir=/usr/share/info --enable-shared --enable-threads=posix 
--disable-checking --with-system-zlib --enable-__cxa_atexit 
--disable-libunwind-exceptions --enable-java-awt=gtk 
--host=i386-redhat-linux
Thread model: posix
gcc version 3.4.6 20060404 (Red Hat 3.4.6-3)

openssl version
OpenSSL 0.9.7a Feb 19 2003

In the config, I have:

client=yes
debug=debug
socket = l:TCP_NODELAY=1
pid = /tmp/stunnel1.pid
foreground=yes

[hcsuno]
accept=4090
connect = testbed-hcs1:4094
cert = hcs-key-and-cert-1024.pem
key = hcs-key-and-cert-1024.pem


   *********    *****    **                  Gary Coulbourne
  *************************.*                 Bear Activist
****** *********** ** *******o
******* ********* **** ****`-            Systems Administrator
******* ********* *****                   http://www.bears.org
  ****** ********** ****                      bear at bears.org
## *****  ***** ##  ****                         KB3INA
### *****      ###   ****           Animal Conservation/Preservation
#,,, ***,,,    ##,,,  **,,,