Hi to all,
We recently ran in to a problem using stunnel on our AIX 5.2 machines. This is our situation:
we are running stunnel as a client to protect http (and a proprietary protocol) traffic between our company and a customer. We are using client-athentication to grant access to the server and on our client-side we validate the server-certificates. This all works fine.
Due to new security-restrictions, we are moving all our private-key material into HSM-boxes, in our case these are nCipher nShield HSM's. According to the documantion on there website ( http://active.ncipher.com ) we builded openssl nCipher-enabled. Based on this openssl-version, we builded stunnel. All the build-options are shown below in this message. The compiler used is the IBM C compiler.
According to the manuals from stunnel we needed to put in an extra line to the config-file which shows "engine = chil" (chil is the Cryptographic Hardware Interface Library, the way the nCipher HSM's are known to openssl)
After all this, we created key's in the hsm, using the nCipher tools and loaded them into stunnel (using this method you get a "pseudo-pem-file", which is actually a reference to the hardware-key). Fire-up stunnel using the commands from nCipher to preload the key's and we are all good to go!
When this all seems to work very stable after a few months, with thousands of connections and ditto megabytes transfered, we wanted to have all the logging to our syslogserver (it runs syslog-ng) and this is the place where my problem hits in: When commenting-out the option "output = /home/bart/stl/var/stunnel.log" and restarting stunnel, stunnel dies after the first or second connection, leaving no logging, corefiles or whatever behind.
Using plain key's (so no HSM, but with the same build which is aware of the HSM), syslog is no problem. Using HSM-key's with plaintext logfiles (no syslog) works also excellent.
As i don't have any ideas anymore where to search, i want to ask you if somebody ever have seen this behaviour or knows where it could be caused by. (And maybe how to solve it....)
Thanks in advance!
Best regards,
Bart Somers.
############################# # Build- and config-options # #############################
bart@server1:./stunnel -version stunnel 4.11 on powerpc-ibm-aix5.2.0.0 UCONTEXT+POLL+IPv4 with OpenSSL 0.9.8 05 Jul 2005
Global options cert = home/bart/stl/etc/stunnel/stunnel.pem ciphers = ALL:!ADH:+RC4:@STRENGTH debug = 5 key = /home/bart/stl/etc/stunnel/stunnel.pem pid = /home/bart/stl/var/run/stunnel.pid RNDbytes = 64 RNDfile = /dev/urandom RNDoverwrite = yes session = 300 seconds verify = none
Service-level options TIMEOUTbusy = 300 seconds TIMEOUTclose = 60 seconds TIMEOUTconnect = 10 seconds TIMEOUTidle = 43200 seconds
bart@server1:./openssl version OpenSSL 0.9.8 05 Jul 2005
compile-options: Openssl:
$ export LDFLAGS="-L/opt/nfast/toolkits/hwcrhk" $ ./config --prefix=/home/bart/stl –DNFast –ldl make and make install
Stunnel:
$ export CPPFLAGS="-I/home/bart/stl/include" $ export LDFLAGS="-L/opt/nfast/toolkits/hwcrhk" $ ./configure –-prefix=/home/bart/stl –-with-ssl=/home/bart/stl -–disable-libwrap make and make install
Config-file: Include engine = chil
Start-up: $ with-nfast –f /home/bart/stl/tmp/ff /home/bart/stl/sbin/stunnel /home/bart/stl/etc/stunnel/stunnel.conf
Relevant information from the config-file:
engine = chil cert = /home/bart/stl/etc/stunnel/my-key.pem key = /home/bart/stl/etc/stunnel/my-key-hsm.pem ;(no chroot options used) pid = /prj/stl_004/tmp/test/stunnel-client.pid debug = 7 ; output = /home/bart/stl/var/stunnel.log # when in comment, problems! client = yes
[test] accept = 9400 connect = 10.10.10.10:443
All other values are the default from the example.
On Wednesday 18 of January 2006 15:19, Bart Somers wrote:
We recently ran in to a problem using stunnel on our AIX 5.2 machines.
[cut]
As i don't have any ideas anymore where to search, i want to ask you if somebody ever have seen this behaviour or knows where it could be caused by. (And maybe how to solve it....)
[cut]
stunnel 4.11 on powerpc-ibm-aix5.2.0.0 UCONTEXT+POLL+IPv4 with OpenSSL 0.9.8 05 Jul 2005
I'd try to: 1. Upgrade stunnel to version 4.14. 2. Try to monitor it with "truss" command. 3. Try another threading model ./configure --with-threads=model to select threading model (ucontext/pthread/fork)
Best regards, Mike
On 1/18/06, Michal Trojnara Michal.Trojnara@mobi-com.net wrote:
On Wednesday 18 of January 2006 15:19, Bart Somers wrote:
We recently ran in to a problem using stunnel on our AIX 5.2 machines.
[cut]
stunnel 4.11 on powerpc-ibm-aix5.2.0.0 UCONTEXT+POLL+IPv4 with OpenSSL 0.9.8 05 Jul 2005
I'd try to:
- Upgrade stunnel to version 4.14.
- Try to monitor it with "truss" command.
- Try another threading model ./configure --with-threads=model to select threading model (ucontext/pthread/fork)
I upgraded to 4.14 and builded stunnel with all three threading models:
1) ucontext gave the same problem as the above mentioned 4.11 ucontext version 2) pthread gave me a strange error during the "./config" phase:
checking pthread.h usability... no checking pthread.h presence... yes configure: WARNING: pthread.h: present but cannot be compiled configure: WARNING: pthread.h: check for missing prerequisite headers? configure: WARNING: pthread.h: see the Autoconf documentation configure: WARNING: pthread.h: section "Present But Cannot Be Compiled" configure: WARNING: pthread.h: proceeding with the preprocessor's result configure: WARNING: pthread.h: in the future, the compiler will take precedence configure: WARNING: ## ------------------------------ ---- ## configure: WARNING: ## Report this to the stunnel lists. ## configure: WARNING: ## ---------------------------------- ## checking for pthread.h... yes
After this, i couldn't build the software in the "make" phase. I'm sorry i forgot to capture that error, but it came from the pthread.h header file. I'll capture the entire error next week if you're interested. For the time being i'm also looking for another build-machine, because i'm not sure this problem can be caused by an old version of GCC (2.9, shame).
3) the fork option builded nicely, no warnings whatsoever, but has the same behaviour as the ucontext build, crashes with both syslog and engine=chil enabled.
I haven't had the possibility to use truss on the binary's, hopelfully i'll try that next week.
Thanks for your information so far, if i have any news, i'll post it here.
Best regards,
Bart Somers
Bart Somers wrote:
- the fork option builded nicely, no warnings whatsoever,
but has the same behaviour as the ucontext build, crashes with both syslog and engine=chil enabled.
Crashing fork build is really strange since the main process does not use the engine at all... Did you properly "make clean && make && make install"?
I'm afraid I can't help you unless you could provide me with a testing environment (a shell account).
Best regards, Mike
On 2/10/06, Michal Trojnara Michal.Trojnara@mobi-com.net wrote:
Bart Somers wrote:
- the fork option builded nicely, no warnings whatsoever,
but has the same behaviour as the ucontext build, crashes with both syslog and engine=chil enabled.
Crashing fork build is really strange since the main process does not use the engine at all... Did you properly "make clean && make && make install"?
OK, maybe crashing is a heavy word, it just stops working without any notice. I'm allmost 100% sure i did a "make clean" before i started, but after the weekend (the machine is at the office) i'll rerun this one from a fresh source-tree (direct from .tar) and inform you.
I'm afraid I can't help you unless you could provide me with a testing
environment (a shell account).
Thanks in advance! I will see what is possible about this.
Best regards,
Bart Somers