We have the following setup in our production servers.
> Windows [ ... -> Apache -> Stunnel 4.50 ] --> AIX [Stunnel 4.50 -> JBoss]
* As we had experienced the Stunnel in AIX died in a random time during testing stage, we added a cron job to start Stunnel in case stunnel disappeared.
Recently, in our production servers, the stunnel disappeared and being started by corn repeatedly during a short period of time in the same day. We so far encounter similar problem for 2 times. Below are the restart time of the most recent incident.
> 11:06:00,11:16:01,11:17:00,11:19:00,11:22:00,11:23:00,11:29:01,11:32:00,11:34:01,11:35:00,11:39:01
> ...
What are the possible causes that would make it disappeared after those processing of SIGCHLD signal?
What may caused the signal dispatching other than killing them manually for shutdown?
2012.05.23 11:16:01 LOG5[11862064:1]: stunnel 4.50 on powerpc-ibm-aix6.1.0.0 platform
2012.05.23 11:16:01 LOG5[11862064:1]: Compiled/running with OpenSSL 0.9.8m 25 Feb 2010
2012.05.23 11:16:01 LOG5[11862064:1]: Threading:PTHREAD SSL:ENGINE Auth:none Sockets:POLL,IPv6
2012.05.23 11:16:01 LOG5[11862064:1]: Reading configuration from file /opt/stunnel/4.50/prd/stunnel0-app/stunnel-server.conf
2012.05.23 11:16:01 LOG6[11862064:1]: Unable to retrieve any random data from /home/appadmin/.rnd
2012.05.23 11:16:01 LOG7[11862064:1]: Wrote 0 new random bytes to /home/appadmin/.rnd
2012.05.23 11:16:01 LOG6[11862064:1]: Initializing SSL context for service prdweb_in
2012.05.23 11:16:01 LOG4[11862064:1]: Insecure file permissions on /opt/stunnel/4.50/prd/stunnel0-app/PRD.private.key
2012.05.23 11:16:01 LOG7[11862064:1]: Certificate: /opt/stunnel/4.50/prd/stunnel0-app/PRD.certnew-b64.cer
2012.05.23 11:16:01 LOG7[11862064:1]: Key file: /opt/stunnel/4.50/prd/stunnel0-app/PRD.private.key
2012.05.23 11:16:01 LOG7[11862064:1]: Could not load DH parameters from /opt/stunnel/4.50/prd/stunnel0-app/PRD.certnew-b64.cer
2012.05.23 11:16:01 LOG7[11862064:1]: Using hardcoded DH parameters
2012.05.23 11:16:01 LOG7[11862064:1]: DH initialized with 2048-bit key
2012.05.23 11:16:01 LOG7[11862064:1]: SSL options set: 0x01000004
2012.05.23 11:16:01 LOG7[11862064:1]: Option SO_REUSEADDR set on accept socket
2012.05.23 11:16:01 LOG7[11862064:1]: Service prdweb_in opened FD=7
2012.05.23 11:16:01 LOG7[12517524:1]: Created pid file /opt/stunnel/4.50/prd/stunnel0-app/stunnel.pid
2012.05.23 11:16:01 LOG7[12517524:258]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:258]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:515]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:515]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:772]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:772]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:1029]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:1029]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:1286]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:1286]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:1543]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:1543]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:1800]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:1800]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:2057]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:2057]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:2314]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:2314]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:2571]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:2571]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:3085]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:3085]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:2828]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:2828]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:3342]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:3342]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:3599]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:3599]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG6[12517524:1286]: SSL accepted: new session negotiated
2012.05.23 11:16:01 LOG6[12517524:1286]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:01 LOG7[12517524:1286]: Remote FD=20 initialized
2012.05.23 11:16:01 LOG7[12517524:1286]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:01 LOG7[12517524:3856]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:3856]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:4113]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:4370]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:4113]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:4370]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:4627]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:4627]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG6[12517524:1800]: SSL accepted: new session negotiated
2012.05.23 11:16:01 LOG6[12517524:1800]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:01 LOG7[12517524:1800]: Remote FD=25 initialized
2012.05.23 11:16:01 LOG7[12517524:1800]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:01 LOG7[12517524:4884]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:4884]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:5141]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:5141]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:5398]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:5398]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:5655]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:5655]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:5912]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:5912]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:6169]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:6169]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG6[12517524:3085]: SSL accepted: new session negotiated
2012.05.23 11:16:01 LOG6[12517524:3085]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:01 LOG7[12517524:3085]: Remote FD=32 initialized
2012.05.23 11:16:01 LOG7[12517524:3085]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:01 LOG7[12517524:6426]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:6426]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG7[12517524:6683]: Service prdweb_in started
2012.05.23 11:16:01 LOG7[12517524:6683]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:01 LOG6[12517524:258]: SSL accepted: new session negotiated
2012.05.23 11:16:01 LOG6[12517524:258]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:01 LOG7[12517524:258]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG6[12517524:4370]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:4370]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:4370]: Remote FD=36 initialized
2012.05.23 11:16:02 LOG7[12517524:4370]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG6[12517524:5398]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:5398]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:5398]: Remote FD=37 initialized
2012.05.23 11:16:02 LOG7[12517524:5398]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG7[12517524:6940]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:6940]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG6[12517524:5655]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:5655]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:5655]: Remote FD=39 initialized
2012.05.23 11:16:02 LOG7[12517524:5655]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG7[12517524:7197]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:7197]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG7[12517524:5655]: Sending socket write shutdown
2012.05.23 11:16:02 LOG7[12517524:7454]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:7454]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG7[12517524:7711]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:7711]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG6[12517524:515]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:515]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:515]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG6[12517524:1029]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:1029]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:1029]: Remote FD=44 initialized
2012.05.23 11:16:02 LOG7[12517524:1029]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG7[12517524:7968]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:7968]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG7[12517524:8225]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:8225]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG6[12517524:772]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:772]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:772]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG6[12517524:8225]: SSL accepted: previous session reused
2012.05.23 11:16:02 LOG6[12517524:7968]: SSL accepted: previous session reused
2012.05.23 11:16:02 LOG7[12517524:8225]: Remote FD=48 initialized
2012.05.23 11:16:02 LOG7[12517524:8225]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG7[12517524:7968]: Remote FD=49 initialized
2012.05.23 11:16:02 LOG7[12517524:7968]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG7[12517524:8482]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:8482]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG6[12517524:3599]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:3599]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:3599]: Remote FD=51 initialized
2012.05.23 11:16:02 LOG7[12517524:3599]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG6[12517524:3856]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:3856]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:3856]: Remote FD=52 initialized
2012.05.23 11:16:02 LOG7[12517524:3856]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG7[12517524:3856]: Sending socket write shutdown
2012.05.23 11:16:02 LOG7[12517524:4370]: Sending socket write shutdown
2012.05.23 11:16:02 LOG7[12517524:8739]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:8739]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG6[12517524:4113]: SSL accepted: new session negotiated
2012.05.23 11:16:02 LOG6[12517524:4113]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:02 LOG7[12517524:4113]: Remote FD=54 initialized
2012.05.23 11:16:02 LOG7[12517524:4113]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG6[12517524:8482]: SSL accepted: previous session reused
2012.05.23 11:16:02 LOG7[12517524:4113]: Sending socket write shutdown
2012.05.23 11:16:02 LOG7[12517524:8482]: Remote FD=55 initialized
2012.05.23 11:16:02 LOG7[12517524:8482]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:02 LOG7[12517524:8996]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:8996]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG7[12517524:9253]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:9253]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG7[12517524:9510]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:9510]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG7[12517524:9767]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:9767]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG7[12517524:10024]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:10024]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:02 LOG7[12517524:10281]: Service prdweb_in started
2012.05.23 11:16:02 LOG7[12517524:10281]: Option TCP_NODELAY set on local socket
2012.05.23 11:16:03 LOG6[12517524:5912]: SSL accepted: new session negotiated
2012.05.23 11:16:03 LOG6[12517524:5912]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
2012.05.23 11:16:03 LOG7[12517524:5912]: Remote FD=62 initialized
2012.05.23 11:16:03 LOG7[12517524:5912]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG6[12517524:8739]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG7[12517524:8739]: Remote FD=63 initialized
2012.05.23 11:16:03 LOG7[12517524:8739]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG6[12517524:8996]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG7[12517524:8996]: Remote FD=64 initialized
2012.05.23 11:16:03 LOG7[12517524:8996]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG6[12517524:9510]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG7[12517524:9510]: Remote FD=65 initialized
2012.05.23 11:16:03 LOG7[12517524:9510]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG6[12517524:9253]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG7[12517524:9253]: Remote FD=66 initialized
2012.05.23 11:16:03 LOG7[12517524:9253]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG6[12517524:9767]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG7[12517524:9767]: Remote FD=67 initialized
2012.05.23 11:16:03 LOG7[12517524:9767]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG6[12517524:10024]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG6[12517524:7197]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG7[12517524:7197]: Remote FD=69 initialized
2012.05.23 11:16:03 LOG7[12517524:7197]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG6[12517524:6683]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG7[12517524:6683]: Remote FD=70 initialized
2012.05.23 11:16:03 LOG7[12517524:6683]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG6[12517524:10281]: SSL accepted: previous session reused
2012.05.23 11:16:03 LOG7[12517524:10281]: Remote FD=71 initialized
2012.05.23 11:16:03 LOG7[12517524:10281]: Option TCP_NODELAY set on remote socket
2012.05.23 11:16:03 LOG7[12517524:10024]: Remote FD=68 initialized
2012.05.23 11:16:03 LOG7[12517524:1]: Dispatching signals from the signal pipe
2012.05.23 11:16:03 LOG7[12517524:10024]: Option TCP_NODELAY set on remote socket