Hi,
I'm trying to setup yet another service of this kind. I've seen this has been largely discussed several times on this list (but without valid solutions), and I'm writting this email as some kind of last resort after hours of testing and debugging.
Pretty simple configuration:
# cat /etc/stunnel/stunnel.conf cert = /etc/ssl/certs/stunnel4/my-cert.crt key = /etc/ssl/certs/stunnel4/my-cert.key
sslVersion = SSLv3 foreground = yes pid = /tmp/stunnel4.pid
socket = l:TCP_NODELAY=1 socket = r:TCP_NODELAY=1
debug = 7 output = /var/log/stunnel.log
[service] accept = 195.78.X.X:6697 connect = 195.78.X.X:1357 transparent = source
Accept and connect IP's are the same.
# /usr/local/bin/stunnel -version stunnel 4.56 on i686-pc-linux-gnu platform Compiled/running with OpenSSL 0.9.8o 01 Jun 2010 Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP
Global options: debug = daemon.notice pid = /usr/local/var/run/stunnel/stunnel.pid RNDbytes = 64 RNDfile = /dev/urandom RNDoverwrite = yes
Service-level options: ciphers = ALL:!SSLv2:!aNULL:!EXP:!LOW:-MEDIUM:RC4:+HIGH curve = prime256v1 sessionCacheSize = 1000 sessionCacheTimeout = 300 seconds sslVersion = TLSv1 for client, all for server stack = 65536 bytes TIMEOUTbusy = 300 seconds TIMEOUTclose = 60 seconds TIMEOUTconnect = 10 seconds TIMEOUTidle = 43200 seconds verify = none
# /usr/local/bin/stunnel -sockets stunnel 4.56 on i686-pc-linux-gnu platform Compiled/running with OpenSSL 0.9.8o 01 Jun 2010 Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP
Socket option defaults: Option Name | Accept | Local | Remote |OS default ----------------+----------+----------+----------+---------- SO_DEBUG | -- | -- | -- | no SO_DONTROUTE | -- | -- | -- | no SO_KEEPALIVE | -- | -- | -- | no SO_LINGER | -- | -- | -- | 0:0 SO_OOBINLINE | -- | -- | -- | no SO_RCVBUF | -- | -- | -- | 87380 SO_SNDBUF | -- | -- | -- | 65536 SO_RCVLOWAT | -- | -- | -- | 1 SO_SNDLOWAT | -- | -- | -- | 1 SO_RCVTIMEO | -- | -- | -- | 0:0 SO_SNDTIMEO | -- | -- | -- | 0:0 SO_REUSEADDR | yes| -- | -- | no SO_BINDTODEVICE | -- | -- | -- |write-only TCP_KEEPCNT | -- | -- | -- | 9 TCP_KEEPIDLE | -- | -- | -- | 7200 TCP_KEEPINTVL | -- | -- | -- | 75 IP_TOS | -- | -- | -- | 0 IP_TTL | -- | -- | -- | 64 TCP_NODELAY | -- | yes| yes| no IP_FREEBIND | -- | -- | -- | no
And the timeout, the same I've seen suffering to other people:
2013.08.03 19:52:12 LOG7[18496:3074533056]: Service [service] accepted (FD=3) from MY_HOME_ADDRESS:34836 2013.08.03 19:52:12 LOG7[18496:3074530160]: Service [service] started 2013.08.03 19:52:12 LOG5[18496:3074530160]: Service [service] accepted connection from MY_HOME_ADDRESS:34836 (blah blah ssl stuff) 2013.08.03 19:52:12 LOG6[18496:3074530160]: Negotiated TLSv1/SSLv3 ciphersuite: DHE-RSA-AES256-SHA (256-bit encryption) 2013.08.03 19:52:12 LOG6[18496:3074530160]: Compression: null, expansion: null 2013.08.03 19:52:12 LOG6[18496:3074530160]: IP_TRANSPARENT socket option set 2013.08.03 19:52:12 LOG6[18496:3074530160]: local_bind succeeded on the original port 2013.08.03 19:52:12 LOG6[18496:3074530160]: connect_blocking: connecting 195.78.X.X:1357 2013.08.03 19:52:12 LOG7[18496:3074530160]: connect_blocking: s_poll_wait 195.78.X.X:1357: waiting 60 seconds 2013.08.03 19:52:21 LOG3[18496:3074530160]: connect_blocking: connect 195.78.X.X:1357: Connection timed out (110) 2013.08.03 19:52:21 LOG5[18496:3074530160]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
iptables stuff and that, is in place:
# cat /proc/sys/net/ipv4/conf/all/rp_filter 0 # cat /proc/sys/net/ipv4/ip_forward 1
And did the iptables part:
iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100
Debugging the whole thing, it can be seen that stunnel tries to connect:
[pid 16823] connect(9, {sa_family=AF_INET, sin_port=htons(1357), sin_addr=inet_addr("195.78.X.X")}, 16) = -1 EINPROGRESS (Operation now in progress)
BUT the service running in 1357 does this:
# tcpdump -i eth1 -n port 1357 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 19:52:52.586773 IP 195.78.X.X.1357 > MY_HOME_ADDRESS.34853: Flags [S.], seq 2655966098, ack 546202865, win 5840, options [mss 1460,nop,nop,sackOK], length 0
And, according to that, it looks obvious to me that this setup will never work, since that ACK packet goes to my home box, and not to the local connection.
So either I'm doing something wrong (I hope I am!!) or this thing definitely doesn't work...
Any help/ideas/whatever, please?
Thanks,
Rubén.
Forgot to say: 2.6.32-5 & Debian 6.0
Rubén.
El 03.08.2013 20:24, Ruben Cardenal escribió:
Hi,
I'm trying to setup yet another service of this kind. I've seen this has been largely discussed several times on this list (but without valid solutions), and I'm writting this email as some kind of last resort after hours of testing and debugging.
Pretty simple configuration:
# cat /etc/stunnel/stunnel.conf cert = /etc/ssl/certs/stunnel4/my-cert.crt key = /etc/ssl/certs/stunnel4/my-cert.key
sslVersion = SSLv3 foreground = yes pid = /tmp/stunnel4.pid
socket = l:TCP_NODELAY=1 socket = r:TCP_NODELAY=1
debug = 7 output = /var/log/stunnel.log
[service] accept = 195.78.X.X:6697 connect = 195.78.X.X:1357 transparent = source
Accept and connect IP's are the same.
# /usr/local/bin/stunnel -version stunnel 4.56 on i686-pc-linux-gnu platform Compiled/running with OpenSSL 0.9.8o 01 Jun 2010 Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP
Global options: debug = daemon.notice pid = /usr/local/var/run/stunnel/stunnel.pid RNDbytes = 64 RNDfile = /dev/urandom RNDoverwrite = yes
Service-level options: ciphers = ALL:!SSLv2:!aNULL:!EXP:!LOW:-MEDIUM:RC4:+HIGH curve = prime256v1 sessionCacheSize = 1000 sessionCacheTimeout = 300 seconds sslVersion = TLSv1 for client, all for server stack = 65536 bytes TIMEOUTbusy = 300 seconds TIMEOUTclose = 60 seconds TIMEOUTconnect = 10 seconds TIMEOUTidle = 43200 seconds verify = none
# /usr/local/bin/stunnel -sockets stunnel 4.56 on i686-pc-linux-gnu platform Compiled/running with OpenSSL 0.9.8o 01 Jun 2010 Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP
Socket option defaults: Option Name | Accept | Local | Remote |OS default ----------------+----------+----------+----------+---------- SO_DEBUG | -- | -- | -- | no SO_DONTROUTE | -- | -- | -- | no SO_KEEPALIVE | -- | -- | -- | no SO_LINGER | -- | -- | -- | 0:0 SO_OOBINLINE | -- | -- | -- | no SO_RCVBUF | -- | -- | -- | 87380 SO_SNDBUF | -- | -- | -- | 65536 SO_RCVLOWAT | -- | -- | -- | 1 SO_SNDLOWAT | -- | -- | -- | 1 SO_RCVTIMEO | -- | -- | -- | 0:0 SO_SNDTIMEO | -- | -- | -- | 0:0 SO_REUSEADDR | yes| -- | -- | no SO_BINDTODEVICE | -- | -- | -- |write-only TCP_KEEPCNT | -- | -- | -- | 9 TCP_KEEPIDLE | -- | -- | -- | 7200 TCP_KEEPINTVL | -- | -- | -- | 75 IP_TOS | -- | -- | -- | 0 IP_TTL | -- | -- | -- | 64 TCP_NODELAY | -- | yes| yes| no IP_FREEBIND | -- | -- | -- | no
And the timeout, the same I've seen suffering to other people:
2013.08.03 19:52:12 LOG7[18496:3074533056]: Service [service] accepted (FD=3) from MY_HOME_ADDRESS:34836 2013.08.03 19:52:12 LOG7[18496:3074530160]: Service [service] started 2013.08.03 19:52:12 LOG5[18496:3074530160]: Service [service] accepted connection from MY_HOME_ADDRESS:34836 (blah blah ssl stuff) 2013.08.03 19:52:12 LOG6[18496:3074530160]: Negotiated TLSv1/SSLv3 ciphersuite: DHE-RSA-AES256-SHA (256-bit encryption) 2013.08.03 19:52:12 LOG6[18496:3074530160]: Compression: null, expansion: null 2013.08.03 19:52:12 LOG6[18496:3074530160]: IP_TRANSPARENT socket option set 2013.08.03 19:52:12 LOG6[18496:3074530160]: local_bind succeeded on the original port 2013.08.03 19:52:12 LOG6[18496:3074530160]: connect_blocking: connecting 195.78.X.X:1357 2013.08.03 19:52:12 LOG7[18496:3074530160]: connect_blocking: s_poll_wait 195.78.X.X:1357: waiting 60 seconds 2013.08.03 19:52:21 LOG3[18496:3074530160]: connect_blocking: connect 195.78.X.X:1357: Connection timed out (110) 2013.08.03 19:52:21 LOG5[18496:3074530160]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
iptables stuff and that, is in place:
# cat /proc/sys/net/ipv4/conf/all/rp_filter 0 # cat /proc/sys/net/ipv4/ip_forward 1
And did the iptables part:
iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100
Debugging the whole thing, it can be seen that stunnel tries to connect:
[pid 16823] connect(9, {sa_family=AF_INET, sin_port=htons(1357), sin_addr=inet_addr("195.78.X.X")}, 16) = -1 EINPROGRESS (Operation now in progress)
BUT the service running in 1357 does this:
# tcpdump -i eth1 -n port 1357 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 19:52:52.586773 IP 195.78.X.X.1357 > MY_HOME_ADDRESS.34853: Flags [S.], seq 2655966098, ack 546202865, win 5840, options [mss 1460,nop,nop,sackOK], length 0
And, according to that, it looks obvious to me that this setup will never work, since that ACK packet goes to my home box, and not to the local connection.
So either I'm doing something wrong (I hope I am!!) or this thing definitely doesn't work...
Any help/ideas/whatever, please?
Thanks,
Rubén.
stunnel-users mailing list stunnel-users@stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users [1]
Links: ------ [1] https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users
On 2013-08-03 20:24, Ruben Cardenal wrote:
And did the iptables part:
iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100
Debugging the whole thing, it can be seen that stunnel tries to connect:
[pid 16823] connect(9, {sa_family=AF_INET, sin_port=htons(1357), sin_addr=inet_addr("195.78.X.X")}, 16) = -1 EINPROGRESS (Operation now in progress)
BUT the service running in 1357 does this:
# tcpdump -i eth1 -n port 1357 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 19:52:52.586773 IP 195.78.X.X.1357 > MY_HOME_ADDRESS.34853: Flags [S.], seq 2655966098, ack 546202865, win 5840, options [mss 1460,nop,nop,sackOK], length 0
It looks like you configured your server and stunnel on the same host. As the result returning packets won't ever hit the PREROUTING chain of the mangle table, thus stunnel won't receive them. http://www.iptables.info/en/structure-of-iptables.html
I'm sure the documentation I wrote could be better...
Mike
El 03/08/13 21:39, Michal Trojnara escribió:
On 2013-08-03 20:24, Ruben Cardenal wrote:
And did the iptables part:
iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100
Debugging the whole thing, it can be seen that stunnel tries to connect:
[pid 16823] connect(9, {sa_family=AF_INET, sin_port=htons(1357), sin_addr=inet_addr("195.78.X.X")}, 16) = -1 EINPROGRESS (Operation now in progress)
BUT the service running in 1357 does this:
# tcpdump -i eth1 -n port 1357 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 19:52:52.586773 IP 195.78.X.X.1357 > MY_HOME_ADDRESS.34853: Flags [S.], seq 2655966098, ack 546202865, win 5840, options [mss 1460,nop,nop,sackOK], length 0
It looks like you configured your server and stunnel on the same host. As the result returning packets won't ever hit the PREROUTING chain of the mangle table, thus stunnel won't receive them. http://www.iptables.info/en/structure-of-iptables.html
I'm sure the documentation I wrote could be better...
Mike
Hi Mike,
I've given a try to what you say. I've moved stunnel to another box, while leaving the service where it was. Removed iptables stuff from the service's box, and leaving there just the backend service. In the new box, compiled the same last stunnel version and used the same configuration, changing only the connect part, applying iptables, etc. And this is what is seen (please note both servers are in perfect ntp sync):
stunnel log, same thing, only changing connect IP:
2013.08.03 23:23:09 LOG6[26792:139762318628608]: connect_blocking: connecting SERVICE:1357 2013.08.03 23:23:09 LOG7[26792:139762318628608]: connect_blocking: s_poll_wait SERVICE:1357: waiting 10 seconds 2013.08.03 23:23:18 LOG3[26792:139762318628608]: connect_blocking: connect SERVICE:1357: Connection timed out (110) 2013.08.03 23:23:18 LOG5[26792:139762318628608]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
stunnel process:
[pid 569] 23:23:09 bind(9, {sa_family=AF_INET, sin_port=htons(50573), sin_addr=inet_addr("HOME")}, 16) = 0 [pid 569] 23:23:09 sendto(6, "<30>Aug 3 23:23:49 stunnel: LOG6[534:140271099127552]: local_bind succeeded on the original port", 97, MSG_NOSIGNAL, NULL, 0) = 97
[pid 569] 23:23:09 write(8, "2013.08.03 23:23:09 LOG6[534:140271099127552]: connect_blocking: connecting SERVICE:1357\n", 96) = 96 [pid 569] 23:23:09 connect(9, {sa_family=AF_INET, sin_port=htons(1357), sin_addr=inet_addr("SERVICE")}, 16) = -1 EINPROGRESS (Operation now in progress)
tcpdump in service's box:
23:23:09.015940 IP MYHOME.50573 > SERVICE.1357: Flags [S], seq 3062456285, win 5840, options [mss 1460,nop,nop,sackOK], length 0 23:23:09.015969 IP SERVICE.1357 > MYHOME.50573: Flags [S.], seq 3120433945, ack 3062456286, win 5840, options [mss 1460,nop,nop,sackOK], length 0 23:23:12.013679 IP MYHOME.50573 > SERVICE.1357: Flags [S], seq 3062456285, win 5840, options [mss 1460,nop,nop,sackOK], length 0 23:23:12.013701 IP SERVICE.1357 > MYHOME.50573: Flags [S.], seq 3120433945, ack 3062456286, win 5840, options [mss 1460,nop,nop,sackOK], length 0 23:23:12.813537 IP SERVICE.1357 > MYHOME.50573: Flags [S.], seq 3120433945, ack 3062456286, win 5840, options [mss 1460,nop,nop,sackOK], length 0 23:23:18.817538 IP SERVICE.1357 > MYHOME.50573: Flags [S.], seq 3120433945, ack 3062456286, win 5840, options [mss 1460,nop,nop,sackOK], length 0
tcpdump in stunnel's box:
23:23:08.343486 IP MYHOME.50573 > STUNNEL.6697: Flags [S], seq 1065154599, win 14600, options [mss 1452,sackOK,TS val 121378180 ecr 0,nop,wscale 7], length 0 23:23:08.343547 IP STUNNEL.6697 > MYHOME.50573: Flags [S.], seq 40478691, ack 1065154600, win 5840, options [mss 1460,nop,nop,sackOK], length 0 23:23:08.363058 IP MYHOME.50573 > STUNNEL.6697: Flags [.], ack 1, win 14600, length 0 23:23:08.678217 IP MYHOME.50573 > STUNNEL.6697: Flags [P.], seq 1:151, ack 1, win 14600, length 150 23:23:08.678271 IP STUNNEL.6697 > MYHOME.50573: Flags [.], ack 151, win 6432, length 0 23:23:08.682553 IP STUNNEL.6697 > MYHOME.50573: Flags [.], seq 1:1453, ack 151, win 6432, length 1452 23:23:08.682566 IP STUNNEL.6697 > MYHOME.50573: Flags [P.], seq 1453:1809, ack 151, win 6432, length 356 23:23:08.697844 IP MYHOME.50573 > STUNNEL.6697: Flags [.], ack 1809, win 17424, length 0 23:23:08.993183 IP MYHOME.50573 > STUNNEL.6697: Flags [P.], seq 151:493, ack 1809, win 17424, length 342 23:23:09.006506 IP STUNNEL.6697 > MYHOME.50573: Flags [P.], seq 1809:1884, ack 493, win 7504, length 75 *23:23:09.007004 IP MYHOME.50573 > SERVICE.1357: Flags [S], seq 3062456285, win 5840, options [mss 1460,nop,nop,sackOK], length 0* 23:23:09.022762 IP MYHOME.50573 > STUNNEL.6697: Flags [.], ack 1884, win 17424, length 0 23:23:09.278792 IP MYHOME.50573 > STUNNEL.6697: Flags [P.], seq 493:567, ack 1884, win 17424, length 74 23:23:09.316716 IP STUNNEL.6697 > MYHOME.50573: Flags [.], ack 567, win 7504, length 0 23:23:09.328032 IP MYHOME.50573 > STUNNEL.6697: Flags [P.], seq 567:689, ack 1884, win 17424, length 122 23:23:09.328073 IP STUNNEL.6697 > MYHOME.50573: Flags [.], ack 689, win 7504, length 0 *23:23:12.004564 IP MYHOME.50573 > SERVICE.1357: Flags [S], seq 3062456285, win 5840, options [mss 1460,nop,nop,sackOK], length 0* 23:23:18.006575 IP STUNNEL.6697 > MYHOME.50573: Flags [R.], seq 1884, ack 689, win 7504, length 0
In bold the 2 SYN packets present in stunnel's tcpdump towards the service's box. The rest of the traffic. between MYHOME and STUNNEL, is just part of the SSL negotiation.
So: service's box receives a SYN packet from my home IP address (originated from stunnel's box), and answers with a proper ACK packet. That's ok. But as that ACK reply has as destination an external IP, goes to the box's default gateway (and not to the box where stunnel is running) and gets lost.
So unless I'm missing something, I think that kind both remote service and stunnel should be in the same server for this to work (as a minimun). Which leaves me in the same place I started with :(
Rubén.
On 2013-08-04 00:02, Rubén Cardenal wrote:
So: service's box receives a SYN packet from my home IP address (originated from stunnel's box), and answers with a proper ACK packet. That's ok. But as that ACK reply has as destination an external IP, goes to the box's default gateway (and not to the box where stunnel is running) and gets lost.
The very purpose of of "transparent = source" is to make your server think it's connected directly by the clients. The returning packets obviously need to be routed back through the stunnel box to achieve this purpose. Otherwise the mangle PREROUTING tricks wouldn't make sense, would they?
Using this feature is quite easy at the user-space level (this is what stunnel handles), but quite tricky at the kernel level (netfilter and routing configuration). A good HOWTO would be very useful.
Mike
El 04/08/13 09:05, Michal Trojnara escribió:
The very purpose of of "transparent = source" is to make your server think it's connected directly by the clients. The returning packets obviously need to be routed back through the stunnel box to achieve this purpose. Otherwise the mangle PREROUTING tricks wouldn't make sense, would they?
Yep. That's why the "need to be on separate boxes" part seemed strange to me. As the incoming connections are from any, the only option would be the stunnel box being the gw for the service box. So I added a static route in the service box for my home IP via the stunnel box, and in that case, of course it works. But it's not a practical solution.
I've found the way of making it work without needing to alter that default gateway. Apart from the configurations to be done in the stunnel box, in the service box:
iptables -t mangle -I OUTPUT -p tcp --sport SERVICE_PORT -s SERVICE_IP -j MARK --set-mark 1 (this can be tweaked in every single case to granularly catch only desired packets) ip rule add fwmark 1 lookup 100 ip route add 0.0.0.0/0 via STUNNEL_IP table 100 (other people might need to add "dev XXX")
Note: /proc/sys/net/ipv4/ip_forward doesn't need to be touched in any of the boxes.
Using this feature is quite easy at the user-space level (this is what stunnel handles), but quite tricky at the kernel level (netfilter and routing configuration). A good HOWTO would be very useful.
Indeed! Thanks for the hints ;)
Regards,
Rubén.
-
Michal Trojnara -
Ruben Cardenal