Using putty to debug ssh through a firewall - SSH

This is a discussion on Using putty to debug ssh through a firewall - SSH ; I currently use ssh to access things outside of a firewall. The
firewall allows outbound traffic on port 23, so I run an ssh server on
that port (sshd for cygwin). I then connect with an ssh client on
that ...

Using putty to debug ssh through a firewall

I currently use ssh to access things outside of a firewall. The
firewall allows outbound traffic on port 23, so I run an ssh server on
that port (sshd for cygwin). I then connect with an ssh client on
that port, and can set up any port forwarding that I desire. It
worked beautifully until just the other day. Now, when I connect, I
the message exchange hangs. Putty doesn't give enough detail on what
the message exchange actually is, so here is some output from the
cygwin ssh implementation using maximum verbosity:

And it just hangs there until the connection times out. Any ideas on
what might be causing a hang at that stage?

Re: Using putty to debug ssh through a firewall

X-No-Archive: Yes

So let me get this straight your pretty much running ssh and sshd on
the same computer?
OR (Let me try and understand.)
Your behind a (company's) firewall and your running an sshd on port 23
on your workstation?
Please give a little more detail.

To access things outside of a firewall through your workstation you
need an sshd (SSH Server)
on the outside to connect to (home computer?).
Then you can use an ssh client (like PuTTY) running on your
workstation to connect to that server
and route something like a web browser through it so you bypass your
firewall's filter or something.

Also, please post your sshd_config file (/etc/sshd_config) so people
can recommend some changes if need be
but either way I can't thing of a solution (on your limited
description, no offense) other than recommending an upgrade
on your client? Your OpenSSH version is the latest one though.

Home computer - sshd server on cygwin, no firewall
Remote computer - putty ssh client behind a firewall
> To access things outside of a firewall through your workstation you
> need an sshd (SSH Server)
> on the outside to connect to (home computer?).
> Then you can use an ssh client (like PuTTY) running on your
> workstation to connect to that server
> and route something like a web browser through it so you bypass your
> firewall's filter or something.

Right, I've been doing this for a long time. It spontaneously stopped
working. I see the handshaking start, but then it just dies.
> Also, please post your sshd_config file (/etc/sshd_config) so people
> can recommend some changes if need be
> but either way I can't thing of a solution (on your limited
> description, no offense) other than recommending an upgrade
> on your client? Your OpenSSH version is the latest one though.

I will post it at the end of this message, although I will note that
1) it hasn't changed, and 2) I use 2 different servers to connect to
(in case my computer isn't available). Mine is sshd on cygwin, the
other is sshd on gentoo linux. That's my backup for when my computer
at home overheats and dies. Both are different versions of sshd that
are configured for port 23 that have worked flawlessly, and both
stopped working at the exact same time. Obviously, something changed
on the firewall.
> Here are some links about bypassing a firewall through SSH:http://polishlinux.org/apps/ssh-tunn...om/surfatwork/
> Just google for other such tutorials.

Any tutorial will not be at the level of support I need. If you look
at the debug messages I provided, you can see where the processing
stops. It is during the handshaking. At the beginning of a session,
the ssh server sends its version id string, and the client responds
with its own version id string. Shortly thereafter, all messages
timeout. It's as if the firewall is deciphering the message
handshaking and killing one of the messages.

I also tried stepping down to ssh version 1, with similar results.
> Don't mind me asking or don't even reply to this question but: Are you
> up to no good? :-(

Nope. Everything is on the table. Sometimes you really just need
access to a home computer.

Re: Using putty to debug ssh through a firewall

NightStrike writes:
>I currently use ssh to access things outside of a firewall.
[...]
>debug1: SSH2_MSG_KEXINIT sent
>
>And it just hangs there until the connection times out. Any ideas on
>what might be causing a hang at that stage?

KEXINIT messages can be quite large. Guess: could it be?

Re: Using putty to debug ssh through a firewall

X-No-Archive: Yes

"That's my backup for when my computer at home overheats and dies."
I recommed that you use Wake-On-Lan/Wan to start your computer when
need be (sshd runs as a system service so it will start before windows
login).http://en.wikipedia.org/wiki/Wake-on-LANhttp://www.depicus.com/wake-on-lan/
As for your problem with connecting, the system admin could easily (if
he/she is smart enough) block SSH connections from occuring. It
wouldn't be too hard. And I kind of disagree with J. Nevins, if it
worked before why is it not working now?

Man, I'm dumbfounded: Did you try reinstalling Cygwin's OpenSSH?

Re: Using putty to debug ssh through a firewall

On Aug 18, 11:58 am, purpmint...@gmail.com wrote:
> As for your problem with connecting, the system admin could easily (if
> he/she is smart enough) block SSH connections from occuring. It
> wouldn't be too hard. And I kind of disagree with J. Nevins, if it
> worked before why is it not working now?
>
> Man, I'm dumbfounded: Did you try reinstalling Cygwin's OpenSSH?

The firewall would have to watch for packets containing, for instance,
the version strings of an SSH connection. Remember, I'm going over
port 23, not port 22. Port 23 is still open, so it's not a simple
port closure.

Reinstalling cygwin's openssh would not make sense. Again, it doesn't
work connecting to not one, but two totally separate sshd
installations that worked perfectly and both stopped at the same time,
one on cygwin and another on gentoo.

What I need is a better way to get lower level debugging to see
exactly where it's dying in the handshaking. 'ssh -vvv' obviously
isn't enough.

Oh, and for the record, I have tried from numerous workstations behind
the firewall that all worked fine until some concrete time at which
everything stopped working.

I will also try a different port.. 21 or 80 or something else that's
open.

Re: Using putty to debug ssh through a firewall

On Aug 18, 9:46 am, Jacob Nevins
wrote:
> NightStrike writes:
> >I currently use ssh to access things outside of a firewall.
> [...]
> >debug1: SSH2_MSG_KEXINIT sent
>
> >And it just hangs there until the connection times out. Any ideas on
> >what might be causing a hang at that stage?
>
> KEXINIT messages can be quite large. Guess: could it be
> ?

It's possible, but I don't think so. I will post tomorrow similar
information but where I force SSH v1 protocol. There will be a
similar hanging point. What would be a good tool to use to see if a
KEXINIT message is in fact being transmitted, however slowly?

Re: Using putty to debug ssh through a firewall

>>>>> "NS" == NightStrike writes:

NS> On Aug 18, 9:46 am, Jacob Nevins
NS> wrote:
>> NightStrike writes: >I currently use ssh to
>> access things outside of a firewall. [...] >debug1:
>> SSH2_MSG_KEXINIT sent
>>
>> >And it just hangs there until the connection times out. Any ideas
>> on >what might be causing a hang at that stage?
>>
>> KEXINIT messages can be quite large. Guess: could it be
>> ?

NS> It's possible, but I don't think so. I will post tomorrow similar
NS> information but where I force SSH v1 protocol. There will be a
NS> similar hanging point. What would be a good tool to use to see if
NS> a KEXINIT message is in fact being transmitted, however slowly?

Wireshark will show you this, as that portion of the protocol is not yet
encrypted.

Re: Using putty to debug ssh through a firewall

On Aug 20, 1:32 am, NightStrike wrote:
> On Aug 18, 9:46 am, Jacob Nevins
> wrote:
>
> > NightStrike writes:
> > >I currently use ssh to access things outside of a firewall.
> > [...]
> > >debug1: SSH2_MSG_KEXINIT sent
>
> > >And it just hangs there until the connection times out. Any ideas on
> > >what might be causing a hang at that stage?
>
> > KEXINIT messages can be quite large. Guess: could it be
> > ?
>
> It's possible, but I don't think so. I will post tomorrow similar
> information but where I force SSH v1 protocol. There will be a
> similar hanging point. What would be a good tool to use to see if a
> KEXINIT message is in fact being transmitted, however slowly?

Re: Using putty to debug ssh through a firewall

On Aug 20, 6:30 am, "Richard E. Silverman" wrote:
> >>>>> "NS" == NightStrike writes:
>
> NS> On Aug 18, 9:46 am, Jacob Nevins
> NS> wrote:
> >> NightStrike writes: >I currently use ssh to
> >> access things outside of a firewall. [...] >debug1:
> >> SSH2_MSG_KEXINIT sent
> >>
> >> >And it just hangs there until the connection times out. Any ideas
> >> on >what might be causing a hang at that stage?
> >>
> >> KEXINIT messages can be quite large. Guess: could it be
> >> ?
>
> NS> It's possible, but I don't think so. I will post tomorrow similar
> NS> information but where I force SSH v1 protocol. There will be a
> NS> similar hanging point. What would be a good tool to use to see if
> NS> a KEXINIT message is in fact being transmitted, however slowly?
>
> Wireshark will show you this, as that portion of the protocol is not yet
> encrypted.
>
> --
> Richard Silverman
> r...@qoxp.net

Wireshark is a very handy tool. It shows me trying to send the
KEXINIT message over and over again as a TCP Resend.

Re: Using putty to debug ssh through a firewall

On Aug 23, 5:18 pm, NightStrike wrote:
> On Aug 20, 6:30 am, "Richard E. Silverman" wrote:
>
>
>
>
>
> > >>>>> "NS" == NightStrike writes:
>
> > NS> On Aug 18, 9:46 am, Jacob Nevins
> > NS> wrote:
> > >> NightStrike writes: >I currently use ssh to
> > >> access things outside of a firewall. [...] >debug1:
> > >> SSH2_MSG_KEXINIT sent
>
> > >> >And it just hangs there until the connection times out. Any ideas
> > >> on >what might be causing a hang at that stage?
>
> > >> KEXINIT messages can be quite large. Guess: could it be
> > >> ?
>
> > NS> It's possible, but I don't think so. I will post tomorrow similar
> > NS> information but where I force SSH v1 protocol. There will be a
> > NS> similar hanging point. What would be a good tool to use to see if
> > NS> a KEXINIT message is in fact being transmitted, however slowly?
>
> > Wireshark will show you this, as that portion of the protocol is not yet
> > encrypted.
>
> > --
> > Richard Silverman
> > r...@qoxp.net
>
> Wireshark is a very handy tool. It shows me trying to send the
> KEXINIT message over and over again as a TCP Resend.- Hide quoted text -
>
> - Show quoted text -

Are there any futher suggestions?

Re: Using putty to debug ssh through a firewall

In article <1188489286.280801.72610@g4g2000hsf.googlegroups.co m>
NightStrike writes:
>On Aug 23, 5:18 pm, NightStrike wrote:
>> On Aug 20, 6:30 am, "Richard E. Silverman" wrote:
>>
>> > >>>>> "NS" == NightStrike writes:
>>
>> > NS> On Aug 18, 9:46 am, Jacob Nevins
>> > NS> wrote:
>> > >> NightStrike writes: >I currently use ssh to
>> > >> access things outside of a firewall. [...] >debug1:
>> > >> SSH2_MSG_KEXINIT sent
>>
>> > >> >And it just hangs there until the connection times out. Any ideas
>> > >> on >what might be causing a hang at that stage?
>>
>> > >> KEXINIT messages can be quite large. Guess: could it be
>> > >> ?
>>
>> > NS> It's possible, but I don't think so. I will post tomorrow similar
>> > NS> information but where I force SSH v1 protocol. There will be a
>> > NS> similar hanging point. What would be a good tool to use to see if
>> > NS> a KEXINIT message is in fact being transmitted, however slowly?
>>
>> > Wireshark will show you this, as that portion of the protocol is not yet
>> > encrypted.
>>
>> Wireshark is a very handy tool. It shows me trying to send the
>> KEXINIT message over and over again as a TCP Resend.- Hide quoted text -
>>
>
>Are there any futher suggestions?

Why? Your observation seems like a perfect match for the problem
described at the page above. Did you do any further investigation along
those lines, e.g. trying the workaround described at that page?

Re: Using putty to debug ssh through a firewall

On Aug 30, 5:33 pm, p...@hedeland.org (Per Hedeland) wrote:
> In article <1188489286.280801.72...@g4g2000hsf.googlegroups.co m>
>
>
>
> NightStrike writes:
> >On Aug 23, 5:18 pm, NightStrike wrote:
> >> On Aug 20, 6:30 am, "Richard E. Silverman" wrote:
>
> >> > >>>>> "NS" == NightStrike writes:
>
> >> > NS> On Aug 18, 9:46 am, Jacob Nevins
> >> > NS> wrote:
> >> > >> NightStrike writes: >I currently use ssh to
> >> > >> access things outside of a firewall. [...] >debug1:
> >> > >> SSH2_MSG_KEXINIT sent
>
> >> > >> >And it just hangs there until the connection times out. Any ideas
> >> > >> on >what might be causing a hang at that stage?
>
> >> > >> KEXINIT messages can be quite large. Guess: could it be
> >> > >> ?
>
> >> > NS> It's possible, but I don't think so. I will post tomorrow similar
> >> > NS> information but where I force SSH v1 protocol. There will be a
> >> > NS> similar hanging point. What would be a good tool to use to see if
> >> > NS> a KEXINIT message is in fact being transmitted, however slowly?
>
> >> > Wireshark will show you this, as that portion of the protocol is not yet
> >> > encrypted.
>
> >> Wireshark is a very handy tool. It shows me trying to send the
> >> KEXINIT message over and over again as a TCP Resend.- Hide quoted text -
>
> >Are there any futher suggestions?
>
> Why? Your observation seems like a perfect match for the problem
> described at the page above. Did you do any further investigation along
> those lines, e.g. trying the workaround described at that page?

On the surface, I might agree with you. However, the problem
description on that page talks about significantly large messages.
The KEXINIT message as captured by Wireshark is not that big.
Further, the connection fails in SSH v1 mode, as well, where messages
like KEXINIT are not present.

I did some testing with the ssh server running in debug mode, and what
I found is intriguing. I will post the log output later tonight.

Re: Using putty to debug ssh through a firewall

On Aug 30, 5:33 pm, p...@hedeland.org (Per Hedeland) wrote:
> In article <1188489286.280801.72...@g4g2000hsf.googlegroups.co m>
>
>
>
> NightStrike writes:
> >On Aug 23, 5:18 pm, NightStrike wrote:
> >> On Aug 20, 6:30 am, "Richard E. Silverman" wrote:
>
> >> > >>>>> "NS" == NightStrike writes:
>
> >> > NS> On Aug 18, 9:46 am, Jacob Nevins
> >> > NS> wrote:
> >> > >> NightStrike writes: >I currently use ssh to
> >> > >> access things outside of a firewall. [...] >debug1:
> >> > >> SSH2_MSG_KEXINIT sent
>
> >> > >> >And it just hangs there until the connection times out. Any ideas
> >> > >> on >what might be causing a hang at that stage?
>
> >> > >> KEXINIT messages can be quite large. Guess: could it be
> >> > >> ?
>
> >> > NS> It's possible, but I don't think so. I will post tomorrow similar
> >> > NS> information but where I force SSH v1 protocol. There will be a
> >> > NS> similar hanging point. What would be a good tool to use to see if
> >> > NS> a KEXINIT message is in fact being transmitted, however slowly?
>
> >> > Wireshark will show you this, as that portion of the protocol is not yet
> >> > encrypted.
>
> >> Wireshark is a very handy tool. It shows me trying to send the
> >> KEXINIT message over and over again as a TCP Resend.- Hide quoted text -
>
> >Are there any futher suggestions?
>
> Why? Your observation seems like a perfect match for the problem
> described at the page above. Did you do any further investigation along
> those lines, e.g. trying the workaround described at that page?
>
> --Per Hedeland
> p...@hedeland.org

Just for the record, I did try numerous MTU configurations to no
avail. Also, note that the page mentions that logging in should be
ok, whereas the issue should manifest itself when transmitted large
files or some such thing.

Re: Using putty to debug ssh through a firewall

On 2007-09-05, NightStrike wrote:
> Just for the record, I did try numerous MTU configurations to no
> avail. Also, note that the page mentions that logging in should be
> ok, whereas the issue should manifest itself when transmitted large
> files or some such thing.

One other thing that can look very similar is if you have a firewall
that either doesn't understand or is misconfigured with regard to TCP
window scaling. Such a system can think that the TCP window is much
smaller (in terms of bytes) than it really is and not pass all of the
traffic.

Have a look at the initial TCP packet and see if the WSCALE option
is set and if it is, try disabling it on one of the hosts to see if it
helps.

Re: Using putty to debug ssh through a firewall

On Sep 5, 8:35 pm, Darren Tucker wrote:
> One other thing that can look very similar is if you have a firewall
> that either doesn't understand or is misconfigured with regard to TCP
> window scaling. Such a system can think that the TCP window is much
> smaller (in terms of bytes) than it really is and not pass all of the
> traffic.
>
> Have a look at the initial TCP packet and see if the WSCALE option
> is set and if it is, try disabling it on one of the hosts to see if it
> helps.

I am guessing not.

The first packet (SYN) had the following options:
MSS=1460
NOP
NOP
SACK Permitted

The ACK/SYN Reply had the following options:
MSS=536
NOP
NOP
SACK Permitted

The ACK to that had no options.

Following those first three packets came the version string from the
server:
SSH-2.0-OpenSSH_4.6\n

Packet 5 was my client saying the same thing:
SSH-2.0-OpenSSH_4.6\n

Packets 6 and 7 were handshaking packets. They never go through, and
packet 6 is retried 5 times until it times out.

Re: Using putty to debug ssh through a firewall

On 2007-09-07, NightStrike wrote:
> Packets 6 and 7 were handshaking packets. They never go through, and
> packet 6 is retried 5 times until it times out.

That's interesting. If you do a "netstat -n" and pick out the TCP
connection in question, can you see the value of the "Send-Q" on
both ends? If so, does it remain non-zero?

Actually, you've reminded me of something else: some NAT devices
choke on TCP connections where the IP TOS changes. Try either
rebuilding with "./configure --with-cflags=-DIP_TOS_IS_BROKEN"
or using a proxycommand such as netcat which doesn't set the ToS flags:

Re: Using putty to debug ssh through a firewall

On Sep 8, 2:25 am, Darren Tucker wrote:
> On 2007-09-07, NightStrike wrote:
>
> > Packets 6 and 7 were handshaking packets. They never go through, and
> > packet 6 is retried 5 times until it times out.
>
> That's interesting. If you do a "netstat -n" and pick out the TCP
> connection in question, can you see the value of the "Send-Q" on
> both ends? If so, does it remain non-zero?

I'm afraid you lost me on that one. The connection is in the state of
ESTABLISHED. Where do I find this Send-Q information?
> Actually, you've reminded me of something else: some NAT devices
> choke on TCP connections where the IP TOS changes. Try either
> rebuilding with "./configure --with-cflags=-DIP_TOS_IS_BROKEN"
> or using a proxycommand such as netcat which doesn't set the ToS flags:
>
> ssh -o 'ProxyCommand nc %h %p' youserver
>
> Might be time to add these to the FAQ...

I don't see any difference in either the triple-verbose output of the
ssh client nor the Wireshark readout. I am using the version of
netcat that comes with cygwin. Can you elaborate more on what this
"proxycommand nc" thing is doing?

Re: Using putty to debug ssh through a firewall

On 2007-09-13, NightStrike wrote:
> On Sep 8, 2:25 am, Darren Tucker wrote:
>> On 2007-09-07, NightStrike wrote:
>>
>> > Packets 6 and 7 were handshaking packets. They never go through, and
>> > packet 6 is retried 5 times until it times out.
>>
>> That's interesting. If you do a "netstat -n" and pick out the TCP
>> connection in question, can you see the value of the "Send-Q" on
>> both ends? If so, does it remain non-zero?
>
> I'm afraid you lost me on that one. The connection is in the state of
> ESTABLISHED. Where do I find this Send-Q information?

Ah, I see from you reply below that you're using Windows. Netstat on
Windows helpfully does not provide that information in order to prevent
you from being confused by information that might help you understand
what's going on. I don't know of any other way to get the information.
>> Actually, you've reminded me of something else: some NAT devices
>> choke on TCP connections where the IP TOS changes. Try either
>> rebuilding with "./configure --with-cflags=-DIP_TOS_IS_BROKEN"
>> or using a proxycommand such as netcat which doesn't set the ToS flags:
>>
>> ssh -o 'ProxyCommand nc %h %p' youserver
>>
>> Might be time to add these to the FAQ...
>
> I don't see any difference in either the triple-verbose output of the
> ssh client nor the Wireshark readout. I am using the version of
> netcat that comes with cygwin. Can you elaborate more on what this
> "proxycommand nc" thing is doing?

A ProxyCommand is run by ssh to make the network connection instead of
ssh making a TCP connection itself. The command above tells ssh to use
the "nc" ("netcat") command to make the connection (you'll need to install
it if it's not already installed).

Re: Using putty to debug ssh through a firewall

On Sep 14, 4:50 pm, Darren Tucker wrote:
> On 2007-09-13, NightStrike wrote:
>
> > On Sep 8, 2:25 am, Darren Tucker wrote:
> >> On 2007-09-07, NightStrike wrote:
>
> >> > Packets 6 and 7 were handshaking packets. They never go through, and
> >> > packet 6 is retried 5 times until it times out.
>
> >> That's interesting. If you do a "netstat -n" and pick out the TCP
> >> connection in question, can you see the value of the "Send-Q" on
> >> both ends? If so, does it remain non-zero?
>
> > I'm afraid you lost me on that one. The connection is in the state of
> > ESTABLISHED. Where do I find this Send-Q information?
>
> Ah, I see from you reply below that you're using Windows. Netstat on
> Windows helpfully does not provide that information in order to prevent
> you from being confused by information that might help you understand
> what's going on. I don't know of any other way to get the information.

Yes, cygwin. I could boot with a knoppix cd if it would help.

> >> Actually, you've reminded me of something else: some NAT devices
> >> choke on TCP connections where the IP TOS changes. Try either
> >> rebuilding with "./configure --with-cflags=-DIP_TOS_IS_BROKEN"
> >> or using a proxycommand such as netcat which doesn't set the ToS flags:
>
> >> ssh -o 'ProxyCommand nc %h %p' youserver
>
> >> Might be time to add these to the FAQ...
>
> > I don't see any difference in either the triple-verbose output of the
> > ssh client nor the Wireshark readout. I am using the version of
> > netcat that comes with cygwin. Can you elaborate more on what this
> > "proxycommand nc" thing is doing?
>
> A ProxyCommand is run by ssh to make the network connection instead of
> ssh making a TCP connection itself. The command above tells ssh to use
> the "nc" ("netcat") command to make the connection (you'll need to install
> it if it's not already installed).

Ah, I see. So you are trying to write more "raw" data, yes?

Here is what I found with some more digging using Wireshark:

I loaded up Wireshark to monitor the connection, and I made a telnet
connection. This would allow me to have finer control over what was
going on. What I found was that the usual sequence occurs:

Now since I'm using telnet, I can control what I do next. I tried
sending a simple carriage return (\r\n). This packet never goes
anywhere (it just keeps getting retried.) This says to me that
communication got blocked once the server sent its version string. Is
it possible that the firewall is monitoring the packet contents for
this, and is blocking anything transmission once it sees the
beginnings of SSH handshaking? I think it's clear at this point that
the issue is before the KEXINIT packet. The issue starts right after
the server sends a version string.

What I'd like to do (if possible) would be to try sending a munged
version string. Any ideas on that? The only server right now is
gentoo (my other server running cygwin just got a reformat last
night.. I corrupted the registry, and yada yada yada, it got
formatted . Gentoo is by its nature all from source, so I imagine
I'd have to edit the source somehow. Or maybe I could create a driver
program that does something useful?