[GLLUG] A weird networking problem. Help!

[Tim Woodall]

On Thu, 22 Jun 2017, Tim Woodall via GLLUG wrote:

> I have the most bizarre networking problem and I'm struggling to think
> what could possibly be causing it (other than a failing hack attempt by
> the NSA or the like - I don't think I'd be interesting enough for anyone
> to deliberately try to intercept my communications though)
>
> The symptoms manifest as failing (outbound, haven't tried inbound while
> it's happening) SSH connections. The SSH connections seem to fail to
> anywhere.  But http and https connections to the same host work fine.
> (and there's no MITM certificate interception happening). Running ssh
> and telling it to connect to a non ssh service fails in the way you
> expect. Running SSH to an ssh service "hangs" and eventually times out.
>
> SSH connections fail even if I try to run them over a non-standard port.
>
> The SSH connections are failing from multiple machines - initially I
> suspected my laptop was playing up from the heat, but I switched to
> another laptop that I haven't used for a while and that has the same
> problem.
>
> Rebooting the laptop helps for a short while. Rebooting the ADSL router
> helps for a longer while.
>
> I use a separate access point connected to a port on the ADSL router -
> I've just switched that cable to a different point on the router and so
> far ssh is still working.
>
> I think it's got to be the ADSL router, but I cannot for the life of me
> imagine what could be going on that's breaking SSH but not HTTP/HTTPS
> other than some attempt at deep packet inspection that is (deliberately
> or accidentally) causing SSH connections to fail completely. It's been
> happening for about two days now.
>
> If it happens again can anyone think of what tests I should do? I have
> physical access to both ends of one connection so I can potentially log
> the traffic at both ends. I can also change the port I'm using. (I
> haven't tried doing ssh to port 80/443 but I might try that if
> my cable move doesn't permanently fix the problem.
>
> Although the problem is very annoying and breaks a lot of things I do,
> the fact that it seems to only affect SSH connections is troubling.
>
> I haven't (yet) tried an ipv6 connection. The endpoint I have access to
> doesn't, currently, have ipv6 on it's external interface as I've never
> gotten around to updating my firewall rules for ipv6. But I could try
> without a firewall with a bit of config change.
>
> Tim.
>
>

It's starting to make sense. Looks like my router is suddenly deciding
to stop forwarding almost anything reliably other than port 23, 25, 80,
110, 143, 443 (probably a few more but I don't have a good way to check
other than just trying things)

Strange that it works for a while before deciding to be my own private
filter to the internet. It's worked perfectly until a few days ago.

Now that I have a way to have a stable connection I can monitor the
other
end while I try to connect from this end and I can see that the packets
never normally arrive.

Interestingly when I hammer the router with 256 telnet connections
across 256 ports as quickly as possible then many of the TCP syn packets
do make it through.

I think my router is very sick! I cannot connect from A->B (because B
only listens on port 22) but I can ssh from A->C on port 23 and then
from C to B on port 22.

What fooled me was that I normally run my ssh server on a non standard
port - and so when that broke plus ssh on a standard port too then I
throught it was SSH that was a problem rather than the port number.

Tim.