Debugging what is going wrong with a VPN setup is difficult. The IKE protocol is “chatty”, and negotiates back and forth between the two ends for several rounds. The GUI offers not much help, it is either  UP or Down. Most of the real debugging happens inside the CLI.

One problem in particular that has always bugged me is that you need access to the end machines involved to initiate traffic across the link. The network admin typically doesn’t have direct access on the computers on either side of the VPN in order to initiate that traffic. I’ll show you a method that can be used to initiate traffic from that network as well.

Here are some basic steps to troubleshoot VPNs for FortiGate.

In IKE/IPSec, there are two phases to establish the tunnel. Phase1 is the basic setup and getting the two ends talking. Then IKE takes  over in Phase2 to negotiate the shared key with periodic key rotation as well as dealing with NAT-T (NAT tunnelling), and all the other “higher-end” parameters.

The first trouble shooting step is to verify your parameters are all correct and matching.

For Phase1, is the end gateway dynamic or static? Fortigate to Fortigate can use both Main and Aggressive modes for dynamic connections, but many other brands can not. In general, if you are supporting a dynamic IP client end, you will have to use Aggressive mode Phase1, so make sure that mode is set for dynamic clients. If this a static config, you should use Main mode for Phase1, which is a bit more secure on the initial handshake.

For Phase2, are both sides setup to use PFS? Replay Detection? Dead-peer detection? While most VPN setups include a set of encryption and hash algorithms, you only need one that are the same. The reason for the set is to offer many choices. In practice, just pick one that your base client supports and go from there. Now-a-days, AES256/SHA1 is probably supported across the board, and that is all I ever use. You don’t have to match the set of them exactly, each side just needs a common one to talk.

After that all checks out, we need to see what IKE is doing that is failing.

So SSH or console into the CLI.

If this is debugging a VDOM
(like in this case), you may have to switch into the root VDOM if you
are the system admin of the firewall as opposed to a VDOM admin.

fgt300C-fw # config vdom
fgt300C-fw # edit root
current vf=root:0

fgt300C-fw (root) #

as the diag commands are only available in the individual VDOMs or from the root VDOM for the system admin.

To enable debug logging on the console (should be default) do

fgt300C-fw (root) # diagnose debug console

To enable debugging output

fgt300C-fw (root) # diagnose debug enable

Phase1 debugging isn’t too useful. IKE/Phase2 debugging is where the problem almost always is. Lets turn on full debugging logs there.

fgt300C-fw (root) # diagnose debug application ike -1

Now, the problem I’ve always run up against is getting the tunnel to trigger to open up with traffic running on the link. You either have to conference in somebody with access to help you, or use this nifty trick…

Open another SSH connection to the FW CLI.  (If this is a VDOM, you’ll have to ‘conf vdom; edit “vdom3″ to get into
the VDOM context where the network is you want to troubleshoot).

Set the ping source IP address to be in the inside network of the host you are trying to troubleshoot..

fgt300C-fw (vdom3) # execute ping-options source 172.30.3.254

And now, ping away from the CLI in order to bring up the tunnel interface

fgt300C-fw (vdom3) # execute ping 192.168.0.1

(assuming 192.168.0.1 is an existing host only reachable via the VPN tunnel, and the ping service is allowed through the tunnel).

fgt300C-fw (vdom3) # execute ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=46.9 ms
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=47.3 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=45.5 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=66.3 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=45.7 ms

--- 192.168.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 45.5/50.3/66.3 ms

The trick here is that you are source as the network you are setting up, which should trigger the tunnel to come up if it isn’t up already, and you can see real live traffic. I don’t know how many times I’ve been stuck on a conference call waiting for whoever had access to do something to get around to doing the test I asked of them.

Back in the first debug window, you should see a whole bunch of IPSec and IKE messages fly past on the screen.

You have to learn to pick out the lines that are important, and zone in on them as everything is flying by. Learn to pause the display (or do a quick ‘diag debug dis’ to stop the output). Scrolling back and zeroing in on the one error out of 100 lines is going to be your key skill here.

If all is well, you should get something about the SA being established with the SPI value (not important).

ike 3:MyVPN_GW:18690:MyVPN:49143: added IPsec SA: SPIs=939fc892/b54d030

and of course, if it is configured for SNMP, something like

ike 3:MyVPN_GW:18690:MyVPN:49143: sending SNMP tunnel UP trap

is a nice confirmation that all is well with the VPN.

If you are seeing a lot of errors repeating with Phase1, and you see messages like

ike 3:MyVPN_GW:18698: sent IKE msg (P1_RETRANSMIT): ....

Most likely the problem is a mismatch preshare key for the VPN tunnel, as it isn’t passing out of P1 (which doesn’t have much to negotiate).

Also check again if this is dynamic client (generally requiring Aggressive mode) or a static connection that probably should be set to Main mode, but could be using Aggressive Mode.

If you don’t have a common encryption alg/hash, you should see some errors like..

ike 3:MyVPN_GW:18707: no SA proposal chosen

As it can’t find a matching SA between the two ends using the same encryption algorithm/hash combo to encrypt the tunnel. Fixup the encryption alg/hash and everything should go better.

The hardest problems to detect are different keylength timers (you’ll just have to review them on both sides to make sure your P1 and P2 keylife timers are identical on both sides). Problems that you encounter with different timers show up as a VPN that works for a while, but then stops work, and won’t come up unless you bounce both sides. With valid timers the same on both sides, the VPN should keep up and key rollovers happen automatically.

Also, DPD may not always negotiate. One side may have it on and let a VPN connection stay up for a certain time until the timer kicks off and closes the connection for the lack of keep-alive packets. Make sure both sides have it on, or both sides have it off.

There are a few other error conditions that may come up, but these are the more common errors.

The most important thing with the low level debugging like this is to learn to pick out the important error lines from all the rest of the junk flying by. It just takes practice. You may want to deliberately break an existing setup just to see what happens. But once you can zero in on that one error line out of a 100 that is important, it will be a lot easier to troubleshoot what problems may come at you.

“Dice predicted that IPv4 IPs would be completely allocated in the next 18 months.  Obviously (if this is true), they are interested in having people know something about IPv6 so they can have such expertise in their job exchange.

Do we as users of ipHouse need to be concerned about hardware, software, other implications of this — DSL, DNS, etc., etc. …”

Our network engineer Doug McIntyre responded and sent this to our customer:

The popular press picking up news about the pending IPv4 exhaustion is pretty much high gloss over any of the real meat of the matter.   The effects that this will have on the average user in the United States will be nil for at least 5 years if not longer. ISPs and the regional registrars will be sitting on their IPv4 pools, and there may be a space crunch if some new ISP takes off that can connect millions of devices in short order.  That could be read as – wireless carrier or cable company, but these companies have a large amount of IP space already.  Where the current crunch has been for some time, is in Asia.  China alone is quoted at having something on the order of 400 million people now on the Net.

IPv6 has been in testing for ages. I had IPv6 tunnels going in the 1990′s for the ISP I was working for at the time.  Needless to say, the ipHouse network core is full IPv6 routing.  Our website is available via IPv6:

$ dig +noall +answer www.iphouse.com aaaa
 www.iphouse.com.      597      IN      AAAA      2001:4980:0:4000::1

In fact, many of our servers are using IPv6; DNS, SMTP, POP, IMAP, members, ipMom, and NOC servers have been using IPv6 since October 2010.

Implementing IPv6 for hosting customers is very straightforward.  We’ve been doing it on a case by case basis as colocation customers request it.

Right now, one big issue is the consumer grade CPE equipment Those hardware manufacturers can’t be bothered to do anything until they absolutely need to.  As a result, most DSL boxes can’t do IPv6 connectivity natively to this day.  All the major enterprise network hardware can though.  Obviously all of our stuff does it just fine (Cisco, Juniper, F5, Fortigate).

How this will all play out in the end years down the road is, probably in Asia first, the lack of IPv4 addresses will start forcing access customers there only be able to get IPv6 addresses for their connections.  Thus, any content provider (ie. Google, Facebook, JoeBob Store) will be either forced to have a native IPv6 presence or their content will only be accessible through third party proxy gateways that convert it to IPv4.  (The first two companies I named already have IPv6 presence, it’s the little guys that may have to worry)  When the big IPv6 to 4 proxy gateways get so overloaded to be unusable any longer, those with native IPv6 presence will get the business that all these consumers are driving.

For people already connected here is the US, there will be not much noticeable change at all. Eventually the CPE access devices driven by the requirements in Asia will trickle back to the US, and people here will be able to have access in either IPv4 or IPv6.  If you’ve got an IPv4 address today, as most US connections do, you’ll be fine.  There is very little reason for content providers to disable IPv4 when the vast majority of customers are accessing them via IPv4.

A few days later we got this question about IPv6:

“Wondering what your plans are to support this with your home customers.  I have a Cisco 678 modem that I’m guessing doesn’t support it, but thought that maybe tunnels could be setup or something.  DNS might be a question, too.  Just thought I’d ask, because I know it’s just on the horizon.”

Again, our man Doug answered the question:

Our website is fully up on IPv6, as well as DNS, and the rest of our network.   We do offer native IPv6 DSL connections, but as you state the 678 doesn’t support it.  In fact, there’s hardly *any* consumer grade hardware that supports native IPv6.  You can do it with Cisco IOS based hardware (ie. Cisco 887), or by hacking some of the other consumer stuff and putting the OpenWRT OS on them.   Doing a tunnel is easy enough, but its probably easier for you just to go to HE’s Tunnelbroker.net service and have it done automatically by them.  Most of the IPv6 peer interconnects happen where their tunnels terminate anyway, plus they offer a lot of tutorials and such there.  If you do want the tunnel to terminate here just let me know.

The customer followed up and confirmed to us what we suspected:

“Just ran into this that confirmed that Cisco isn’t doing ipv6 for consumers but Netgear appears to be:  http://www.networkworld.com/news/2011/020811-cisco-linksys-ipv6.html “

So how does IPv6 affect our customers?  It doesn’t affect them – yet.   Once IPv6 becomes the only address available to an end users, the content they want  access to  needs to be available via IPv6, and content providers will make sure they can be seen.

If you’ve got technical questions or comments, send them our way.  We love to talk tech.

-Peace

Bil

If you have a Dialup, DSL or similar account at ipHouse with a ‘primary account’ within an iphouse.com, bitstream.net, pro-ns.net or goldengate.net domain you can now self-order additional POP mailboxes, and access an overview of mailboxes related to your primary account.

Check it out by logging into ipMom using your primary account.  You will see a new link titled ‘Add Mailbox’ under the ‘Add Services’ section of the menu.  You will also see ‘Account Overview’ under the Billing section.  You will be able to see your existing mailboxes and will be able to add new ones.  If you are already using all mailboxes that come with your account and you want to add a new one, you can check off an agreement to be billed for the new mailbox and a new POP mailbox will be added.  The new mailbox will be on to the next bill that is sent out (all billing information will stay the same).

You will not be able to delete mailboxes through ipMom.  If you do need to delete a mailbox, please email support@iphouse.net or billing@iphouse.net and we will be glad to help you out.

Customers with their own domain names who purchase mailboxes in blocks of ten will continue to enjoy the same admin control they currently have over their mailbox blocks.

We hope our customers enjoy this feature and we are excited for new things to come…:)

This is the mail system at host smtpin-2.iphouse.net.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

<blogtest@iphouse.com>: permission denied. Command output: maildrop: maildir
    over quota.

Breaking this error down; the first part tells us what server issued the rejection. In this case it was the host “smtpin-2.iphouse.net” – the mail server responsible for receiving email for the email address blogtest@iphouse.com   The second part tells us who we were sending the email to, in this case it was sent to testblog@iphouse.com  The third part tells us the error type, as in a permanent or temporary error. A permanent error means that the email won’t be delivered. A temporary error means that the sending mail server ought to try again later.  In this case it says “Permission Denied” – that’s a permanent error. The fourth part of this error message tells us why the email was rejected, in this case it says “Command output: maildrop: maildir over quota”. What this means is that the service “maildrop” says that the Mail Directory (maildir) is over quota – the mailbox is full. What this means for us the sender, is that we shouldn’t bother trying to send that person anymore email until they clear out their overstuffed mailbox. There isn’t anything we can do about it other that calling that person up to tell them that their mailbox is full.

Here’s the next error bounceback:

Unable to deliver message to: <blogtest@iphouse.com>
Delivery failed for the following reason:
smtpin-2.iphouse.net[216.250.188.181] responded with failure: 550 5.7.1 black
listed URL host ***.********.net by .black.uribl.com

This has been a permanent failure.  No further delivery attempts will be made.

I did make one edit to this error message, I replaced the blacklisted URL with a bunch of asterisks (*). I don’t see any reason to give a spammy site any more traffic than it already has. OK, so let’s break this error down. The first part of this error message tells us who we were sending the email to, in this case blogtest@iphouse.com was the intended recipient. The second part tells us what server issued the rejection, in this case it was the server smtpin-2.iphouse.net – so it was the recipient’s mail server issuing the rejection, not the sender’s. The third part tells us what kind of rejection it was, and why it was rejected. In this case the error code was “550″ (I’ll go into error codes later), and that the email was rejected because it had a link to a website (the one I used ***** to hide), which happened to be on the blacklist “black.uribl.com”. The last part is just another note that this was a permanent error, in case you didn’t already know that that’s what a 550 is. What this bounce back message means for us the sender is that we had a link to a website in our email that a spam filter on the recipient’s mail server didn’t like. We’ve got a few options here. The simplest thing would be to send the person a new email, but this time not include the offending website link. If we think that the website in question should not be on this blacklist in the first place (mistakenly listed), then we can always go to uribl.com and request that the offending website be removed from their list. Not all blacklists let you do this, but plenty do – and as it so happens, uribl.com lets you request removals. Another option would be to contact the intended recipient to tell them that a spam filter of theirs is rejecting that link. The intended recipient could then follow up with their IT staff, or their ISP. I suppose the final option would be to try to “game the filter” by breaking up the link a bit. For example, let’s say that iphouse.com was blacklisted somehow. I could try sending the person a link to iphouse.com like this:  www (dot) iphouse (dot) com    A human reading this ought to be able to figure out your intent, whereas a spam filter might be tricked.

Here’s the next error bounce back:

<testblog@iphouse.com>: host smtpgrey-2.iphouse.net[216.250.190.161] said: 550
    5.1.1 <testblog@iphouse.com>: Recipient address rejected: User unknown (in
    reply to RCPT TO command)

The first part of this error message tells us who we were sending the email to, in this case testblog@iphouse.com was the intended recipient. The second part tells us what server issued the rejection, in this case it was the server smtpgrey-2.iphouse.net – so it was the recipient’s mail server issuing the rejection, not the sender’s. The third part tells us what kind of rejection it was, and why it was rejected. It’s another 550, so a permanent rejection. It was rejected because the User was Unknown. This means that the email address doesn’t exist. If you caught the mistake in the email address right off the bat, plus 10 points for you. I accidentally sent an email to “testblog@iphouse.com” instead of “blogtest@iphouse.com”. Oops. As a funny aside, I actually generated this bounce back message accidentally when I was aiming for another type of bounce back. It took me a moment to catch my mistake. I guess that’s what more coffee is for…..    Anyhoo, what this means for us the sender, is that we just need to send out a new email, but this time type in the correct address. If you’re absolutely positively 100% double plus certain that you didn’t make any typo in the email you sent out, go ahead and look for the typo anyhow. It’s really easy to miss certain things like a “,” instead of a “.”, or a letter out of place. If your double check doesn’t yield any typos, and you’re certain that the address is a valid one, you might want to contact the intended recipient to inform them that something is amiss regarding their email address.