• Centralized management over several to many access-points.
• Unified access policies.
• Ease of deployment.
• Rogue AP scanning for PCI/DSS compliance.

Debugging what is going wrong with a VPN setup is difficult. The IKE protocol is “chatty”, and negotiates back and forth between the two ends for several rounds. The GUI offers not much help, it is either  UP or Down. Most of the real debugging happens inside the CLI.

One problem in particular that has always bugged me is that you need access to the end machines involved to initiate traffic across the link. The network admin typically doesn’t have direct access on the computers on either side of the VPN in order to initiate that traffic. I’ll show you a method that can be used to initiate traffic from that network as well.

Here are some basic steps to troubleshoot VPNs for FortiGate.

In IKE/IPSec, there are two phases to establish the tunnel. Phase1 is the basic setup and getting the two ends talking. Then IKE takes  over in Phase2 to negotiate the shared key with periodic key rotation as well as dealing with NAT-T (NAT tunnelling), and all the other “higher-end” parameters.

The first trouble shooting step is to verify your parameters are all correct and matching.

For Phase1, is the end gateway dynamic or static? Fortigate to Fortigate can use both Main and Aggressive modes for dynamic connections, but many other brands can not. In general, if you are supporting a dynamic IP client end, you will have to use Aggressive mode Phase1, so make sure that mode is set for dynamic clients. If this a static config, you should use Main mode for Phase1, which is a bit more secure on the initial handshake.

For Phase2, are both sides setup to use PFS? Replay Detection? Dead-peer detection? While most VPN setups include a set of encryption and hash algorithms, you only need one that are the same. The reason for the set is to offer many choices. In practice, just pick one that your base client supports and go from there. Now-a-days, AES256/SHA1 is probably supported across the board, and that is all I ever use. You don’t have to match the set of them exactly, each side just needs a common one to talk.

After that all checks out, we need to see what IKE is doing that is failing.

So SSH or console into the CLI.

If this is debugging a VDOM
(like in this case), you may have to switch into the root VDOM if you
are the system admin of the firewall as opposed to a VDOM admin.

fgt300C-fw # config vdom
fgt300C-fw # edit root
current vf=root:0

fgt300C-fw (root) #

as the diag commands are only available in the individual VDOMs or from the root VDOM for the system admin.

To enable debug logging on the console (should be default) do

fgt300C-fw (root) # diagnose debug console

To enable debugging output

fgt300C-fw (root) # diagnose debug enable

Phase1 debugging isn’t too useful. IKE/Phase2 debugging is where the problem almost always is. Lets turn on full debugging logs there.

fgt300C-fw (root) # diagnose debug application ike -1

Now, the problem I’ve always run up against is getting the tunnel to trigger to open up with traffic running on the link. You either have to conference in somebody with access to help you, or use this nifty trick…

Open another SSH connection to the FW CLI.  (If this is a VDOM, you’ll have to ‘conf vdom; edit “vdom3″ to get into
the VDOM context where the network is you want to troubleshoot).

Set the ping source IP address to be in the inside network of the host you are trying to troubleshoot..

fgt300C-fw (vdom3) # execute ping-options source 172.30.3.254

And now, ping away from the CLI in order to bring up the tunnel interface

fgt300C-fw (vdom3) # execute ping 192.168.0.1

(assuming 192.168.0.1 is an existing host only reachable via the VPN tunnel, and the ping service is allowed through the tunnel).

fgt300C-fw (vdom3) # execute ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=46.9 ms
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=47.3 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=45.5 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=66.3 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=45.7 ms

--- 192.168.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 45.5/50.3/66.3 ms

The trick here is that you are source as the network you are setting up, which should trigger the tunnel to come up if it isn’t up already, and you can see real live traffic. I don’t know how many times I’ve been stuck on a conference call waiting for whoever had access to do something to get around to doing the test I asked of them.

Back in the first debug window, you should see a whole bunch of IPSec and IKE messages fly past on the screen.

You have to learn to pick out the lines that are important, and zone in on them as everything is flying by. Learn to pause the display (or do a quick ‘diag debug dis’ to stop the output). Scrolling back and zeroing in on the one error out of 100 lines is going to be your key skill here.

If all is well, you should get something about the SA being established with the SPI value (not important).

ike 3:MyVPN_GW:18690:MyVPN:49143: added IPsec SA: SPIs=939fc892/b54d030

and of course, if it is configured for SNMP, something like

ike 3:MyVPN_GW:18690:MyVPN:49143: sending SNMP tunnel UP trap

is a nice confirmation that all is well with the VPN.

If you are seeing a lot of errors repeating with Phase1, and you see messages like

ike 3:MyVPN_GW:18698: sent IKE msg (P1_RETRANSMIT): ....

Most likely the problem is a mismatch preshare key for the VPN tunnel, as it isn’t passing out of P1 (which doesn’t have much to negotiate).

Also check again if this is dynamic client (generally requiring Aggressive mode) or a static connection that probably should be set to Main mode, but could be using Aggressive Mode.

If you don’t have a common encryption alg/hash, you should see some errors like..

ike 3:MyVPN_GW:18707: no SA proposal chosen

As it can’t find a matching SA between the two ends using the same encryption algorithm/hash combo to encrypt the tunnel. Fixup the encryption alg/hash and everything should go better.

The hardest problems to detect are different keylength timers (you’ll just have to review them on both sides to make sure your P1 and P2 keylife timers are identical on both sides). Problems that you encounter with different timers show up as a VPN that works for a while, but then stops work, and won’t come up unless you bounce both sides. With valid timers the same on both sides, the VPN should keep up and key rollovers happen automatically.

Also, DPD may not always negotiate. One side may have it on and let a VPN connection stay up for a certain time until the timer kicks off and closes the connection for the lack of keep-alive packets. Make sure both sides have it on, or both sides have it off.

There are a few other error conditions that may come up, but these are the more common errors.

The most important thing with the low level debugging like this is to learn to pick out the important error lines from all the rest of the junk flying by. It just takes practice. You may want to deliberately break an existing setup just to see what happens. But once you can zero in on that one error line out of a 100 that is important, it will be a lot easier to troubleshoot what problems may come at you.

Back in the day when we started out routing networks for customers, generally the practice was to route a whole Class C (or /24 now-a-days, this was before CIDR  was widely deployed as well), or maybe 2 or 4 depending on how many IP addresses that customer, or larger department needed. Large companies didn’t have connected networks, each workgroup or department was usually separate, or if they were really big, we’d help get them a Class B network with the Internic.

There wasn’t the prevelant use of firewalls, these connections were directly on, or if there was a firewall, it was a proxy type, with the stations having the ports blocked and needing to proxy out for things like FTP or HTTP if that was even a consideration. It was pretty apparent even than that having a total of about 2 million Class C’s available wasn’t going to be able to service a lot of customers with these types connections. Especially when you look now-a-days and see a large ISP like Comcast easily having 2 million subscribers in one larger metro area. So, out came CIDR which saved us from handing out /24s in every case and it certainly cut down on handing out IP addresses wastefully.

It took a while for people to get the concept of CIDR and even longer for gear to fully support it properly but it helped.

In the meantime NAT was created.

At first it was more one-to-one translation so that if some workstations were in an unfortunate range they could be translated to another range of IP addresses that worked better. Then PAT (aka NAPT) came about that actually let us hide a range of machines behind one IP address on the outside. This was starting to look a lot more like what the current crop of residential routers allows today; hide a whole range of networks behind one public IP address so that we can go back to the model of having one IP address per customer. Finally what allowed that was modifying existing protocols that broke with NAT to be more NAPT friendly, recognizing that something can still be in the middle. IPSec got NAT-T transport, and enterprise class firewalls got things like ALG which allowed more deeper inspection into protocols to help things along like FTP, SIP, and H.323 which do weird things embedding raw IP addresses inside the protocol itself rather than just the headers.

The only reason the Internet could grow at the rate it did was that NAT saved the ISPs from having to deploy huge tracts of IP addressing that wasn’t going to be used (do you have 256 devices connected up in your home network? Multiple your home by millions of others in your metro area). Now that we have exhausted IPv4 it seems like they should have done more, but really at the time, nobody envisioned 78% (240 million) of the US population to be connected on the Internet, let alone the adaption rates around the globe.

What is broken now.

When I was working with these large enterprise companies they had deployed NAT to all their borders.

They’ve totally isolated themselves off from the world, anything going out and going in has to be NAT thus when we needed to get into their networks with the VPN tunnels it produced all sorts of interesting problems that never ever was imagined way back when the Internet was first launching.

Since NAT was required talking even back to us over the VPN tunnel, we determined that things like terminal server (TS) gateway which our customer was trying to deploy would be failing because of the NAT requirements. I don’t know of any firewall that has an ALG for TS gateway but even if there was one, it most likely wouldn’t have been deployed by these large dinosaur of an enterprise company in their IT department.

Thankfully we weren’t doing any VoIP connections for these people but those would have broken too, as SIP and H.323 and MGCP all require ALGs on the firewall level to deal with NAT. Since we were hidden behind NAT things would have broken pretty hard getting these types of packets back to them.

Finally, since they NAT everything in and out, any and all external DNS are broken too, so they have to take over DNS deployments for our customers machines, further isolating them off from the rest of the Internet. Normal DNS entries and domains don’t work in the large IT enterprise enviornment, custom DNS has be requesitioned, setup, tested and deployed at each site, negating any benefit for a single authoritative source (which when DNSSec takes off, will be promptly ignored because it won’t be used).

The future.

Since the widespread rollout of IPv6 recently it has been so nice to get back to the roots of the concepts of unique IP addressing for end-to-end communication. Unfortunatly, these kinds of mindsets that have isolated the large enterprise off the Internet directly are pushing IPv6 deployments to have the same sort of NAT in, NAT out mentality matching what IPv4 rollouts they have already.

NAT66 has several drafts going through the IETF process now. I was really hoping that we could do away with all the nastiness of NAT and we certainly will in the general deployment for most people. But I fear the driving force behind these large enterprise IT will again make things so convoluted and isolated that we will be dealing with NAT issues for quite some time to come.

unless you don’t realize that its been enabled for you, in which case you’re likely to spend an hour bashing your head into something trying to get nfs to work. ufw is normally driven from the command line, although a GUI is also available.

you’ll need to have root privileges to run ufw.

the command to see whether or not ufw is running is ‘ufw status’. if ufw is not running, you should see…

$ ufw status
Status: inactive

if ufw is running, you’ll see something like this instead…

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
Bind9                      DENY        Anywhere
22                         ALLOW       Anywhere
3306                       DENY        Anywhere
Apache Full                ALLOW       Anywhere

here, ufw is active, and is configured to deny or allow specific types of traffic. for example, connections to port 22 (ssh) are allowed from anywhere, whereas connections to port 3306 (mysql) are denied from anywhere. in addition to simple port numbers, ufw can recognize applications, such as ‘Apache Full’ (ports 80 and 443/tcp). for more information, see the Application Integration section of the man page.

the basic command for opening ports is ‘ufw allow 161′. ufw will also refer to the /etc/services file if you specify services by name, ‘ufw allow snmp’. either will allow connections to port 161 (SNMP) from anywhere.

these examples imply ‘from any to any’, but you can also specify source and host addresses. for example, ‘ufw allow from 10.0.0.42 to any port 161′ only allows connections to port 161 from a single address. you can also specify a netblock, such as 10.0.0.0/24. there are additional options, including specifying protocol (tcp or udp) and direction, and limiting or rejecting connections (instead of dropping them); see the man page for these and for more examples.

also note that rules are processed in order, and the first match wins. more specific rules should come first, followed by more general rules. you can insert a rule into the list using ‘ufw insert 2 allow snmp’ (here, into the number 2 slot, moving the current number 2 and following rules down one slot).

ufw isn’t a bad thing to have running on your ubuntu host, and is particularly important if its not behind a network firewall. but if you’re having problems getting a new service running, it’s probably something worth looking at.

Hardware configurations, including manufacturer choices, operating systems versions and configurations, firewall rules and ongoing maintenance of all the above heavily impact the performance and reliability of your systems.

Regardless of whether you have computers in your broom closet, colocated at your ISP or deployed in the cloud, your company needs a good system administrator looking out for your network and machines. Good system administrators know the pros, cons and quirks of different hardware, operating systems and network configurations. They know about possible vulnerabilities first because they are on private security lists you don’t even know exist. They’ve got your back. George Reese of enStratus, expanded on this in a recent post that compares programmers and sys admins.

One of the big differences between an ipHouse virtual machine (which is essentially deployed in a local cloud) and deploying a server with one of the national cloud providers, is the sys admin expertise that comes with your ipHouse machine. We work with you to make sure the system configuration is optimized for your business applications. We can also administer the machine for you, keeping it securely patched and up-to-date.

On Monday, April 27, the wise and knowing Minnesota Department of Public Safety (MDPS),  Alcohol and Gambling Enforcement Division (AGED) delivered written notice to 11 telephone / Internet service providers demanding they “prohibit access to all Minnesota-based computers to nearly 200 online gambling websites.”  Here’s a link to the press release. 

Ok, this is the Internet we’re talking about, right?  You know, the Information Superhighway?

I am guessing that these 11 respectable companies are recognized as Common Carriers by the great state of Minnesota.  That must be the only criteria for being selected for this list, otherwise, we at ipHouse would have received a request too.  Just for the sake of clarity, as of this posting, we have not received a request from the AGED.  But if we had received a request, we would have asked for some kind of legal backing.  And that’s where this falls down.  The great state of Minnesota is relying on the Wire Act of 1961 to enforce this ridiculous request.  

What I can’t see is how this request can be enforced, even using the Wire Act.  Before I snicker at any enforcement discussion I’ll put that question aside and just wait and see.

Now, as a citizen, I understand that the Minnesota Department of Transportation does not expect the companies who build our roads and bridges to enforce the speed limits on the roads they build.  Further, we would never expect or request these same construction companies to do vehicle contraband inspections at the state border.  So, WHY ON EARTH does the Minnesota Department of Public Safety  think that they should conscript the builders of the Internet (Information Superhighway, get it?) to do their enforcement?    Why not go after the people who are committing crimes instead of the people who build the roads?  You don’t task road builders with catching drunk drivers, do you?

John Willems is the director of AGED and I can’t help but wonder  what he was really thinking when he said this:

“In broader context, the long-running debate on online gambling continues to raise significant issues, including absence of policy and regulation, individual rights, societal impact, international fair-trade practices, and funding for criminal and terrorist organizations.”

Does he really think that Joe the Plumber is betting on the Red Sox and innocently funding Al-Qaeda?   Come on.  Isn’t the whole terrorist thing a little over used? 

I agree that there is a long running debate on gambling in our society.  But it’s not just online gambling.   To me, the issue of gambling in our society PALES in comparison to some of the other issues Mr. Willems mentions; individual rights and international fair trade practice.  If Minnesota is going to remain competitive in the WORLD, we cannot be xenophobicly locking down our borders to international trade across any of our transit ways, be it by water, air, rail, road or Internet.

Now, as you look at these various transit ways, all of them EXCEPT the Internet have a specific geographic nexus.  Nearly all transit ways have ports of entry and it’s easy to see geographic boundaries between nations and states.  It’s pretty easy to understand the nexus of a shipment of goods coming across the St. Lawrence sea way is the port of entry at Duluth harbor.  It’s all very black and white.  But the Internet is in as gray area and different because the NEXUS of the transaction is vague.  What is the nexus of a Minnesotan purchasing software from Belgium or India?  What happens when part of the software is written in China?    The nexus of Internet transactions are VAGUE.

It appears that Mr. Willems has defined the nexus of online gambling is at the individual users computer, right here in Minnesota.  If that’s right, then Mr. Willems should target the individuals who are committing the crimes.  Why not go to the credit card companies and ask them to report all the transactions between the citizens of Minnesota and these 200 gambling websites?  Because he can’t afford to.  It’s easier for him to push on the road builders instead of all the motorists who use the roads.

We all know that as citizens of Minnesota have REAL problems that need REAL attention.  Like drunk driving and alcohol addiction.  Like air pollution and lung disease.  If Mr. Willems wants to protect the citizens of the great state of Minnesota, maybe he should focus on some of the more pressing problems facing the state.

I’m a firm believer in regulating things to protect our society.  Regulating polluters so future generations can enjoy the outdoors seems obvious to me.  Regulating alcohol sales to prevent underage drinking, I’m all on board.  So why not legalized and regulated online gambling?  It could be a revenue source for the state just like the other areas that Mr. Willems has under his jurisdiction.  Mr. Willems, why not be progressive and start regulating online gambling like you do with bricks and mortar gambling?  

Whatever the outcome Mr. Willems, just don’t ask me to collect your revenue for you.  I’m neither an enforcer nor a tax collector.  I’m a road builder.  

Peace.

-Bil

Well THEY have thought of a way to track and sell information about you using cookies placed on your computer while shopping.  I have been a privacy hawk when it comes to cookies for a while (more on that below).  As such I was surprised that a  NYTimes article was the one to inform me about a newish Internet marketing technique that uses behavioral targeting and cookies across multiple sites.   Read the article quick or you may have to register for their website to read it.   Basically there are companies (the two biggest are eXelate and BlueKai) that work with online merchants to place tracking cookies on your computer and mate them with information about your interests.

Those interests may be garnered from products you add to a shopping cart, to search terms on those websites, and pages/products you read about on the websites.  These Cookie Monsters then essentially sell the cookies with targeted information to buyers.  I suppose there is an argument that “you are going to get ads anyways so and might rather look at targeted ads rather than random ones”.  But what would stop a online store from pairing these “anonymous preferences” with your personal information they get from their shopping cart?  I suppose the Cookie Monster’s terms of service say they can’t do that, but I am sure it will happen.  After all, anti-spam companies are now spam-promoting their anti-spam services.

So, there you go – yet another thing to worry about.

If you are paranoid about cookies (like me – go ahead make fun of me in comments)…  I use Mozilla Firefox and have privacy preferences set up to block all cookies.   For sites that I trust that require cookies, I add that domain name as an accepted cookie.  Firefox also lets me add these exceptions with a condition that deletes the cookie at the end of the browsing session.  Therefore when I went to BlueKai’s preferences page to see what info they have about me I got a pleasant message “We currently do not have anonymous information of your online preferences.”

While we are on the subject…  Google recently announced behavioral targetting although they call it “interest-based” advertising.   When I read Google’s disclosure it just doesn’t seem as intrusive and open to improper capture of preferences with personal data as these other solutions.  Just in case I’m drinking the Google Kool-Aid, here are a couple other blogger takes…

• Transparent Google?
• Google Watches its Language

- Eric Snyder

Phishing is spam that attempts to extract personal information from the recipient. Here are some quick points about Phishing:

1. Email asks for your password: ipHouse will never ask for your password via email. This is a common policy with many companies so feel free to make it your own policy: Never send a password via email even if you think you know the recipient.

2. Strange reply-to address: The reply-to email address is not an official email address. ipHouse employees and internal addresses are all @iphouse.net. This latest round had the reply-to as an email address in Brazil (.br) or a yahoo.com address. A general rule for anyone is to always check a provider’s website for valid contact information. When going to their website type in the address yourself or use an existing valid bookmark. Do not click a link in an email even if it looks valid is it may be a “masked” URL whose destination is a different address.

3. Credit card fraud. While this email was looking for passwords, many Phishing scams ask for credit card numbers. And for decades there have been phone-based credit card Phishing scams. ipHouse will never ask for your credit card number via email nor ever via a call we initiate. Feel free to make it your own policy with everyone – never send a credit card number via email and never give your credit card number out to someone unless you initiate the call.

4. Spam filters don’t catch everything. While our multiple levels of Antispam catch most Phishing expeditions, some can get through. This one was harder to catch as it didn’t have any off-site hyperlinks and had enough words that it looked valid to the filters. We don’t publish for spammers how we adjust but trust me that we do adjust. Of course we do want to see what might get through. For example, yesterday alone ipHouse blocked 1,463,418 spam, Phishing, and viruses. We pride ourselves on an extremely low “false positive” rate. If a spam or Phishing message does get through, please forward it with full headers to spam@ipHouse.net. If you have an individual question or concern, our Support team can help.

5. Learn more! Here are some links to several sites’ take on Phishing:

• Blogs about Phishing: PhishingScam
• Popular OS: Apple, Microsoft
• Popular Guides (always with a grain of salt please): WikiPedia , About
• Trade/Industry groups: APWG, National Cyber Security Alliance, AARP
• Government: Stop-Think-Click

- Eric