I touched on monitoring in an earlier post but I thought that I would expand on my thoughts.

Let me just get this out there: LogicMonitor (company site) is awesome. It’s not perfect (what is?), but it’s amazing, simple, straightforward, and it works. It combines effective monitoring with graphing (metrics); it’s easy to understand and customize and it works.

Repeat: It works.

I’ve done some work with other monitoring and graphing/measurment solutions; mostly Zabbix, Nagios, and Cacti. They all have their strengths and weaknesses. LogicMonitor also has it’s plusses and minuses but all in all it works amazingly well with the number of minuses to be very small.

Nagios has, in my opinion, the best monitoring engine. The automatic back off and flap detection combined with per-host customization that can happen in Nagios has not been matched yet. However, configuring Nagios is a nightmare. I got really good at it and I don’t want to ever do it again. Looking at a blank Nagios setup makes me cringe. Tools like NagioSQL help but it’s still ridiculous. Using Nagios as a customer facing solution would take up too much time and my time is precious to me and our business.

Cacti is not a monitoring system but it is a great graphing solution, unless your RRD data gets corrupted or lost. Now, that doesn’t happen much, but when it does, it’s annoying.

Zabbix is a great all in one system with a horrible interface. I hate to quibble, I still use Zabbix but I get headaches everytime I try to do something. The top down task selection with a history at the bottom is counterintuitive. Getting Zabbix to send out alerts is a chore. And requires per-host agents for different operating systems while the SNMP interface works well only if the device you are monitoring fits within the very small pre-configured templates that come with the package. Yes, I can build new templates, repeatedly but LogicMonitor does this without requiring extra time.

With our recently launched vmForge service offering, we wanted to add an excellent and easy to implement monitoring solution. It was something that we wanted to be able to set up for customers easily while also offering something that they could set up and manage themselves.

Mike did quite a bit of digging but didn’t find anything that fit the bill entirely. Until he stumbled on LogicMonitor.

It initialy attracted our attention because it was network agent based. This allows us to put agents behind firewalls and NAT configurations without worrying about all of the details. The agent just requires outbound connectivity over HTTPS.

We decided to give it a try and we were instantly impressed! It automatically detects available datasources and adds threshold points and instrumentation graphing of operations in a single view. We can add rules and chains for alerting the engineering staff. It has a lot of features laid out in an easy to understand way. It uses SNMP, vendor APIs, and WMI depending on the target host.

It makes sense so we  fired up an evaluation and not long after signed up for services for our own use.

The developers of LogicMonitor have been great to work with. They have been open to feedback, excited to test things that they haven’t come across before. We receive queries on how a specific type of device should be measured and bug reports are handled professionally and efficiently.

The only thing that I don’t like is that the agent requires Java but that’s the cost of convienence.

The only things missing right now are support for IPv6 (which can’t come too soon) and a back off ability with flap detection. (spouses are happier when not woken up to dropped detection events)

Oh well, it’s still better than editing Nagios files!

I’m looking forward to working with LogicMonitor further and I highly recommend them.

Setting up a monitoring agent on your local network is easy. The server hosting the agent just needs a JRE (Java Runtime Environment) installed using version 1.6 or greater and must be able to make an outgoing SSL connection. To monitor Windows systems, you’ll need to install the agent on a Windows server.

Login to the LogicMonitor website, click on the “Settings” tab, then on “Agents” in the left navigation, then on the “Add” button. Click past the introduction, and indicate whether you’ll be installing the LogicMonitor agent on a Windows or Linux server. Download the agent installer, or copy the link and use wget to download the installer directly to your Linux system. Run the installer to install the agent on your server then return to your web browser and click “Next” to verify that its been installed correctly and is able to communicate with the LogicMonitor system.

To begin monitoring a host on your network, click on the “Hosts” tab, then on the “Add Hosts” button and select “New Host (wizard)”. Enter the host name or IP address. Note that if your monitoring agent and host are on a private internal network then this should be the IP address visible to your agent. Select your monitoring agent (if you have more than 1), and LogicMonitor will go ahead and verify that its able to gather information about the host.

NOTE: at this time, LogicMonitor does not support IPv6

I was in great need of Nick’s help because at first I thought I could create just any host name. So of course I chose the name barf. Well you can’t do that. You need to use a machine name that already exists. Nick said I should choose smtpgrey-2.iphouse.net or smtpgrey-1.iphouse.net (inbound SMTP border servers in use on our network).

Once I figured that out it was all smooth sailing!

Below is the Logic Monitor interface.

If you look to the left, my name is Gen.

So you say you would like to add a host. Click on the ‘add a host’ button on the left side. Then this screen pops up.

You have to pick a host name that already exists. Now you select an monitoring agent. I chose worldgen. You do need to choose one that will Windows based if it is a Windows machine so you can use WMI if your firewall allows such. Remember that.

Once you pick your agent, the wizard will check to see if every choice you made works out. Apparently, I did things correctly.

I decided that I do not want to add another host. I think I am good for now.

Okay, okay…I am sure I want to exit.

So now, this is what Logic Monitor really does. It monitors.

You can toggle around and see what has gone on with your CPU usage and gather load averages plus a whole slew of other statistics.

Now it is monitoring Disk. Watch it monitor.

DNS status seems to be neutral.

Okay this is the part that I think is the most interesting about Logic Monitor, the alerts. I know that no one wants to see a critical message. It sure is fun when you are testing new software though. I did not have any alerts so I moved on to show you someone elses alerts.

Here we go, this is critical as you can see from the bright orange and yellow colors.

There is a section under notes that you can click on and add in a note on what is going on or how you are going to fix it or if you did.

After all of this, I think Logic Monitor seems quite useful. It is nice that when an alert happens (because they do) a page can be sent to your cell phone.

Thanks for reading my series on moving from my single all-in-one server and my small ESXi server to ipHouse’s vmForge VDC product. I previously discussed moving my websites to a virtual webcluster, and moving email to a virtual mailcluster. Now I just had to move three small servers, and install a third.

The first server I moved was a small experimental VM used for testing various network, web and other items. I like to have dedicated testing environment for every operating system that I professionally run. This server was responsible for my personal Teredo tunneling, and was the one I put my CGI testing on from awhile a go. I could have easily moved it, but I wanted see how the export/import from ESXi to vmForge worked. I stopped the machine on my ESXi server, downloaded it as a OVF and uploaded it, via my Windows machine, to my catalog. It imported it as a template. I then deployed the template and deleted the server. It worked flawlessly! All I had to do renumber the machine and I was done.

The next server was a little more complicated. It was originally a CounterStrike:Source server that I had converted into a Apache Tomcat JSP host. Because it already had a working Java setup, I added an OpenFire Jabber server, and a LogicMonitor agent to it. This gave me the ability to monitor my internal network from LogicMonitor, a monitoring solution that we’re looking into. The triple Java duties of this machine, unfortunately, put a big crunch on its RAM, so that took a lot of tweaking on the application level to get them to play nicer with each other.

The next server was a monitoring server that I had set up running Zabbix. I had previously gotten Nagios working on it, but it was too burdensome for me to maintain. I also liked having graphing and service level alerting as well as agent based checks, both active and passive. The biggest problem with Zabbix was getting it initially set up to send alerts, so it was nice to be able to import this machine, that had a working base, than to start from scratch. LogicMonitors does pretty much everything that Zabbix does, and better, but why not have two monitoring solutions? I also set up that machine to be a centralized logging server if I ever want to install a log analyzer like Splunk. I set it to copy the logs to a MySQL database, and to run php-logcon, but that didn’t scale past a few thousand entries.

Next was installing a FreeBSD server to act as a centralized tool, mail environment, and storage space for myself and my friends. I love FreeBSD, the only reason I set up my other servers as Linux boxes was pure laziness on my part, which I’ll pay for later in administration time. Also, they are mostly single purpose appliances, and it’s nice to have some of the Debian style scripting for web built-in. I try to stay fairly OS agnostic, but I do have preferences.

Since my shell server would have the most exposure to the internet, so I wanted a relatively secure system. Also, I would be spending most of my time in that server, so I decided to go with the OS I love. That would also bring things full circle, as my pfSense box and Shell server are both FreeBSD.

I decided on installing FreeBSD 8.2 stable. I sliced my disks like this:

/           512MB
swap        1GB (1x Memory)
/usr        5GB
/var        10GB (Modest space for DB and info)
/home       140GB (An egregious space for storing files)

I installed the OS and ports, and I switched from cvsup to csup awhile ago, and updated my ports-supfile and stable-supfiles to point to a local(ish) mirror, and checked out /usr/src and /usr/ports. I then updated my kernel config (Tip: compile without debugging if you want it to fit in 512MB ) reinstalled, and rebooted. Voila! A new FreeBSD system. I’ll probably go into doing a comprehensive FreeBSD install in a later post.

I installed Postfix and Dovecot2 for local mail, Apache 2 for user directories, and migrated my users information, passwords, and home directories from my old server. Everything went surprisingly smooth. I installed Mutt for myself, Alpine for one of my users, and a few other pieces of software, and I had a fully running shell server. I was going to run PowerDNS and PowerAdmin on one of my Linux boxes, but I decided to stick with BIND on the FreeBSD server, as it was more efficient for me to edit text files than use a web interface. Weird, I know. Now that my shell server was done, and everything was migrated, I could turn off my old FreeBSD box. I admit that I did feel a little bad as I typed halt into its shell for the last time. It served me well over the last four years.

Now my infrastructure migration was complete, running fully virtualized, lowering my power consumption, gaining redundancy, and boosting performance for the fraction of the cost of having physical infrastructure.

I Win!

Game Over.

I had a problem. All of my personal infrastructure was on an aging server, cobbled together from various parts that were laying around. I had already replaced the motherboard once, and I was not looking forward to doing more maintenance. The system had 5 320gb SATA disks in a RAID 5 setup. Not very fast, and it could only survive one disk failure.

Software-wise the machine had long exceeded what it was designed to do. It was originally designed as a game server, with some web and email. I had added several other services to it as I learned and played. Data was spilling out of its assigned slices. Symlinks were used strategically but it was still a mess.

It was time to replace the server, and ipHouse’s new vmForge VDC was the ideal place to do so. I bartered with my boss by offering to get rid of my power hungry hardware in exchange for a modest virtual data center. He accepted. I was allowed to provision a VDC with 16 GiB of RAM, 8GHz of CPU, and 500 GiB of storage and a /28 worth of IP addresses.

I also had another server, with ESXi running on it, with a few VMs used for monitoring and FCGID based websites, and another FreeBSD server that had a few test items on it that I wasn’t using much. It was time to reduce and consolidate my infrastructure.

The first thing to decide was the network layout. Unfortunately, a Virtual Domain (VDOM) off of one of our Fortinet Firewalls was out of the question as they are too valuable to comp to an employee. So I was left with a vShield Edge device or running my own firewall as a VM.

Running my own VM won out as I wanted to do some rudimentary load balancing (something beyond round-robin DNS) and the vShield couldn’t do it (this feature isn’t exposed through VMware’s vCloud Director in version 1.5). I had previously futzed with pfSense, and was comfortable with using it. Unfortunately, this choice would cut into my available resources.

In order to facilitate my firewall, I asked our network guy to route my /28 to a /30. I also asked him to route me an IPv6 /64. I then set up my VDC (I provision them as part of my job) with two networks, an external one with the /30 on it, and an internal one with a bogus RFC 1918 network, set to DHCP so that it wouldn’t assign those addresses.

After the VDC was set up I booted my Windows XP VM in VMware Fusion, and uploaded the pfSense ISO to my VDC’s private catalog. Then I built a networking vApp, added a VM, “inserted” the ISO, booted it up, and installed pfSense via the console. Very easy, very quick.

After installing pfSense I was disappointed to find out that the current 2.0 release barely supports IPv6. I’m a big IPv6 fan, so I bit the bullet, and moved up to 2.1beta code as this would allow me to run IPv6 natively in all (well, most of) the services.

I installed additional few packages: Open-VM-Tools, OpenVPN Client Export, pfBlocker, snort, and mailreport. Using Snort as and IDS required some configuration and white-lists to avoid blocking my own networks, the rest was very straightforward.

After installing and configuring pfSense, I shut down the VM, and added it, as a template, to my private catalog. That way, if in the future goofed it up, I could quickly deploy a replacement. This took a bite out of my storage limit but the VM is very small and is well worth the price.

Now that I had the firewall straightened out I could figure out what I wanted to do.

I decided that I wanted to clusterize my web and mail services, have a shell server for myself and my friends who still wanted access to a UNIX(like) server, and have a couple of servers for miscellaneous applications. Lastly, I wanted one set up as a rSyslog/monitoring server.

I had a lot of work to do.

Next week: “The Webcluster”

ipHouse has never outsourced any type of operations prior to this move. This bold move in efficiency makes ipHouse the premier facility manned and operated 24×7 by a private contractor.

Over the last few months, we have been negotiating with The Wonka Candy Company, directly with president Charlie Bucket for the upgrade to our operations. This might come as surprise to many of you, but Wonka Industries has been branching out for years into other services besides making some mind-blowing, awesome candy. (Though it must be noted, their expansion into day care facilities failed miserably.)

Among the changes to our data center, we will also be upgrading our on-site security measures, our fire suppression solution, and our emergency elevator escape system.

We have also negotiated our new off-site storage with TheCandyMountainSecureStorageServicesCompany(tm) (TCMSSSC). This is great news as it helps us achieve the elusive 5 nines Gobstopper rating for data centers.

Security

The new security measures are sugary sweet, and truly state of the art. We’ll not only be using the Oompa-Loompa workforce for the operations of the data center, but we’ve also contracted with Chocolate Security Services to supply us with their Oompa-Loompa Security Force agents to guard both the front and rear entrances.

We have also initiated changes to our security card system by moving away from easily bypassed proximity cards to Snozzberry Scented Access Cards (SSAC). Digital security systems are easily bypassed because of the single 1s and 0s used in binary computing, the SSAC, though, is purely analog allowing for an infinite combination of scents for authscentification.

Emergency

Our updates to the fire suppression system will be happening during the month of April, 2011 as we move away from the current dry pipe solution to the very first rollout of the Bubble Fire Suppression Solution (BFS) from Bucket Services, Inc. Instead of water (used in dry pipe), Halon (poisonous), or FM200 (where did the air go) uses in the past, the new BFS system actually uses bubbles filled with a secret gas to take the fire up and away from the servers, equipment, and personnel in the facility. Mr. Charlie Bucket says “This is by far, the best system to move fire away from expensive equipment and up into the cutting fans, where the fire will be chopped into smaller, more manageable pieces and mixed into Atomic Fireballs and Big Tex Jelly Beans.”

To handle any impossible emergency events in the data center, the Factory Airation Individual Lift (FAIL) subsidiary will be installing their patented Great Glass Elevator Emergency and Security Cylinders throughout the facility, including both entrances. These can be used by oompa loompas and humans alike to escape an unlikely flood from the new HVAC, which uses hundreds of thousands of gallons of chocolate to remove heat from the data center. They may also be used for security; when a human is determined to be a “bad egg” in the “veruca trap”, they can, and will, be shot into orbit.

Tickets

When customers are coming down to work on their network after these new measures are put into place, a lottery of golden tickets will be used to create some excitement. By working with our customers through multiple – multiple choice questionnaires, we have determined that this will be the most effective way to for their system admins to stop by the data center to patch and update their systems regularly. The winning Golden Tickets Winners (GTW) give the system admins an exclusive 5 minutes alone to wade into, or relax next to, our chocolate waterfall.

While nobody wants to have to come in to fix something broken, this lottery system is designed to add some excitement back into the process.

This is something that has been in the works for some time. If you currently have an access card issued in the past five years, you may already be a winner. Fold your access card LENGTHWISE in half so that the printed side ends up in the middle. You will feel a snap as the card’s contents are revealed.

If your card does not reveal a golden ticket, you are not a winner and will no longer be able to cross the new security perimeter to access your server. You are welcome to try again. Additional access cards are available individually or by the case. Contact your sales representative with any questions or to order.

As an added potential benefit; customers visiting our data center to work on their equipment might win a free candy bar, chocolate coated Windows Server 2008 R2 (the chocolate coating makes it go down easier), apple iPod (tastes like real apples, shaped like a real iPod), or even a full and exclusive tour of our data center by boat.

Finally, with all of these new changes, we will be rolling out a new logo for our Data Security Center Solution Service, seen just below.

Hard Shell Data Center Services

A gallery of pictures of a senior Oompa-Loompa doing an inspection of our current facility is also linked below.

Inverse Investigative Techniques

Reverse Inverse Investigative Techniques

Type Testing of Crash Cart Services

First Test of Upper Rack Security System

Oracle Hands Senior Oompa-Loompa Worker a Wire

Oompa-Loompa on Break

We wish to thank all of the Oompa-Loompas who let us take their photos as we continue to integrate their services into our company.

——

We live in a society that has mostly agreed on what is right and what is wrong.  We have coined the term, Common Values in order to build communities where we can agree on what is acceptable behaviour.  To enforce our safe communal living, we turned these common values into laws.  As a society, we have set up governments to enforce the laws (common values) and protect us from those who would harm us.

As members of society, I think we’re all in agreement that theft is a bad thing.  No matter what country you are from, it’s pretty much a given that stealing is frowned upon.  It’s a pretty common value, maybe even sacrosanct.

As an individual living in a society with common values, government is the enforcer of the laws and government prosecutes suspected thefts.  What is interesting is that if an individual is convicted of theft, they receive a punishment based on the severity of the crime, and that punishment is roughly agreed upon by the COMMUNITY.  As a defendant, you have rights of due process and would be able to defend yourself according to the law of the land.  Further, in this country a defendant is protected from the Government, as the Government must also follow the law of the land.

Let’s flip it around.  As a society, we have agreed that if you are the victim of a crime, you will take your grievance to the government and file a claim in court.  As a plaintiff, you also are protected by the laws of the land and due process.  This is how even a single individual can take on a corporation and win.  There is DUE PROCESS for each side, Plaintiff and Defendant that as a society, we have all agreed upon using.

The process of using the governmental courts to settle disputes is what our society has agreed upon for longer than I know without Googling it.   If you feel like an individual or a corporation or even the government has wronged you, you must take your claim to court.

So why are corporations looking to ISP’s to become the enforcers of copyright law? The Anti-Counterfeiting Trade Agreement is being manipulated into a Copyright enforcement act.  But isn’t the enforcement of violated law the job of Government?

If you feel your copyright has been violated, file a claim in court!

There is a push for ISP’s to become the monitor, the snitch, and the enforcer.  That’s not what we’re good at.  I hate to say this, but isn’t that the job of Government?

Lions and tigers and bears!  Oh My!

This is my 4th post to this blog and I am seeing a recurring theme.  I feel like the future of the Internet is at stake.  I don’t know what the outcome of this treaty will be, but I’m hoping that some smart people start looking at this issue RIGHT AWAY and make sure EVERYONE is required to follow due process.

Peace.

-Bil

One big issue at hand is, what happens to the data that you create when using the Internet.  “Data I create?  I don’t create any data when I’m on the Internet…do I?”  Yes, you do.

Currently when you do any of the following, you are likely creating data that can be tracked.

You create data…

• When you make searches at Google.
• When you look at movies at Netflix.
• When you check scores at ESPN.
• When you read customer reviews on Amazon.
• When you search for someone on Facebook.
• When you watch a video on Youtube.

All of these are innocuous, but together, they create a profile of you, and can reveal some very private data.

Let’s start with the first item:  Search terms.

Lets stipulate that the actual search term you use on a search engine is private data, similar to a request you make at the library or at a book store.   To follow on, it’s strongly possible that the results that are sent to your browser are private data.  Today, it requires a search warrant to see the contents of your computer hard drive so I can infer that the results from the search engine are private data.

“Whew, I’m safe, right?”

Nope.  In order to use the search engine, it’s possible that you’ve given “consent” to use the data you supplied and have waived any privacy rights you may have had. Further, the search results are logged before they are sent to you.  This creates a big gray area for data privacy that is not currently protected.  And from the content providers point of view, It’s NOT private data.

This goes for all type of data you send across the Internet.  The search requests you make, the stock quotes you review, the movies you download, the books you buy.  The list goes on and on.

“Wait a minute, why would someone even WANT this data?”

The motivation for companies to keep your privacy intact is two fold.  Penalties from regulatory bodies and the all important revenue.  If a company will face a penalty or lose revenue, they will likely keep your privacy intact.  But if they analyze the situation, they may conclude that selling the data is more financially beneficial than protecting your privacy.  This is not new to data that companies hold, but it’s new in context to the online world in which we live in.

Today, much online content is “free”, with only the hidden cost being you accept some loss of privacy.  We are so used to clicking “accept” that we’ve lost track of the value of what we are giving up.  It’s compounded by the good track records of the companies that are collecting data.   So far, their use of the data has not directly affected us, so who cares if someone knows what movies we like?  ”So really, nothing bad has happened so far, right?”

Right.  But that’s because the data is broken into chunks that are hard to combine.  I would guess that Travelocity and Orbitz and Expedia don’t share too much data because there is probably not an financial model that makes it profitable.  But let’s take another model and see what happens… Comcast has an on demand video solution, as does Apple and Netflix.  Should Netflix and Apple be worried that Comcast is going to start reviewing what their visitors are doing?  Does Comcast wants to have the online video business for themselves?

Luckily or not, each website you visit has only a piece of your online escapades.  The New York Post does not know what articles you read at the Washington Times.  Fidelity can’t see what stocks you traded at Etrade.  From the content providers point of view, you’re a statistic only when you visit them.

Which leads me to the next thought.  The ISPs’ point of view.

Above, I talked about data collection from each web site being possibly harmful.  That’s nothing.  Really.

The real loss of privacy will come when ISPs’ start collecting data on your browsing habits.  Think about it.  As much as Google knows what you’re doing when you visit Google, your ISP really knows what you’re doing at every website you visit.  And they can read your mail (like Google) and track your IM conversations and capture your VOIP calls… They know all that you do online and everything else about you.  SCARY.

Thankfully today, ISPs’ do only a little TRAFFIC monitoring.  ISPs’ legitimately monitor traffic to:

• To protect their revenue (keep customers online and happy).
• To protect their assets (network).
• To protect their customers (SPAM filtering).

Most ISPs’ don’t monitor the CONTENTS (data) of the traffic they manage except to comply with regulation and law enforcement.  Really, most monitoring is often “look at header info and discard”.  It’s important here for me to point out that any data collected by ipHouse is not held in order to create profiles of users.

Traditionally, ISPs’ have NOT monitored data because it was just too hard to do.   But that’s all changed.  Deep Packet Inspection technology has advanced to the point of being able to transparently evaluate traffic for specific patterns and usage without impacting the consumer experience.  This allows the ISP to deliver “tailor made” content to users.  Remember Travelocity not seeing Orbitz or Expedia data?  Forget that.  The ISP can now sell all travel related “traffic” from its subscribers to the highest bidder. Or bidders.

Deep Packet Inspection technology allows the management of traffic and/or data according to a set of policies that promote security or revenue or censorship or whatever.  The ISP sets the policy according to their desires.  “Really? My ISP can just monitor my data if they want to?”  Yes.  But there may be existing law that prevents the monitoring of data and that needs to be proven.

If the existing law is shown to not be applicable to ISPs’, it might make a lot of sense (and dollars) to monitor customer data.  But all things have a cost.  One anticipated cost is that spying wouldn’t be done just for profit.  How long would it be until ALL data is monitored and reviewed?  If ISPs’ monitor data, should they block data based on some policy for decency or obscenity? Who’s policy would that be?  Should ISPs’ be responsible for any and all security or ethical breaches (by whos standards?) that occur because of the data on their network?   Should ISPs’ send all suspicious activity to some authority for review?  Data monitoring could become mandatory.

So, should ISPs’ monitor thier customer data?  I say no. This is MY ethical position.  It’s ethically wrong to spy on people.   Further, I feel it is ethically wrong to profit from spying.  ISPs’ should NOT monitor data for profit or for government.

The power and beauty of the Internet is in its ability to bring people together across cultures, faiths and boundaries.  Once one group or government starts dictating “inappropriate” content, the Internet becomes simply a tool for that organization to push their own agenda and the “one world” quality of the Internet is lost.

Peace.

-Bil

Thoughts for comments:

• Privacy is not a technical issue and should not be addressed by ISPs’.
• ISPs’ should remain neutral to content of the data streams they manage.
• Our society should rethink privacy from a contextual integrity perspective.
• Online Privacy == Network Neutrality

Further Reading:

Paul Ohm : The Rise and Fall of Invasive ISP Surveillance

Daniel Solove : Understanding Privacy