The bandwidth needs of our customers are continuing to increase - and the pace is accelerating. Due to skyrocketing usage of video on demand, Netflix has become the biggest single source of Internet traffic. Business applications such as Cloud Hosting, offsite data backup, electronic medical records and video conferencing are similarly driving bandwidth needs.

2011 has been a year of unprecedented bandwidth demands requiring fiber-optic delivery. I think 2012 will probably quadruple our 2011 activity.

Due to our relationship with Zayo Bandwidth, FORETHOUGHT.net is uniquely positioned to deliver fiber-based services in the Denver Metro area. Our Denver fiber ring is over 100 miles, and cuts through the heart of downtown, the Denver Tech center, Inverness, Parker and Centennial.

To prepare for the fiber tsunami, we have been training to gain the skills needed to keep up. We are now doing our own in-building fiber demarc extensions, reducing costs and slashing delivery times. We have established local banking relationships to supply the capital needed to do expensive outside plant and building entrance work. And we are starting to deploy our own metro MPLS network to tie it all together and provide seamless integration of voice, video, Internet and Transparent LAN Services to our customers metro-wide.

It is a very exciting time to be in telecommunications - what has been promised for so long is finally starting to come to fruition. Ethernet anywhere - with massive (10Gbps) bandwidth available.

The Electronic Frontiers Foundation (EFF) is making hay about HR 1981 - a bill that adds a money laundering crime to the books. They are claiming that this bill “Orders Internet Companies to Spy On You.”

https://secure.eff.org/site/Advocacy?cmd=display&page=UserAction&id=497

Well I don’t know who is running the EFF these days, but this is bunk. This is the relevant clause from the proposed legislation (which the EFF does not bother to link to).

`(h) Retention of Certain Records- A provider of an electronic communication service or remote computing service shall retain for a period of at least 18 months the temporarily assigned network addresses the service assigns to each account, unless that address is transmitted by radio communication (as defined in section 3 of the Communications Act of 1934).’.

http://thomas.loc.gov/cgi-bin/query/z?c112:H.R.1981:

I’ve run an ISP for 16 years and so have some insight as to what this means.

As you may know, the Internet uses numeric “IP Addresses” to define the endpoints of a communication. When most users log on to the Internet, they share a big pool of those IP Addresses with other users - you’re not all online at the same time, so this helps to share the scarce IP Address resources. For purposes of billing, security, and (yes) law enforcement, all Internet Providers (ISPs) keep a history of what user had what IP address at a given time.

Now, it’s important to understand the limits of this information. This information exists at your ISP only. It would be given to the government only under a court order such as a subpoena. The government does not automatically have access to this. My company has a strict privacy policy, as do most ISPs. If an ISP did give this information without a court order, they could be subject to prosecution under privacy laws, or sued for violation of their own user contracts.

The other thing that is important to understand, is that this does not tell the government what web sites you have been visiting. That information exists only in two places: on your computer (in your web browser history), and on the access logs of each individual web site you visit - that information is decentralized and distributed across thousands of computers on the Internet. The government would have to issue subpoenas to every web site in the world to try to reconstruct it. I can assure you that is not practical, and it has never been done. Thus EFF’s claim that this law “lets the government force ISPs to spy on you” is laughable, and grossly inaccurate.

A requirement that ISPs keep this information for 18 months facilitates law enforcement and to my mind is perfectly reasonable. Given the facts above, it can be time consuming for law enforcement to track down actual criminal activity on the Internet. 18 months is not burdensome. (Some previous versions of this same requirement in other legislation were for 5 years).

My conclusion is that the EFF has been taken over by folks who happily hyperventilate and attempt to scare people into doing their bidding. It’s sad.

So, don’t contribute to the scare!

Now, there may be valid reasons to oppose this bill - the bill creates a new Federal crime of “Financial facilitation of access to child pornography”, with hooks in the money laundering code. This may or may not be a good idea. But let’s at least oppose the bill for the right reasons - for real reasons - and not unfounded scare-tactics.

Ok, everyone who loves pages of legalese and tricky fine print to subscribe to what ought to be basic services, raise your hand.

Right, you guys who are the in-house counsel and marketing dweebs for Qwest and Comcast, you can leave the room now, we’re not talking to you.

When I say “fine print”, I’m talking everything that makes the great deal you’ve just been promised, not so great. For example: you get a $99/mo price on a TV/internet/phone triple-play, which is great. But then quietly, after 6 months, the price quietly goes to $150.00/mo (that was in the fine print). And since you’re an existing subscriber, you can’t take advantage of promotions (more fine print). And you threaten to cancel and maybe they back down and let you do it, or you find a way to get around the restrictions by ordering cable TV in your dog’s name. We’ve all played these games.

But what kind of customer relationship is that? What kind of way is that to run a company? We promise the world *.

*- What we really mean is, unless, if, and in case.

Building in from day one to your customer relationship a scenario where most of them are guaranteed to get pissed off, does not seem to me to be a strategy for long-term relationships.
foreThought.net’s current customers have had service with us an average of over 7 years (we’ve been in business for 16 years). We would not have that kind of customer loyalty if we randomly and quietly increased prices as our “reward” for customer loyalty.

With the mass-market advent of Cloud Computing, we should take this time to evaluate data center needs.

I’m going to call a Cloud-focused datacenter a Nimbus:

Cumulonimbus (Cb) is a low to middle level cloud with considerable vertical development (family D2) that is tall, dense, and involved in thunderstorms and other inclement weather. Cumulonimbus originates from Latin: Cumulus “accumulated” and nimbus “rain”. It is a result of atmospheric instability. These clouds can form alone, in clusters, or along a cold front in a squallline. They create lightning through the heart of the cloud. Cumulonimbus clouds form from cumulus clouds (namely fromcumulus congestus) and can further develop into a supercell, a severe thunderstorm with special features.

And they contain a great deal of power. I think it suits!

So let’s review the key aspects of Cloud Computing and see how it impacts the design of a Nimbus.

High Density

This is an existing trend, amplified greatly. Dell has inexpensive servers that can put 960 processor cores in a single rack - that’s 2.1 Terahertz of processing capacity, which could serve 10,000 hosted desktops or more. Ten such racks in a Nimbus could provide all the computing needs for entire cities.

Fire Suppression

A system that shuts down the whole Nimbus, such as typical FM200 systems, is unworkable. Instead, utilize per-cabinet fire sensors with ability to alarm and remotely power down a rack. VMs in the rack can be hot-migrated to other racks in seconds, before the shutdown.

Access and physical security

In retail businesses, it’s all about the real estate - Location, Location, Location. But in Cloud Computing, this equation is altered dramatically.
There is never a need for end-users to physically visit a Nimbus. This dramatically alters security considerations: cages, key cards every time you turn around, special locking cabinets, video cameras, on-site security guards, little booties to cover your shoes, biometric sensors - these are all very expensive components that are simply not necessary in a Nimbus.
We can locate Nimbi without consideration for convenience of visitors - parking, physical proximity to customer, etc. A Nimbus could be 60 miles outside a major metro area. Anywhere, really, so long as it is proximate to a fiber ring.
We can put Nimbi close to the end-users. A Colorado ring could provide a Nimbus in Durango serving Durango, with hot backups and disaster recovery in Denver. Or vice-versa.

Power

Cloud Computing, through virtualization and consolidation of computational tasks, can be radically more efficient. While power density is proportional to the computational density, and so would be very high in a Nimbus, total power use is lower and more predictable.

Power doesn’t have to be as reliable in a Nimbus as in a traditional data center. If there is a power failure, the UPS can trigger a migration of VMs to backup data centers, and have this complete before the power fails.

Size

As a real estate business, traditional colocation data centers have significant economies of scale given some of the infrastructure requirements, security, so some existing data centers are enormous. Also, existing data centers basically sell space by the square foot. So the more square feet, the better. Given the density potential of a Nimbus, however, there is no need to have huge data centers. There is actually significant benefit to geographically dispersing Cloud Computing infrastructure among a large number of small (under 1000 ft) Nimbi.
We can spread Nimbi in areas with inexpensive real estate and cheap power. (We can, in fact, site Nimbi in areas where they can self-generate most of their power). Many traditional data centers have a high proportion of wasted space. We reduce rent, by not paying for real estate that doesn’t directly generate revenue.

Redundancy and Disaster Recovery

This architecture for Nimbi provides a high level of diversity for disaster recovery.

Single massive datacenters can and do fail — EV1Servers in Texas had the best of the best infrastructure, but when their transformer exploded, thousands of customers were down for 3 days.

A highly diverse network of Nimbi would provide rock star redundancy and reliability without any special application awareness required.

Summary

A network of Nimbi - small powerful computational nodes - can provide the computational services the market needs while minimizing cost and waste.

“Cloud Computing” or “Utility Computing” are among the biggest buzzwords in the technology industry right now. But what do they mean?

FORETHOUGHT.net has developed its vision of Cloud Computing.

Like your electricity which comes from numerous power generators connected to the “Grid” (i.e., the power distribution network), cloud computing uses the Internet to connect you to computing nodes. These computing nodes could be anywhere in the world - or, even more likely - distributed around the world.

One of the Cloud’s enabling technologies is virtualization. Server virtualization technologies such as VMware and XenServer enable virtual computer environments to be independent of hardware. In fact, a virtual server can be migrated - live, and without disruption - between different physical servers. Your server’s “brain” could move from Los Angeles to New York in seconds, with no perceivable down time to you.

This key technology allows vastly improved reliability, uptime, and data security than previous computing models.

Virtualization also allows you to instantly add RAM or Disk storage to your server, often without even having to reboot your virtual server.

Gone are the days of downtime due to a hard drive failure, or a roof leak shutting down your data closet, or painful, expensive upgrades between operating system versions. Dedicate a virtual server to each of your.

The second key component of Cloud Computing is a shift in the software industry, away from expensive one-time license fees, and to small fixed monthly fees for software rental. Indeed, virtually all Microsoft software is now available to rent: the full Microsoft Office suite is available for $15.00 per month per user, or less, and this gives one access to the latest, greatest versions all the time. Why pay thousands of dollars per user every 3 years when you can simply rent it?

With the advent of Cloud Computing, there is no more reason for you to host your own servers in your office than there is for you to host your own web site, or your own phone system. These are all simply applications which can be delivered to you over the Internet.

And finally, with desktop virtualization technology, computing for individual users can be hosted in a professional data center instead of on a desk. By hosting desktops, we solve many traditional IT challenges such as user data backups, user security, remote access, downtime, and the expense of rolling out new computers to your whole organization every 3 years.

For one view into our vision, imagine the following capabilities and benefits of our Cloud services to your organization:

* Affordable, Fortune-500, Enterprise-Grade IT Services for any size organization
* Scale cost effectively and with Fortune-500 Reliability
* Never outgrow your IT infrastructure again. Most services can be scaled, instantly, online with no disruption.
* New services and servers can be deployed in seconds
* No more waiting for new hardware to ship. No more expense, hassle, disruption of your office for upgrades. Moving to a new office is a snap.
* No more lost productivity due to server failure: all hosted services are fault tolerant. In the event of server failure, service is switched to backups in seconds with little or no disruption.
* If the Internet goes down, you can be running again in seconds using a backup 4G, DSL, or Cable connection.
* A’la Carte - use only the services you need.
* Stop buying and throwing away computers every few years. Turn old computers into high-performance thin clients, forever.
* Expert Management available for all services

We can put almost every IT component into the cloud: mail servers, file servers, desktops, web/ftp/application servers, firewall and VPN. All backed up, with instant failover and fault tolerance, and with inexpensive software rental that lets you immediately scale your IT costs along with your headcount.

Our plan is to bundle our Cloud platform with our Metro Ethernet services, providing unique capabilities compared to pure “over the top” Cloud services.

As IT professionals, what are some of the challenges you see in deploying Cloud Computing to your users?

Term contracts are a part of the telecom landscape. There’s no way to avoid them, really - it’s very capital-intensive to install and turn up new telecommunications services, so companies like FORETHOUGHT.net require a minimum term in order to make sure we can get that investment back, and make a profit.

When our contracts expire, they go month-to-month.

But there are still telecom companies out there that have contract clauses that automatically renew your service agreement for a year or even for the original term, possibly up to three years, unless you cancel within a narrow time window. These clauses are known as “auto-renewal” clauses, though I’ve heard of them as “greenfield” clauses too.

These terms are purely for the carrier’s benefit, and what they do is put incredible time pressure on you, the customer, to quickly make a decision. Many of these auto-renew clauses only give you a 30-day window. You can’t cancel before, and you can’t cancel afterward! It’s like having a high-pressure salesman baked into the agreement. And it’s tricky. It’s usually buried in the fine-print of the contract, and the sales reps never point it out. It’s in their interest to have you virtually locked in forever, except for small windows.

Something these companies don’t like you to know, however, is that most of them will strike those clauses if you request it.

Although sometimes it depends on the sales rep. I recently had to argue extensively with one carrier who tried to stick us with an auto-renew clause. I said, “I won’t sign this, all of our other carriers have agreed to remove auto-renew language, I need you to as well.”, and I got back a slew of the most ridiculous rationalizations I’ve ever heard, for why it was in my interest to have the auto-renew. Well, I stood my ground and took it to the sales manager, and got it removed.

So, be wary! Read your telecom agreements carefully! And don’t put up with any guff. Demand that your contract go month-to-month when its term is up.

(FORETHOUGHT.net has never, and never will, use ‘auto-renew’ language. Once the term is over we will earn your business anew, every single month).

NAT, for those who haven’t already opened a new tab on Google to look it up, is “Network Address Translation”. It’s a technology that maps between public internet addresses (such as 216.241.32.130, the IP for our web site www.forethought.net), and private addresses (such as 192.168.1.2). NAT is what prevented us from running out of IP addresses a long time ago, as there are only about 2.5 billion usable IP addresses, and far more than 2.5 billion devices on the Internet.

NAT translates between the internal addresses you use on your home, or your office network, and public addresses.

NAT is also used as a fireall technology as it effectively prevents any traffic from flowing past a router, that is not part of an established connection - generally one that you initiated.

So, NAT is a good thing but of course comes with a price. There are certain internet
protocols that NAT breaks, such as SIP for Voice over IP, FTP (file transfer), and
any number of others. One of these is obscure but often very important: ICMP Path MTU Discovery.

I know what you’re saying, “You’re killing me with these acronyms!” But please bear with me..

“MTU” is “Maximum Transmission Unit”. It’s the largest packet that can be sent over a particular link. For instance, the MTU on plain old Ethernet is generally 1500 bytes. If the two ends of a connection on the Internet try to send packets bigger than the MTU of a particular link, the packet could get thrown away. So Path MTU Discovery figures out a maximum packet size that can traverse the entire network. The computers on either end use that packet size and all is good!

Except that many NAT routers (most, in fact) break Path MTU Discovery, so they put in place workarounds. Except the workarounds don’t work when you have (drum roll please) Double NAT.

Are you still with me? Good!

Double NAT is what happens when you have one NAT translation behind another NAT translation. This is a case where two is not better than one.
Unfortunately this is becoming very common because it’s now almost impossible to buy a WiFi base station that does not have NAT in it. Many, in fact, have NAT and do not allow you to turn it off. So, if you take one of these and plug it into the back of your DSL modem (which is also doing NAT), you end up with Double-NAT.

Double-NAT breaks other things besides Path MTU Discovery, such as file sharing between a laptop on the Wifi and a desktop on the DSL router.

Below are some diagrams I’ve drawn to show the (bad) Double-NAT scenario, and two possible solutions to Double-NAT. I’ve also thrown one in to outline an “ideal” home network (which is to say one with a minimum of weird issues caused by poor assumptions made by consumer electronics vendors).

Double-NAT generally bites DSL providers, as Cable internet modems now typically are “dumb bridges”, meaning they pass through a public IP address and do not do firewall or NAT. DSL modems as typically provided with for example Qwest DSL or foreThought.net have NAT/firewall on by default.

You will see all kinds of odd behavior with Double-NAT. Some web sites may be slow. Some may not come up at all, or may come up sometimes but not others. You may be able to download certain emails but not certain other emails. It all depends on the size of packets generated by the endpoints, which can sometimes be somewhat random.

To solve this problem, you need to remove one of the NATs. You can remove the NAT in your DSL modem, or you can remove the NAT in your WiFi or other router. Which you can do will depend on your provider.

foreThought.net’s MACH DSL supports transparent bridging just like cable modems. So you can turn off NAT in our modem and leave it on on your wireless. Or, you can keep it on in our modem, and replace your Wifi Router with a Wifi Access Point (AP), which do not have NAT or firewall functions.

I think a lot of people probably suffer through slow Internet, thinking it’s just the way it is, or that their ISP is bad. Since switching providers can be a pain in the arse sometimes people just put up with what they perceive as bad service, because it’s slightly less painful than switching.

So the ISP usually gets the blame, and most don’t have the tools or expertise to properly diagnose “slow internet” problems. Indeed, there are so many things that can cause slow Internet that sometimes it can be extremely time-consuming and frustrating to resolve - if it ever is. The typical Internet user does not have time to spend hours on the phone on a support call, half the time to someone who barely speaks English.

Ultimately the user may switch. Sometimes their “problem” will be solved, because the Internet is fast! But that’s sort of like fixing a burned out light bulb by moving to a new house.

Our industry needs several things. First off, it needs much more sophisticated troubleshooting techniques. Calling tech support at most carriers is basically a game of reading off items from a checklist, most extremely basic and not helpful. This is why many large companies don’t understand the problem with outsourcing support to foreign countries with poor English skills - ”why, anyone can read ten questions off a checklist!”.

Well sometimes a checklist will get it (sometimes it really is just the power cord isn’t plugged in) but many times it will not. In the latter, now what? At some companies this is where you hit the brick wall. But with more sophisticated network testing tools, those agents could help the customer.

This is particlarly the case where the problem isn’t with the service, but with the user’s network. As home and small office networks become more sophisticated with multiple devices, wireless, streaming audio and video being used, the potential for problems and the impact of problems becomes more severe.

As much as we all might think of Internet service as a commodity, as a utility much like the water or power, the fact is that the Internet is fantastically more complex than the water or the power. When the water stops it’s because a pipe is plugged or the pressure has gone, and that’s about it. When the Internet breaks it can be for any of a thousand different reasons, including software problems on their PC. Because of the complexity and because we keep telling people Internet is a utility, oftentimes customers have a hard time understanding the demarc - that we are responsible up to the modem and the customer is responsible for everything past that.

There are a couple ways to solve this. One, we can get better at discussing the demarc, establish proper expectations around that demarc, and saying “not our problem”. Two, we can find a way to take responsibility over the home and office network, and make it our problem. After all, we’re the pros, we have the skills and resources to do the job right. A third way is to use an intermediate gopher like GeekSquad but I don’t think we should add PC’s and Internet to the trades that charge people exorbitant rates by the hour for fixing stuff that should just work in the first place.

What some regular folks have to say about Comcast.. note the hostility towards Comcast’s rate-limiting, bandwidth limits, and customer service.
http://www.engadget.com/2009/04/21/caption-contest-its-deadbeat-tastic/

Internet spam email is a big problem, there is no denying. There are many approaches to identifying and blocking spam, and none of them are foolproof. But there is also a clear difference between techniques - some are less reliable than others. Much less.

An outfit known as SORBS (www.sorbs.net) provides some useful spam-filtering services, some of which we use. They maintain several lists, but the most useful one is a list of “dynamic IP addresses”. Generally users on these IPs should be using their ISP’s email server. This is reasonable, and there are easy ways around it in cases where it’s a problem.

But SORBS also maintains a list of “known spam sources”. We do not use this one, because in our experience this list results in many false positives.

Indeed - we now find ourselves on this SORBS blacklist, all over a total of three (3!!) emails received by them over the past year. Near the end of January 2009, one of our customer email accounts was compromised because it had a weak password, and a lot of spam was sent from it. We fixed the problem pretty quickly, and were able to be removed from most blacklists very quickly. Most administrators are reasonable, and understand that things like this happen.

Not so SORBS. SORBS apparently is run by a cadre of irrational fanatics. Again, over three emails, they refuse to remove us from their list unless we 1) filter outgoing emails by content, or 2) pay them a fine.

A fine? What? $550 is the price they’re asking to be removed from their list. This is apparently quite a revenue source for them. In some countries, it would be called extortion, or racketeering.

They further refuse to provide any help to us whatever, in identifying spam, in letting us know before we are blocked that there is a problem. Their philosophy is “block immediately, provide no information that could help you stop the spam, and pay us to get off the list.”

This is patently irrational and unproductive. All other major spam blocking efforts provide feedback mechanisms, so that email providers like foreThought.net can be proactive in eliminating spam from compromised customers. SORBS refusal to help in any way does not help stop spam - all it does it is make them feel powerful and cause a lot of people a lot of headaches.

We refuse to abide by either of their criteria for being removed from their list. First off, we refuse to filter outbound email by content. Content filtering, for example dropping emails that have the words “bank manager” in them, is extremely unreliable and causes many false positives. Were we to implement even the best of these approaches, our customers would have great ongoing difficulties sending emails, you would have to constantly be on guard not to put key words in your emails.

Second, we’re not going to pay any money whatever to a group of anonymous and unaccountable people who for all we know, are simply lying about having received spam from us. How can we know? They won’t share any information with us.

As a long-time member of the Internet community, I highly recommend that noone use the SORBS “known spam sources” list. SORBS attitude, arrogance and unwillingless to be partners in fighting spam make this list extremely unreliable.