NAT, for those who haven’t already opened a new tab on Google to look it up, is “Network Address Translation”. It’s a technology that maps between public internet addresses (such as 216.241.32.130, the IP for our web site www.forethought.net), and private addresses (such as 192.168.1.2). NAT is what prevented us from running out of IP addresses a long time ago, as there are only about 2.5 billion usable IP addresses, and far more than 2.5 billion devices on the Internet.

NAT translates between the internal addresses you use on your home, or your office network, and public addresses.

NAT is also used as a fireall technology as it effectively prevents any traffic from flowing past a router, that is not part of an established connection - generally one that you initiated.

So, NAT is a good thing but of course comes with a price. There are certain internet
protocols that NAT breaks, such as SIP for Voice over IP, FTP (file transfer), and
any number of others. One of these is obscure but often very important: ICMP Path MTU Discovery.

I know what you’re saying, “You’re killing me with these acronyms!” But please bear with me..

“MTU” is “Maximum Transmission Unit”. It’s the largest packet that can be sent over a particular link. For instance, the MTU on plain old Ethernet is generally 1500 bytes. If the two ends of a connection on the Internet try to send packets bigger than the MTU of a particular link, the packet could get thrown away. So Path MTU Discovery figures out a maximum packet size that can traverse the entire network. The computers on either end use that packet size and all is good!

Except that many NAT routers (most, in fact) break Path MTU Discovery, so they put in place workarounds. Except the workarounds don’t work when you have (drum roll please) Double NAT.

Are you still with me? Good!

Double NAT is what happens when you have one NAT translation behind another NAT translation. This is a case where two is not better than one.
Unfortunately this is becoming very common because it’s now almost impossible to buy a WiFi base station that does not have NAT in it. Many, in fact, have NAT and do not allow you to turn it off. So, if you take one of these and plug it into the back of your DSL modem (which is also doing NAT), you end up with Double-NAT.

Double-NAT breaks other things besides Path MTU Discovery, such as file sharing between a laptop on the Wifi and a desktop on the DSL router.

Below are some diagrams I’ve drawn to show the (bad) Double-NAT scenario, and two possible solutions to Double-NAT. I’ve also thrown one in to outline an “ideal” home network (which is to say one with a minimum of weird issues caused by poor assumptions made by consumer electronics vendors).

Double-NAT generally bites DSL providers, as Cable internet modems now typically are “dumb bridges”, meaning they pass through a public IP address and do not do firewall or NAT. DSL modems as typically provided with for example Qwest DSL or foreThought.net have NAT/firewall on by default.

You will see all kinds of odd behavior with Double-NAT. Some web sites may be slow. Some may not come up at all, or may come up sometimes but not others. You may be able to download certain emails but not certain other emails. It all depends on the size of packets generated by the endpoints, which can sometimes be somewhat random.

To solve this problem, you need to remove one of the NATs. You can remove the NAT in your DSL modem, or you can remove the NAT in your WiFi or other router. Which you can do will depend on your provider.

foreThought.net’s MACH DSL supports transparent bridging just like cable modems. So you can turn off NAT in our modem and leave it on on your wireless. Or, you can keep it on in our modem, and replace your Wifi Router with a Wifi Access Point (AP), which do not have NAT or firewall functions.

I think a lot of people probably suffer through slow Internet, thinking it’s just the way it is, or that their ISP is bad. Since switching providers can be a pain in the arse sometimes people just put up with what they perceive as bad service, because it’s slightly less painful than switching.

So the ISP usually gets the blame, and most don’t have the tools or expertise to properly diagnose “slow internet” problems. Indeed, there are so many things that can cause slow Internet that sometimes it can be extremely time-consuming and frustrating to resolve - if it ever is. The typical Internet user does not have time to spend hours on the phone on a support call, half the time to someone who barely speaks English.

Ultimately the user may switch. Sometimes their “problem” will be solved, because the Internet is fast! But that’s sort of like fixing a burned out light bulb by moving to a new house.

Our industry needs several things. First off, it needs much more sophisticated troubleshooting techniques. Calling tech support at most carriers is basically a game of reading off items from a checklist, most extremely basic and not helpful. This is why many large companies don’t understand the problem with outsourcing support to foreign countries with poor English skills - ”why, anyone can read ten questions off a checklist!”.

Well sometimes a checklist will get it (sometimes it really is just the power cord isn’t plugged in) but many times it will not. In the latter, now what? At some companies this is where you hit the brick wall. But with more sophisticated network testing tools, those agents could help the customer.

This is particlarly the case where the problem isn’t with the service, but with the user’s network. As home and small office networks become more sophisticated with multiple devices, wireless, streaming audio and video being used, the potential for problems and the impact of problems becomes more severe.

As much as we all might think of Internet service as a commodity, as a utility much like the water or power, the fact is that the Internet is fantastically more complex than the water or the power. When the water stops it’s because a pipe is plugged or the pressure has gone, and that’s about it. When the Internet breaks it can be for any of a thousand different reasons, including software problems on their PC. Because of the complexity and because we keep telling people Internet is a utility, oftentimes customers have a hard time understanding the demarc - that we are responsible up to the modem and the customer is responsible for everything past that.

There are a couple ways to solve this. One, we can get better at discussing the demarc, establish proper expectations around that demarc, and saying “not our problem”. Two, we can find a way to take responsibility over the home and office network, and make it our problem. After all, we’re the pros, we have the skills and resources to do the job right. A third way is to use an intermediate gopher like GeekSquad but I don’t think we should add PC’s and Internet to the trades that charge people exorbitant rates by the hour for fixing stuff that should just work in the first place.

What some regular folks have to say about Comcast.. note the hostility towards Comcast’s rate-limiting, bandwidth limits, and customer service.
http://www.engadget.com/2009/04/21/caption-contest-its-deadbeat-tastic/

Internet spam email is a big problem, there is no denying. There are many approaches to identifying and blocking spam, and none of them are foolproof. But there is also a clear difference between techniques - some are less reliable than others. Much less.

An outfit known as SORBS (www.sorbs.net) provides some useful spam-filtering services, some of which we use. They maintain several lists, but the most useful one is a list of “dynamic IP addresses”. Generally users on these IPs should be using their ISP’s email server. This is reasonable, and there are easy ways around it in cases where it’s a problem.

But SORBS also maintains a list of “known spam sources”. We do not use this one, because in our experience this list results in many false positives.

Indeed - we now find ourselves on this SORBS blacklist, all over a total of three (3!!) emails received by them over the past year. Near the end of January 2009, one of our customer email accounts was compromised because it had a weak password, and a lot of spam was sent from it. We fixed the problem pretty quickly, and were able to be removed from most blacklists very quickly. Most administrators are reasonable, and understand that things like this happen.

Not so SORBS. SORBS apparently is run by a cadre of irrational fanatics. Again, over three emails, they refuse to remove us from their list unless we 1) filter outgoing emails by content, or 2) pay them a fine.

A fine? What? $550 is the price they’re asking to be removed from their list. This is apparently quite a revenue source for them. In some countries, it would be called extortion, or racketeering.

They further refuse to provide any help to us whatever, in identifying spam, in letting us know before we are blocked that there is a problem. Their philosophy is “block immediately, provide no information that could help you stop the spam, and pay us to get off the list.”

This is patently irrational and unproductive. All other major spam blocking efforts provide feedback mechanisms, so that email providers like foreThought.net can be proactive in eliminating spam from compromised customers. SORBS refusal to help in any way does not help stop spam - all it does it is make them feel powerful and cause a lot of people a lot of headaches.

We refuse to abide by either of their criteria for being removed from their list. First off, we refuse to filter outbound email by content. Content filtering, for example dropping emails that have the words “bank manager” in them, is extremely unreliable and causes many false positives. Were we to implement even the best of these approaches, our customers would have great ongoing difficulties sending emails, you would have to constantly be on guard not to put key words in your emails.

Second, we’re not going to pay any money whatever to a group of anonymous and unaccountable people who for all we know, are simply lying about having received spam from us. How can we know? They won’t share any information with us.

As a long-time member of the Internet community, I highly recommend that noone use the SORBS “known spam sources” list. SORBS attitude, arrogance and unwillingless to be partners in fighting spam make this list extremely unreliable.

There’s been a lot of talk in the news lately about the “Network Neutrality” principle.

As usual with anything in politics, the term means different things to different people. So I’ll define it here: the most common aspect of the principle is that networks (such as your friendly neighborhood internet provider) should provide unfettered, unfiltered access to the Internet.

I agree with this principle. Once providers get into the business of limiting content, we don’t have an Internet - we’d have multiple versions of the Internet and free speech would be at risk.

However, there are circumstances where a provider must manage traffic on their networks.

All Internet access is shared. The Internet is inexpensive because all of us are sharing its infrastructure, and this works because not all of us are using the Internet at the same time. Larger providers may see 60 to 1 effective oversubscription; i.e. out of 60 customers only 1 may be using the service.

Some folks think oversubscription is inherently bad; but really, without it, the Internet as we know it wouldn’t exist. You have to share bandwidth in order to have the potential for $20 home broadband service.

In the early days of the Internet some providers poorly managed their oversubscription, resulting in congestion and poor quality service for their customers. But now in this day of “fiber glut” bandwidth to the rest of the world is cheap and readily available - at least, it is in large urban areas such as Denver. That makes it easy for broadband service providers such as foreThought.net to ensure there is plenty of excess bandwidth, ensuring all customers will have full speed access to their connections when they want it.
But, there are many ISPs in areas that do not have ready access to that kind of bandwidth. There are in fact thousands of ISPs in rural America for whom internet backbone connections are very expensive and hard to get. An ISP in Alaska may get its Internet connection via a T1 over Satellite, for example, and that may be the only option.

Such providers must manage bandwidth carefully in order to provide broadband service of any kind to their subscribers. It is here that some Network Neutrality advocates overreach, and want to propose regulations that would prevent such rural providers from restricting subcriber use of the service that would cause problems for other subscribers. I am fortunate that I do not have to make such choices.

Now we get to Comcast. Comcast (and presumably other cable broadband subscribers) are declaring their need to manage their bandwidth, and have filtered and/or rate-limited traffic from applications such as BitTorrent. Aside from Comcast’s historical behavior of not telling its subscribers they were subject to such filtering and rate-limiting, one wonders why a company such as Comcast that constantly boasts of its “amazing speeds” would need to enforce such limits.

The problem could stem from Comcast simply being cheap, but at the scale they’re at, Internet bandwidth is almost free for them. Rather, the problem likely stems from the inherent nature of Internet-over-cable.

Cable internet services work according to the “DOCSIS” technical standard. This standard basically specifies that Internet service is transmitted on a cable TV cable, by allocating “channels” of bandwidth. They basically take (for instance) channel 2 and instead of using it to send video, they send Internet data. So far so good.

Cable is inherently a broadcast medium. It was designed to efficiently send the exact same TV signal to thousands of houses at the same time. It’s very good at that. What this means for Internet, however, is that the one “internet” channel we referred to earlier is shared by all the all the houses and businesses that share the same coaxial cable.

Nobody is really sure exactly how many houses might share a cable node; Comcast does not release such information. However, in TV services the larger the number the cheaper it is for Comcast. We may surmise that neighborhoods of hundreds, maybe even thousands, of users will share a cable.

Thus, hundreds or thousands of users will share the same Internet bandwidth.

Using the DOCSIS Wikipedia Article as a reference, we can make a few educated guesses. One channel in DOCSIS is about 42Mbps. We don’t know how many users may share a single channel such as this, but at speeds Comcast is now selling it would only take three such subscribers to degrade other user’s performance.

There are technological improvements possible with the cable architecture, but telco-style central office based services are still better right now.

In a telco-style CO deployment such as foreThought.net uses, we run a gigabit or 10 gigabit into our central node, and every customer has a dedicated DSL or Ethernet line from their house or business, to the CO. So it would take hundreds of customers maxing out their service at the same time to cause a congestion issue for us, and in practice such overuse has never occurred.
So, we see why Comcast is opposed to Network Neutrality on this technical basis. But I wonder if they are opposed to it for other reasons as well. They are on the verge of losing their monopolies on television programming. Video services are going to move to being delivered over the Internet; even with DOCSIS 3.0 and the additional bandwidth it promises, true video on demand will break the cable network. DSL-based broadband networks were made for this, however, since every subscriber has its own dedicated line.

Business users, also, who have Internet bandwidth needs far greater than homes, will put great stress on cable networks.

It seems Comcast has a perverse incentive to discourage its users from actually using their service. Fortunately, this is not a problem foreThought.net faces.

There’s been a lot of talk, and frankly fear-mongering, in the press about the current recession. It’s “the biggest financial disaster since the 30s” by some accounts, although it’s hard to reconcile that with inconsistent economic data (such as the fact of contuining GDP growth through the first half of 2008).

With all the comparisons to the Great Depression, it’s worthwhile to discuss briefly my take on it. I am an incessant optimist, but I just have to respond to all the negativity.
First off, recessions are a normal part of the business cycle. The business cycle is: make investment, operate business, depreciate capital, close business.  You invest capital in order to be able to run a business, produce products, be more efficient. At some point, that capital must be depreciated, because the factory machines are worn out and need replaced, or the product is no longer relevant in the market. At that point you either reinvest and start the cycle over, or you quit and liquidate.
Business cycles occur all the time in microcosm; only when a significant number end at the same time do we have a recession. And in some cases, like this one, an unusual bubble can have ripple effects through the economy. Irrationality in the real-estate market coupled with home-equity-fueled consumer spending were the instigators of this current downturn - a lot of the economy was being driven by the out of control upward spiral of home prices, instead of by sound business investment principles.
But this is an opportunity. Capital has been withdrawn, homes and businesses liquidated, assets available on the market for pennies on the dollar in some cases. Creative entrepreneurs will innovate, and investors who have been banking on real-estate as the best place to put their money, will be looking for new homes for their investment capital. Recessions are an opportunity to do things better. Those who have better ways, will be successful. Those who are stuck with old business models will fail. Recessions are a big economic introspection, and a free market always comes out faster, better and stronger.
foreThought.net is accelerating its expansion in 2009 - this is not a time to be timid, this is a time for us to really get our products out there. We can provide more bandwidth for less money than any of our competition, and other businesses in 2009 are going to be seeking exactly that: cutting their budgets or getting more for the same money. Either way, we can help.
And we’re not alone, there are many other companies doing the same thing we’re doing, looking at this recession as an opportunity.

I believe in free enterprise and free markets, and I believe in the ability of human beings to solve the problems that face us. We’ve been through worse, much worse, before and we will get out of this. foreThought.net is hopefully putting in our part to help make business more efficient. Working together, we’ll get it done.
Merry Christmas,

Jawaid Bazyar

Finally, the day came!

Our conduit was ready. The fiber was here. All that remained was putting it in the ground. How do you do that? What all is involved when you see those guys in the orange vests and hard-hats in the middle of the street, crawling in and out of manholes?

A lot, it turns out. The first step is “pulling” a permit with the city. To get a permit, you need construction experience, a performance bond, and a traffic plan. The traffic plan details where work is to be done, how you are going to block the street, and how you will re-route traffic around the blockage. At least in Denver, it seems pretty straightforward.

On the day, the sign company brought tons of orange road pylons, and road-signs announcing the blockage. It was pretty thrilling, the idea of being responsible for blocking a street and re-routing traffic.

Once all the road markers were up, it was time to start opening manholes! A special tool is used, a type of crowbar with a hook on the end that lets you snag a lip on the bottom of the manhole lid.

What’s in a manhole? Typically a manhole is an opening to a vault. Sometimes as small as 4 feet square, sometimes as large as 20 feet by 40 feet, is a subterranean vault with concrete walls. Holes are drilled through the walls to allow the entry and exit of conduit, in which fiber or other cable is placed. Having vaults and conduit along the path allows for much easier repair of underground cable. If you simply buried the cable, you would have to potentially dig it up every time something went wrong with it. In this picture you can see some of the ducts coming out of the wall:

Sometimes manholes collect noxious gasses, methane and other such things that are not as good for you as breathing oxygen. Thus, positive air pressure is delivered into the manhole using a high-power fan and flexible ductwork, to prevent problems.

Once we identified which conduit was set aside for our use, we put the fiber on a spool and prepared to start pulling it into the manhole.

Since our run was relatively short and the fiber count (24 strand) relatively low, our spool is pretty small. Sometimes the spools are 10 feet across.

The vault this manhole opened into was one of the large ones. It is directly adjacent to the AT&T building downtown, and actually has a door from the building’s basement that opens into the vault. We didn’t know that at the time, so by going in through the manhole we were doing it the hard way.

Inside this vault was a literal ton of conduit, innerduct (flexible conduit inside the conduit), and fiber bundles.

The gear on the left edge is used to pressurize copper cable bundles, to keep water out of them. Remember, all of this is underground. The vault extended 20 t0 25 feet down, basically two stories under the street.

In this particular vault, our cable needed to come in and go back out to another vault. Here is our fiber cable (black), coming out of the blue conduit in the middle:

Now once the fiber was in end to end, we further had to run it upstairs inside our building. This was the hardest part, as the conduit inside our building was small for the cable. But we got it done.

The final step, is splicing this cable with the one we put in last time, joining the two cables together. Also, terminating the fiber, or putting ends on it.

The saga continues…

Hi all,

I have been wanting for some time to improve communication with our customers - and in addition to trying to get newsletters published more regularly, thought that I would experiment with blogging as a way to get timely information out to you without necessarily filling your mailboxes with stuff you might not want to read.

So, welcome!

My first series of posts will be in regards to our first-ever underground fiber build.

Jawaid Bazyar, President