Wicked 3sty for Something

Kids...Don't Sniff Internet Glue...And if you are going to fail, fail open!

I am a little surprised by the up swell of my students interest in network sniffing. I covered a WireShark lesson in a previous post. WireShark is an excellent packet sniffer, but what do you do if you are in a switched environment?

The main difference between a switch and a hub, is that a hub broadcasts its traffic to all computers connected to that hub. A switch does not. A hub emulates a shared medium, such as the old 10-Base5 "thicknet" ethernet. In the old days, before  cat 5 cabling , a single  strand of coaxial cablle (cable TV cable) was run throughout a building or a floor and provided connectivity via "vampire clips" or "alligator clips". The clips would literally bite through the shielding on the coax, touch the center copper cable and sense and transmit MAC frames. Ethernet was a "shared medium". Which is a nice way to say that all computers had to play nicely or the wire would become jammed. The mechanism to prevent this is called Carrier Sense Multiple Access/Collision Detection (CSMA/CD).

Hubs were implemented to mimic the shared nature of the one coaxial wire. A hub re-broadcasts all traffic to all ports. A packet sniffer such as WireShark running on a hub will have access to all traffic sent to or from the hub. When WireShark is attached to a switch, it only has access to the broadcast and possibly multicast traffic. This is the significant benefit of a switched environment. Your data has a straight path to the computer you are sending it to. But, it is a detriment if you are trying to do network analysis or learn about networking protocols through packet sniffing.

Most modern day enterprise switches allow an administer to enable a port to repeat all switched traffic to that port. This would allow  an administrator to run a packet sniffer on a computer attached to that port, and observe all the network traffic coming or going from that switch.  This is all well and good, but it requires an enterprise switch. My home network is a pretty sophisticated fast gigabit network. All my machines are switched via Netgear GS108 switches. They are pretty sophisticated, but they are not enterprise grade. I cannot enable a port to act as a repeater on these switches. That would require a much more expensive switch.

Well what can you do? There are essentially two techniques. You can flood the switch with MAC addresses till it fails, and "fails open". Or you can broadcast your computer running the sniffer as the router for the network, and then intercept (perpetrate a man in the middle attack) packets and then transfer them on behalf of the sending computer. Software tools to implement both techniques are available from the Unix utility package dsniff. Both techniques can be used for nefarious and or positive purposes.

Network sniffing in my life has always been a tool to explore how computers communicate and how individual protocols work. I firmly believe that deconstructing an IP packet in WireShark is a far better learning experience than reading a packet diagram in a book. With that said, NEVER...NEVER EVER sniff at work. Network sniffing can be easily discovered by network administrators. Tools such as anti-sniff
can be run on a network and will report any computers in "promiscuous mode". A computer network card that is sniffing is said to be in promiscuous mode, and a network card in promiscuous mode will take longer to reply than a normal network card. Based upon this, the network administrator would then be alerted to the sniffing, and you could face firing (if at work), expulsion(if at school), and even criminal charges. Only sniff the Internet's glue when at home...or somewhere you won't get caught.

Cambridge Public Internet/Roofnet (circa. Summer 2006)

Here is an article I wrote last Summer regarding the Cambridge Public Internet Initiative. I am afraid since last Summer, I have lost the thread of the project. Although I have recently become much more interested in resuming my interest. I live 3 houses down from a Cambridge Public Library branch. The branch has a WGT634U running Roofnet. I am going to break out one of my WGT634us and "mesh" it with the node at the library. I plan on upgrading the radio card (its only a mini-pci slot, so it snaps right out) to a CM-9 18dBm wireless card. To put this in perspective, the stock atheros card that comes with the WGT634U is a 60mW radio card. The CM-9 is a ~220mW radio card. Gain is logarithmic, so the extra 180mW would realize a potential gain of twice the strength and energy. I plan on coupling this radio card with a surplus directional antenna left over from Watch City Wireless. Once I am able to connect, Ill multiplex it with my Comcast cable, and use it as a backup data line.

The Original Article - Circa Summer 2006

Imagine you are an MIT student, and you can connect up wirelessly to the MIT University network from anywhere in Cambridge, MA. You can connect from any home, coffee shop or even from outside the other venerable Cambridge University Harvard. This will soon be a reality. The city of Cambridge, MIT, and Harvard University have partnered to build a city wide, open source, Linux based mesh wireless network. The Cambridge Public Internet network will provide free Internet access to all Cambridge residents, and visitors. It will allow Harvard and MIT students to connect wirelessly to their academic networks, and will provide them with pervasive Internet access throughout the entire city.

The city of Cambridge and its partners are providing funding and surplus Internet bandwidth to the project. The project uses commodity Netgear WGT634U access points, an Open WRT Linux distribution, and an open source mesh routing software developed at MIT called Roofnet. This combination of unused surplus Internet bandwidth coupled with low cost hardware and free open source software makes the project incredibly inexpensive. The city itself has pledged less than $100,000.00 to make this project a reality. This is far less than the multi-million dollar wireless rollouts in other cities such as San Francisco or Philadelphia.

It is the goal of this Muni-Uni (Municipal-University) alliance to make the Internet accessible and pervasive throughout the entire city of Cambridge. The Roofnet software classifies nodes with an Internet connection as gateways. The city will provide gateways at all 12 municipal buildings and the Universities will provide gateways on various buildings at each campus. Each gateway is an access point on top of a building with a high gain omni-directional antenna. The Roofnet mesh software running on each of these gateways connects to other nodes located throughout the city and shares their Internet connection. This combination of several (20+) Internet gateways and the mesh nature of the network will provide Internet access to all residential and business districts throughout the city.

One of the main advantages of mesh routing with the Roofnet software is the lack of user level configuration. A system administrator does not have to configure the network and the routes each time there is a change. The Roofnet software and mesh routing in particular will self create and configure a network. There is no need for an administrator to plan or design the network beforehand. Each additional node will discover its neighbors and add itself to the network. There is no need to assign IP addresses, use fancy directional antennas, or require other network users to reconfigure or change their equipment for a new node to join the network. The mesh software finds the best possible route through the multi-hop network. Loss of power, radio interference, and ISP problems can all be routed around. A self-creating, self-healing mesh wireless network will be a chief benefit to the city and its partners because of its lack of administrative overhead.

It is the job of the Roofnet software to determine the best link qualities and paths through the mesh wireless network. The Roofnet software uses the SrcRR protocol to determine the best high-throughput routes through the mesh network. This is similar to routing protocols such as OSPF. The major difference is the SrcRR protocol keeps track of 802.11 wireless link states between each of the nodes, and uses this to determine the best route. SrcRR does this through a combination of link loss monitoring, source routing, and forwarded broadcast destination querying.

When searching for a gateway to the Internet all nodes within the mesh will forward a query packet, and append their own identifier to the source packet. Once the query packet has reached the gateway node, the gateway will reply to the query packet sending the query along the same route accumulated along the path to the gateway.  Because of the mesh nature of the network, an originating node may receive replies from several adjacent nodes, and each reply may have traveled across a different path back to the originating node. Each received packet creates an entry in the link state database, and the Dijkstra algorithm is run against the link state database to determine which path is the best to any given destination. Through the process of participating in the mesh network, all nodes within the network will develop adequate links based upon link state.

As the link state database is populated and equalized, the SrcRR protocol continuously measures the 802.11 loss rate of each link. Both nodes on either side of a link will continually monitor the link and report statistics back to each other. Based upon these statistics the SrcRR protocol assigns a probability as to how many times a packet will have to be retransmitted. The SrcRR protocol then combines this probability metric against the link state database, and creates a second table of the best routes through the network.

This use of a probability metric is what allows the network to function as a mesh. Once the probability of packet transmission is less than another link. The SrcRR protocol switches to the new link with the better probability. An example of this would be if a high throughput 500 kbps link were to repeatedly go up and down. LetÕs say because of interference the link were to be only functional 50% of the time. Thus the probability of transmission would be 50%. The SrcRR protocol keeps track of this metric, and would determine if there were a better quality link through the network. If there were another link that was 400 kbps it would be assigned a value of .8 (80% of the 500 kbps link) but were up and functional 75% of the time, this link would have a better packet probability metric of 60%, and the SrcRR protocol would choose it above the higher throughput lower quality link.

500 kbps (100% of the highest throughput)  * 50% uptime =  250 kbps or 50% packet probability
400 kbps (80% of the 500 kbps link) * 75%  = 300 kbps or 60% packet probability

The SrcRR routing protocol is implemented using the Click modular router. The Click router is another project created out of the CSAIL research laboratory at MIT. Click is described as a modular router, because it allows various routing components to be added to it as software modules. The SrcRR protocol is implemented as a Click software module. The various link state monitoring, and general mesh routing are all done within this Click router and the SrcRR modules. On a Roofnet node Click runs as a user level program. To run the Roofnet software there is no kernel level interaction. All you have to do is compile the Click router and Roofnet modules for your platform. Running Click as a user level program, instead of a kernel level program is very useful and beneficial. It makes it very easy to install, run, and is quite portable to other systems.

The Click router essentially does three things. It determines a route to a destination. Ensures that the each link along that route is of adequate quality, and advertises a new route if the previous one should suddenly become unusable. It accomplishes this by reading and writing the lower level 802.11 MAC layer information from the wireless network card. The Roofnet project uses a modified Madwifi driver and requires an Atheros-based wireless card. An Atheros wireless card is a requirement to run a Roofnet node and network. Thanks to the code base submitted by Atheros, the Madwifi Linux kernel driver allows the SrcRR protocol to do its low level meshing magic. In addition an Atheros and Madwifi solution support many wonderful wireless features such as virtual access points, WDS mode, and WPA security. The Madwifi drivers in combination with an Atheros card make a wonderful Linux wireless solution, but because of the Atheros open code base they are a necessity for a Roofnet access point.

The Cambridge public Internet project is using the Netgear WGT634Us as the mesh nodes for the network. The access points come with Linux pre-installed. They have a  200MHz Broadcom  CPU, 32 MB RAM, 8MB of  flash memory, an Atheros 802.11b/g card, two Ethernet interfaces (one for the WAN connection one for the LAN) and a USB 2.0 port. They are wonderful access points, but they are no longer being manufactured by Netgear. Becaue of this, the Roofnet researchers are developing a low cost access point to be used with their software, and are in the process of manufacturing these nodes and have developed a business to support the Roofnet software.

The Roofnet software and general information on the Netgear WGT634U can be found at the Roofnet project site http://pdos.csail.mit.edu/roofnet/doku.php?id=wgt634u . The Roofnet software has gone through several versions but the latest version is now roofnet-2.0.6. To upgrade to this you first have to install roofnet-1.6.1, and then use the built in upgrade tool to upgrade to the latest 2.0.6 release. This need to install roofnet-1.6.1 first, will go away in future versions of the Roofnet software for the WGT634U.

The major difference between Roofnet version 1 and Roofnet version 2 is the inclusion of OpenWRT in the Roofnet version 2 release. Open WRT is described as a Linux distribution for embedded devices. It comes with a fully readable and writable file system, and has an advanced package management system. The version of Open WRT that comes installed with Roofnet is a 2.6 kernel, has a JFFS file system which provides persistent changes, and all the Click and SrcRR software installed and ready to go. In addition the system comes preinstalled and configured with various dameons and Linux tools such as iptables, an SSH daemon, a web server, and even Perl. Thanks to the Open WRT package manager ipkg additional software is very easy to download and install. Although you are limted by the 8 MB of persistent flash memory.

Installation of the Roofnet software onto a Netgear WGT634U access point is incredibly easy. There are several steps, but it is mostly uploading the correct firmware to the Netgear access point in the correct order.

You can install Roofnet on the Netgear WGT634U from any type of computer (Linux/Mac/XP, etc.):
1. Download the roofnet-1.6.1 image file http://pdos.lcs.mit.edu/roofnet/netgear/roofnet-1.6.1.img and the roofnet-2.0.6 binary file http://pdos.csail.mit.edu/roofnet/netgear/roofnet-2.0.6.bin to your computer.
2. Login to the routerÕs admin interface at http://192.168.1.1/ (default login is ÒadminÓ with the password ÒpasswordÓ)
3. Select the Router Upgrade option on the left-hand panel, then upload the image file under the upgrade file section.
4. Hit the upload button and watch the status bar on the screen to check on progress.
5. If you hadnÕt changed the default password, it is now r00fnet(with two zeroes).
6. Load the web interface at http://192.168.1.1. You should now see a ÒUpgrade to Roofnet 2.0Ó option on the top left-hand panel. If you donÕt hit reload, or right-click on the left hand panel and hit reload.
7. Upload the roofnet-2.0.6 binary file under the Upgrade to Roofnet 2.0 panel. The router will reboot twice and generate its ssh keys, which takes a few minutes.
8. When it is done, youÕll see a new web interface at 192.168.87.1, and the node will advertise itself using the essid ÒroofnetÓ. The login is now roofnet and the password is r00fnet with two zeros.

All configuration of the node is done through SSH. At this time there isnÕt an included administrative web interface. To SSH into the access point bring up a terminal and type ssh -l roofnet 192.168.87.1 if you are connected via an Ethernet cable or ssh -l roofnet 10.0.0.1 if you are connected via the wireless network. At the password prompt type in r00fnet with two zeros. Once you are in have a look around. Open WRT is a fully functional Linux distribution. The /bin/mount command shows what file systems are mounted at boot time.

roofnet@OpenWrt:/# mount
/dev/root on / type jffs2 (rw)
none on /dev type devfs (rw)
none on /proc type proc (rw)
none on /tmp type tmpfs (rw,nosuid,nodev)
none on /dev/pts type devpts (rw)
none on /sys type sysfs (rw)

Take notice of the first entry the read write mounted jffs2 / file system. This mount point is a direct link to the 8MB flash memory installed on the Netgear WGT634U circuit board. The / mount point has a normal looking Linux file system. It has an /etc directory for configuration of the system and contains the /etc/init.d startup scripts. It has a /bin and /sbin directory for user level commands. It even has a /www directory for the built-in web server.

To change the default password from r00fnet with two zeros simply run the command passwd roofnet. This command will prompt you for a new password for the roofnet user. Enter in the new password twice, and voila thanks to the readable and writable JFFS file system you have now permanently changed the /etc/passwd file and changed the default password on your access point.

After installation you can use the node as a normal access point, or you can pair it with other Roofnet nodes, and create a mesh wireless network. To use it as a normal access point, connect your Internet connection via an Ethernet patch cable to the WAN interface of the access point. You can use a wireless client such as a wireless enabled laptop or PDA to associate with the network ÒroofnetÓ and once connected surf the web, or check your e-mail. By default every roofnet node will act as an access point and a mesh node. The Roofnet software creates multiple virtual access points (VAP) using the Madwifi driver. Each virtual access point shows up in a /usr/sbin/iwconfig command and they are named ath0, ath1, and ath2.  The first ath0 is the VAP running  as an access point with an essid of ÒroofnetÓ. ath1 and ath2 are the interfaces used by the SrcRR protocol to communicate with other nodes within the mesh network.

ath0      IEEE 802.11b  ESSID:"roofnet"
         Mode:Master  Frequency:2.422 GHz Access Point: 00:0F:B5:3F:2E:0E
         Bit Rate=11 Mb/s   Tx-Power:20 dBm   Sensitivity=0/3
         Retry:off   RTS thr:off   Fragment thr:off
         Encryption key:off
         Power Management:off
         Link Quality=58/94  Signal level=-37 dBm Noise level=-95 dBm
         Rx invalid nwid:31  Rx invalid crypt:0 Rx invalid frag:0
         Tx excessive retries:0  Invalid misc:0   Missed beacon:0

ath1      IEEE 802.11b  ESSID:""
         Mode:Managed  Channel:0  Access Point: Not-Associated
         Bit Rate:11 Mb/s   Tx-Power:20 dBm   Sensitivity=0/3
         Retry:off   RTS thr:off   Fragment thr:off
         Encryption key:off
         Power Management:off
         Link Quality=58/94  Signal level=-37 dBm Noise level=-95 dBm
         Rx invalid nwid:44  Rx invalid crypt:0 Rx invalid frag:0
         Tx excessive retries:0  Invalid misc:0   Missed beacon:0

ath2      IEEE 802.11b  ESSID:""
         Mode:Monitor  Frequency:2.422 GHz Access Point: 0A:0F:B5:3F:2E:0E
         Bit Rate:0 kb/s   Tx-Power:20 dBm  Sensitivity=0/3
         Retry:off   RTS thr:off   Fragment thr:off
         Encryption key:off
         Power Management:off
           Link Quality=58/94  Signal level=-37 dBm  Noise level=-95 dBm
           Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
         Tx excessive retries:0  Invalid misc:0   Missed beacon:0

To change the essid or channel of the ath0 interface you use the command /usr/sbin/iwconfig. The format of the command is iwconfig interface [essid {NN|on|off}]. For example to change the essid from ÒroofnetÓ to Òliving_room_netÓ you would issue the command:

       /usr/sbin/iwconfig ath0 essid living_room_net

Or if you need to change the channel on your node from the default channel 3 to channel 11 you can run the command:

/usr/sbin/iwconfig ath0 channel 11

To make each of these changes permanently cd /etc/init.d and open the startup script S80roofnet with vi. This is the startup script that kicks off and configures the Roofnet software. It creates and configures the ath0 interface and starts up the click router. On lines 43 and 44 of the start up script are the same iwconfig commands you ran above. Change each line to read:

/usr/sbin/iwconfig $AP_DEV channel 11
/usr/sbin/iwconfig $AP_DEV essid living_room_net

After doing so, the changes you made are now permanent and will survive between reboots.

Low cost hardware, free open source software and surplus bandwidth are all that is required to build your own city or town wide mesh wireless network. To get started on a Roofnet implementation check out http://pdos.csail.mit.edu/roofnet/doku.php. It is  the website of the original research project, and it has information on the Roofnet software, and the Roofnet implemented hardware.





Tom is a resident of Cambridge, who is psyched that he lives next to two municipal buildings. He is also a passionate Boston wireless advocate, who looks forward to shooting a WiFi signal up the Charles River.

Open Source Mesh Routing Protocols

There is an excellent discussion going on over in the Open Source Mesh Google Group.

James Ewing from Sveasoft gave an excellent rundown on the state of Open Source mesh protocols. He said:
Couple thoughts on currently available mesh protocols and standards -

CuWIN - OSPF adapted to mesh networks. It is basically designed to
connected seperate subnet "islands" into a larger network. If you want
to link several local bu physically seperate community networks this
would be a good protocol.

AODV - on-demand protocol. Bit of overhead for route discovery. No
builtin wireless specific metrics in the RFC though specific
implementations may exist.

OLSR - link-state protocol. The RFC spec requires each node to
maintain a routing table for the entire mesh, which is impractical. No
builtin wireless route metrics. The olsr.org implementation threw the
RFC out the window and implemented a fisheye algorithm to limit route
table size and an ETX metric for "best wireless route" discovery. The
olsr.org implementation has been deployed in fairly large (1000+) node
networks.

WDS - store and forward 802.11 semi-standard. It suffers from
bandwidth halving at each hop (wheter you have 2 antennas or not). Not
always interoperable among different OEM chipsets.

BATMAN - created as a reaction to percieved limitations in the
olsr.org early implementations without the Fisheye or ETX
modifications. Seems to be a stipped Internet node discovery protocol
with a focus on wireless metrics for optimized routes. Currently at
the alpha or pre-alpha stage.

SCrr - don't know much about this one. Seems to be a Djikstra
algorithm based protocol like OSPF. A criticism in this post
http://listes.ilesansfil.org/pipermail/mesh/2004-April/000019.html
says it does scale well. Apparently used in Meraki units.

802.11s - IEEE L2 mesh networking standard. Uses a modified and
stripped form of AODV at L2 with a stripped version of OSLR as an
option. Most chipset manufacturers seem to be lending support to
802.11s but the only implementation "in the wild" seems to be from
Marvell via a firmware patch to their chips supplied to the OLPC (one
laptop per child) organization. The OLPC firmware runs a ThreadX based
micro kernel and apparently uses a proprietary virtual tunnel or
repeater wireless protocol specific to Marvell for node-to-node
communication. 802.11s uses a common channel for node discovery and
provisioning, similar to how a cellular phone discovers its local
network.

802.11n - multi-channel MIMO technique to increase thoughput over
802.11b/g and 802.11a networks. Since 802.11b/g has only 3 real
channels using n is impractical in most real life situations as 2 of
the 3 available are used for each n node. 802.11a has up to 31
separate channels and will likely be where 802.11n is practical.

I have some experience with the SrcRR protocol, but I was never really impressed by it. It was originally designed as a research protocol in the MIT Roofnet. The roofnet project consisted of 40-60 nodes spread around MIT. Each node's radio card was attached to a large high-gain dipole antenna, which was attached to the flashing or chimney of a partipant's roof. All in all, it is a really neat research project and idea, but the protocol shows its roots. It was designed with mesh healing as one of its primary attributes. This forces the nodes to communicate link information more and run a Dijkstra algorithm more often. Both of which take up considerable network and computational resources.

I have recently been turned on to the B.A.T.M.A.N. protocol. I haven't worked with it, but it appears to have better(more efficient) use of resources. It will determine the best link, and then stick with it. If that link goes down, it will look for a new one. I love the design, and developer's approach.

Forget city wide wireless. The most interesting mesh networks are going to be small-to-medium size, and they are going to be run on low cost Linux routers. I love the idea of a small efficient routing protocols on those little machines. It is definitely the approach to take.

Teaching OSPF and RIP

This semester I am teaching "Applied Networking". The "applied" in the title refers to an expectation for the students to work hands in a hands on environment. Well the BFIT doesn't have a functional, segmented networking lab. I would love to have access to a Cisco lab, similar to the one I used at Bentley. Or the beautiful security lab at Johnson and Wales.

In previous semesters, I tried using a Software based Cisco lab, which was only moderately successful. I have found that the students have a totally different reaction to a hardware system that they can touch. Perhaps I can revisit this idea, and start working on a solution. Older Cisco 2600s and 2900s can be found on Craigslist for short money. The investment in these products would certainly be very beneficial. It would be wonderful if my students left the BFIT with hard Cisco IOS skills, wiring skills, and general applied routing knowledge. It would be such a thrill.

General applied routing knowledge is really missing from this Applied Networking course. I spend a great deal of class time on RIP and OSPF, but frankly nothing beats setting up and configuring routes on a network, and then seeing your packets traverse those routes you built.  My students come into the class with some experience using and setting up home networks. The wired and wireless Netgears, and Linksys devices abstract their routing functions, and my students tend to think of Internet routing within this concept. They have trouble shifting gears, and thinking about WAN routing as a very important aspect of our modern day Internet architecture. It's almost as if little wireless routers have made routing no big deal. Well BGP and OSPF certainly are big deals and are such a big deal, that they could fill an entire semester course.