Network Management

Cisco / Linksys WRT610N and Management
Abstract:
Cisco's Linksys WRT610N is a consumer grade wireless router / access point which contains some higher-end features such as dual band, 5 port gigabit ethernet, and USB2 storage capabilities. When the device was released, the gigabit ethernet and dual-band capabilities were uncontested. Overall, for the price, it was a reasonable device, but some odd behaviors are exhibited.

Logging:
Network logging under the wrt610n was isolated to the network on the LAN interface and could not be forwarded to another network address. This means, centralized logging is impossible if one is trying to use this consumer grade product to provide unified support in an extended family situation, where different households need to be observed. Centralized administration (through a GUI) without centralized logging is pretty useless.

Fault, Performance, and Configuration Management:
With this device, you have some limited Configuration Management, but that is about it.
If you want to know whether a cable, port, or PC on you LAN port is experiencing errors due to something failing?- no way.
If you want to know what your consumption of network resources over your WAN connection to upgrade/downgrade your broadband - no way.
Want to know which device on your LAN is causing your VoIP phone service to degrade - no way.
Want to have your wireless router check into your own private Dynamic DHCP server, so you can help our your family members around the country when they have an problem - no way, unless you want their router name and ip to be broadcast to the rest of the world.
Want to detect viruses/worms infecting your PC's and pumping out tons of traffic, using up your WAN bandwidth - no way.
Want to detecting bots using your network to create a distributed denial service attack [resulting in the visit of local law enforcement] - no way.
Want to create a unified VPN network for your extended family, to share resources with privacy - no way.

Old Linksys VPN and VPN edge routers had some of these capabilites, without the wireless. The wrt610n was an expensive consumer device, with none of the useful features someone who would buy one of these would want. If you need real documented support, buy an old Cisco off of eBay (for less than you will buy this high-end consumer device) and configure it by hand using the following blog as an ADSL reference example.
You have to go with a real firmware from an alternate manufacturer, besides Cisco, to do anything useful with this device, besides act as a gigabit switch. Unfortunately, because this high-end consumer item was fairly rare, most firmware providers consider the software beta, and some experiencing the proverbial "bricking" of their device.
Management Options:

SNMP Management:
Management via SNMP is unavailable. Early Linksys devices offered SNMP, but this is no longer. SNMP offers a safe way to view the configuration and performance performance characteristics of your router, but not with this high-end consumer device. This is a great disappointment, but there are some third-party firmware options.

Command Line Management:
Management via Telnet or SSH is unavailable. This allows fast access into a device, as well as being able to automate some basic operations, such as backing up configurations or collecting performance characteristics without SNMP. Unavailable, without going to a third-party firmware provider.

Web GUI Management:
Management via an HTTP or HTTPS web can be done via the LAN and the Wireless LAN. There is an option to administer the products over the internet, but this is pretty useless without the option of central logging outside of the LAN, SNMP, or command line interface.

Sure, you can wait until someone calls you to tell you the internet connection is flakey, but at that point, you better be prepare to drive to your family mambers house to read the logs locally or to help them navigate the web screen from their local web browser using the ip address.

Secret Web Management GUI:
There is a secret management GUI, where you can get some basic information about the device. Below is a small listing of available options. Many Telco and Bable providers will use the 192.168.1.x address range for their customer premesis terminations, so this following list will be applicable if you configured your Linksys device with a 192.168.3.1 ip address.

http://192.168.3.1/System.asp - Hidden Web Menu: Disable "Microsoft Vista" 6to4 multicast broadcasts
http://192.168.3.1/SysInfo.htm - Hardware Configuration (Vendor, Model, Serial, etc.)
http://192.168.3.1/SysInfo1.htm - RAM Counters: Total, Free, and Buffers
http://192.168.3.1/Cysaja.asp - Packet Counters: Transmit & Receive, both Good & Errors

These undocumented hidden menus can be used to get some of the basic information needed to understand the health of your network, no thanks to Cisco.
Strange Behaviors:
While trying to enable reasonable management of these high-end consumer level devices, one can expect various strange behaviors.

Junk Log Entries:
When logging is enabled, the logging machine would reflect a strange behavior - IPv6 packets would be constantly broadcasted, every 10 seconds, and getting logged! This will fill up the logs with useless data, making the logging facility pretty useless. This is related to IPv6 6to4 functionality.

sunsparc/user$ tail -f /var/adm/messages

Jul 3 14:02:53 wrt610n last message repeated 61 times
Jul 3 14:09:31 wrt610n klogd: IN=br0 OUT= MAC= SRC=fe80:0000:0000:0000:0221:29ff:febf:aaaa DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=96 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=134 CODE=0

Jul 3 14:09:31 wrt610n last message repeated 64 times
Jul 3 14:16:16 wrt610n klogd: IN=br0 OUT= MAC= SRC=fe80:0000:0000:0000:0221:29ff:febf:aaaa DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=96 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=134 CODE=0

Jul 3 14:16:16 wrt610n last message repeated 59 times
Jul 3 14:22:54 wrt610n klogd: IN=br0 OUT= MAC= SRC=fe80:0000:0000:0000:0221:29ff:febf:aaaa DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=96 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=134 CODE=0

Jul 3 14:22:54 wrt610n last message repeated 62 times
Jul 3 14:29:33 wrt610n klogd: IN=br0 OUT= MAC= SRC=fe80:0000:0000:0000:0221:29ff:febf:aaaa DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=96 TC=0 HOPLIMIT=255 FLOWLBL=0 PROTO=ICMPv6 TYPE=134 CODE=0

Debugging Junk Logging:
Going through the standard menus is absolutely worthless. There is no clue to show the user why their logging system is being filled up with absolute junk.

Eliminating Junk Logging:
Are you using IPv6? Well, if the answer is yes, then you are stuck with the horrible logging effects. No fix.

If you are not using IPv6, you can shut off the 6to4 protocol, using a secret menu. Oddly, a developer decided to blame his additional coding requirement on Microsoft, probably because Vista Premium support was placed on his firmware to-do list.

Use the secret "http://ipaddress.of.wrt610n.router/System.asp" menu, to "(X) Disable" the "Vista Premium" functionality. Be aware that when the "Save" button is pressed, the wireless access point will be rebooted. After the outage, be happy that the router will stop the packet creation and junk logging.


Large USB Drive Support:
Trying to attach 1.5TB external drive to the storage port was completely unsuccessful. Don't bother using the USB storage port for serious storage, it does not work well. There are now 4TB drives, I would not expect them to work. No fix.

USB sticks work pretty well, with an embedded FTP server. These have a much smaller capacity and slower to access than spinning disk, but that will change over time.
A stock Apple wireless access point with USB will be a more effective solution for storage. The newer Apple AirPort units offer gigabit ethernet in addition to dual-band now a days. At the time the WRT610N came out, no one offered these features, including Apple.

If USB storage is a requirements, the Apple wireless access point offers vastly more robust management from MacOSX, multiple USB drive support, sharing of mirrored disks in an external USB RAID enclosure, and even USB printing support as an added bonus.

DHCP Reservation:
The DHCP Reservation is able to be configured via "Setup -> Basic Setup -> DHCP Reservation" from the main administration GUI menu. The "Client Name" option can be set in the "Already Reserved" area - which provides for the ability to give a MAC Address an IP Address as well as a Name.

The problem with the Name? It is not used ANYWHERE. How about using the name in the DNS server or the other status menus? No fix.

Abstract:
ZFS was created by Sun Microsystems to innovate the storage subsystem of computing systems by simultaneously expanding capacity & security exponentially while collapsing the formerly striated layers of storage (i.e. volume managers, file systems, RAID, etc.) into a single layer in order to deliver capabilities that would normally be very complex to achieve. One such innovation introduced in ZFS was the ability to provide inexpensive limited life solid state storage (FLASH media) which may offer fast (or at least greater deterministic) random read or write access to the storage hierarchy in a place where it can enhance performance of less deterministic rotating media. This paper discusses the process of upgrading attached external mirrored storage to external network attached ZFS storage.

Case Study:
A particular Media Design House had formerly used multiple external mirrored storage on desktops as well as racks of archived optical media in order to meet their storage requirements. A pair of (formerly high-end) 400 Gigabyte Firewire drives lost a drive. An additional pair of (formerly high-end) 500 Gigabyte Firewire drives experienced a drive loss within one month later. A media wall of CD's and DVD's was getting cumbersome to retain.

The goal was to consolidate the mirrored sets of current data, recent data, and long-term old data onto a single set of mirrored media. The target machine the business was most concerned about was a high-end 64bit dual 2.5GHz PowerMAC G5 deskside server running MacOSX.


The introduction of mirrored external higher capacity media (1.5 TB disks with eSata, Firewire, and USB 2.0 options) proved to be far too problematic. These drives were just released and proved unfortunately buggy. During improper shutdowns or proper shutdowns where the media did not properly flush the final writes from cache in time resulted in horrible delays lasting over a day. Rebuilding the mirrored set upon next startup would take over a day, where access time to that media was tremendously degraded during a rebuild process.

Moving a 1.5TB drives to external USB storage connector on a new top-of-the-line Linksys WRT610N Dual-Band N Router with Gigabit Ethernet and Storage Link proved impossible. The thought is that the business would copy the data manually from the desktop to the network storage nightly, by hand, over the gigabit ethernet. Unfortunately, the embedded Linux file system did not support USB drives of this size. The embedded Linux int he WRT610N system also did not support mirroring or SNMP for remote management.

The decision was to hold-off any final decision until the next release of MacOSX was released, where a real enterprise grade file system would be added to MacOSX - ZFS.


With the withdrawal of ZFS from the next Apple operating system, the decision was made to migrate the all the storage from the Media Design House onto a single deskside ZFS server, which could handle the company's storage requirements. Solaris 10 was the selected, since it offered a stable version of ZFS under a nearly Open Source operating system, without being on the bleeding-edge as OpenSolaris was. If there was ever the decision to change the licensing with Solaris 10, it was understood that OpenSolaris could be leveraged, so long term data storage was safe.

Selected Hardware:
Two Seagate FreeAgent XTreme external drives were selected for storage. A variety of interfaces were supported, including eSATA, Firewire 400, and USB 2.0 At the time, this was the highest capacity external disk which could be purchased with the widest variety of high-capacity storage interfaces off-the-shelf at local computer retailers. 2 Terabyte drives were expected to be released in the next 9 months, so it was important the system would be able to accept them without bios or other file system size limitations. These were considered "green" drives, meaning that they would spin down when not in use, to conserve energy.


A dual 450MHz deskside Sun Ultra60 Creator 3D with 2 Gigabytes of RAM was chosen for the solution. They were well build machines with a current low price-point which could run current releases of Solaris 10 with modern ZFS filesystem. Dual 5 port USB PCI cards were selected (as the last choice, after eSATA and Firewire cards proved incompatible with the Seagate external drives... more on this choice, later.) Solaris offered security with stability, since few viruses and worms target this enterprise and managed services grade platform, and a superior file system to any other platform on the market at the time (as well as today): ZFS. SPARC offered long term equipment supportability since 64 bit was supported for a decade, while consumer grade Intel and AMD CPU's were still struggling to get off of 32 bit.

The Apple laptops and Deskside Server all supported Gigabit Ethernet and 802.11N. Older Apple systems supported 100 megabit Ethernet and 802.11G. A 1 Gigabit Ethernet card for the Sun Ultra60 was purchased, in addition to several Gigabit Ethernet Switches for the office. A newly released Linksys dual-band Wireless N router with 4xGigabit Ethernet ports was also purchased, the first of a new generation of wireless router in the consumer market. This new wireless router would offer simultaneous access to network resources over full-speed 2.4GHz 802.11G and 5GHz 802.11 N wireless systems. The Gigabit ethernet switches were also considered "green" switches, where power was greatly conserved when ports were not in use.


CyberPower UPS's were chosen for the solution for all aspects of the solution, from disk to Sun server, to switches, to wireless access point. These UPS's were considered "green" UPS's, where their power consumption was far less than competing UPS's, plus the displays clearly showed information regarding load, battery capacity, input voltage, output voltage, and component run time.

Speed Bumps:
The 64 bit PCI bus in the Apple Deskside Server and the Sun Deskside Workstation proved notoriously difficult to acquire eSATA cards, which would work reliably. The drives worked independently under FireWire, but two drives would not work reliably on the same machine with FireWire. A pair of FireWire cards was also purchased, in order to move the drives to independent controllers, but this did not work under either MacOSX or Solaris platforms with these external Seagate drives. The movement to USB 2.0 was a last ditch effort. Under MacOSX, rebuild times ran more than 24 hours, which drove the decision to move to Solaris with ZFS. Two 5 port USB 2.0 cards were selected, one for each drive, with enough extra ports to add more storage for the next 4 years. The USB 2.0 cards had a firmware bug, which required a patch to Solaris 10, in order to make the cards operate at full USB 2.0 speed.

Implementation:
A mirror of the two 1.5 Terabyte drives was created and the storage was shared from ZFS with a couple of simple commands.

The configuration is as shown below.

Ultra60/user# zpool status
pool: zpool2
state: ONLINE
config:
 NAME STATE READ WRITE CKSUM
 zpool2 ONLINE 0 0 0
 mirror ONLINE 0 0 0
 c4t0d0s0 ONLINE 0 0 0
 c5t0d0s0 ONLINE 0 0 0
errors: No known data errors
Ultra60/user# zfs get sharenfs zpool2
NAME PROPERTY VALUE SOURCE
zpool2 sharenfs on local

Implementation Results:
Various tests were conducted, such as:

• Pulling the power out of a USB disk during read and write operations
• Pulling the USB cord out of a USB disk during read and write operations
• Pulling the power out of the SPARC Workstation during read and write operations

Under all cases, the system recovered within seconds to minutes with complete data availability and quick access to the data (instead of days of sluggishness, due to completing a rebuild, with the former desktop mirrored solution.)

Even though the SPARC CPU system was vastly slower, in raw CPU clock speed, from the POWER CPU in the Apple deskside unit, the overall performance of the storage area network was vastly superior to the former desktop mirroring attempt using the high-capacity storage.

Copying the data across the ethernet network experienced some short delays, during the time the disks needed to spin up from sleep mode. With future versions of ZFS projecting to support both Level 2 ARC for reads and Intent Logging for writes, the performance was considered more than acceptable until Solaris 10 received sufficient upgrades in the future.

The system was implemented and accepted within the Media Design House. The process of moving old desktop mirrors and racks of CD and DVD media to Solaris ZFS storage began.