For quite a while I had issues with a faulty service module in a remote Cisco. Sometimes the card would lock-up and no longer respond and the only way to revive it was to issue the command service-module wlan-ap 0 reset to reboot it.
Normally one would simply replace it but that wasn’t that simple as the card itself can’t be replaced (and the device is also EOL) and the device is in a remote location without any technical staff on-site. Since rebooting the service module resolved the issue for that moment I looked for a way to automate that and documented that in this post.
For some time I have been looking for a way to do simplistic performance monitoring without the need of two linux (virtual) machines to use the obvious solution iPerf. The iPerf solution is fine for normal situations but not very practical to quickly test a connection between routers as it requires 2 additional machines and configuring them correctly for the networks they should be in.
It turned out that Cisco’s IOS has a ‘hidden’ command ttcp for exactly this purpose. This command ‘hidden’ as it is not visible in the command help and will not and not present in all IOS versions. Cisco states that it is a hidden, unsupported, privileged mode command that requires IOS 11.2 or higher and Feature Sets IP Plus (is- images) or Service Provider (p- images. However, Cisco stresses that the command’s availability may vary from one Cisco IOS software release to another (i.e. it may not exist in some releases). Documentation of it is available on Cisco’s Test TCP (TTCP) manual page (from 2005 so quite old).
Obviously the command works great between two Cisco’s (I could quite quickly measure the performance of my test DSL connection and it measured an expected 1.8Mbit throughput over the 2Mbit DSL connection. But how about a Cisco router and another system? After a little searching I found the nuttcp command, which according to its homepage is available for *nix systems but unfortunately the binary version of it does not install on Mac OS X. Fortunately the Debian Linux distribution has a package available so it was easy to get it installed on a Linux box (will look into an OS X version later when I really need it). In ‘legacy’ mode this command is working fine with Cisco’s implementation and running it is as easy as starting ttcp on the router and answering the prompts:
Router#ttcp
transmit or receive [receive]:
perform tcp half close [n]:
receive buflen [8192]:
bufalign [16384]:
bufoffset [0]:
port [5001]:
sinkmode [y]:
rcvwndsize [4128]:
delayed ACK [y]:
show tcp information at end [n]:
ttcp-r: buflen=8192, align=16384/0, port=5001
The last line is confirming that the router is waiting for a connection) and on the other end starting:
nuttcp <hostname or ip address of router>
This runs the throughput test from the Linux client using the Cisco as a receiver.
For some time I have a small test network with a number of old Cisco routers (mainly 2500 series). Recently I decided to purchase a c1841 through Marktplaats as the IOS versions of the 2500 is really ancient and is very limited for more complex setups nowadays (latest IOS version is 12.3)
With the c1841 I also obtained two SHDSL cards, specifically:
a G.SHDSL WIC (WIC-1SHDSL-V3)
a 2-pair G.SHDSL HWIC (HWIC-2SHDSL)
These cards are described in detail on Cisco’s website. It turned out that the G.SHDSL HWIC card is supported in my main router (a c1921), so I decided to connect my test network to my main router using a DSL connection using these two cards to fully separate the test network from my main networks (and for the fun of it). Cisco had a good guide available to set this up (see Configuring Cisco G.SHDSL HWICs in Cisco Access Routers and Setup back to back CPE connection and ) but as it turned out not to be totally trivial I decided to document my setup here as well.
Cabling
The nice thing about (SH)DSL is that it uses standard phone (CAT-4) cables that can be up to several kilometers long. For my test lab I started off with a standard (2-wire) phone cable with 6-pin RJ-11 connectors. (SH)DSL uses a straight connection where one of the ends should be put in CPE (subscriber) mode and the other one in CPO (office) mode.
As the 2-pair G.SHDSL HWIC (HWIC-2SHDSL) interface has two ports and I temporarily had two c1841 routers with a G.SHDSL WIC (WIC-1SHDSL-V3) I also created a splitter cable as per the Cisco G.SHDSL documentation (diagram below) to establish two DSL connections.
SHDSL RJ-11 splitter schematic
Dual DSL connection with 2 wires using a splitter cable
With the above cable it was quite easy to establish DSL connections between my c1921 router with the 2-pair G.SHDSL HWIC (HWIC-2SHDSL) card and two c1841with an G.SHDSL WIC (WIC-1SHDSL-V3) adapter. Unfortunately the 2-pair G.SHDSL HWIC (HWIC-2SHDSL) can only operate in CPE (client) mode so c1921 was running in CPE mode for both lines and the two c1841s both need to be set to CPO (office) mode. This setup looks like this:
Below is the configuration used:
c1921 with 2 DSL connections to 2 c1841s:
controller SHDSL 0/0/0
dsl-group 0 pairs 0
!
dsl-group 1 pairs 1
!
!
interface ATM0/0/0
no ip address
no atm ilmi-keepalive
pvc 0/35
encapsulation aal5snap
!
pvc 8/35
encapsulation aal5mux ppp dialer
dialer pool-member 2
!
!
interface ATM0/0/1
no ip address
no atm ilmi-keepalive
pvc 0/35
encapsulation aal5snap
!
pvc 8/35
encapsulation aal5mux ppp dialer
dialer pool-member 3
!
!
interface Dialer1
ip unnumbered GigabitEthernet0/0
encapsulation ppp
dialer pool 2
dialer-group 2
!
interface Dialer2
ip unnumbered GigabitEthernet0/0
encapsulation ppp
dialer pool 3
dialer-group 3
!
dialer-list 2 protocol ip permit
dialer-list 3 protocol ip permit
Obviously in a real-live setting additional statements will be required to ensure that firewall, nat, etc. are also correct but that is not the intention of this description. The above sets up the SHDSL adapter in 2-line more and defines two ppp connections over ATM and uses the IP address of the main GigabitEthernet interface also for the dialer interfaces. The corresponding configuration on the two c1841s is:
c1841 with DSL connections to c1921 (two times):
controller DSL 0/0/0
mode atm
line-term co
dsl-mode shdsl symmetric annex B
!
interface ATM0/0/0
no ip address
no atm ilmi-keepalive
pvc 0/35
encapsulation aal5snap
!
pvc 8/35
encapsulation aal5mux ppp dialer
dialer pool-member 1
!
!
interface Dialer0
ip unnumbered Loopback0
encapsulation ppp
dialer pool 1
dialer-group 1
!
ip route 0.0.0.0 0.0.0.0 Dialer 0
!
dialer-list 1 protocol ip permit
!
This (re)uses the IP address of the Loopback0 interface for the dialer interface. Thanks to the default route all 3 Cisco’s will be able to reach each other. During the simple speed tests I was able to do I noticed that the DSL connection could almost reach it’s 2Mbit maximum throughput, even from one c1841 through to c1921 to the other c1841. For more complex routing I would recommend not using static routes but using a routing protocol like EIGRP (which I am using as well and will describe later).
Single DSL connection with 4 wires using a straight cable
As one of the c1841s was a loaner, I then decided to setup the permanent connection slightly different using a 4-wire cable and configured the c1921 and c1841 slightly differently to utilise all 4 wires of the cable. The benefit of this setup is that it doubles the connection speed to approx. 4.5Mbit (still not really amazing considering that my internet connection over fiber cable is 50Mbit, but a bit more). This setup looks like this:
Below is the configuration used:
c1921 4-wire DSL connections to c1841:
controller SHDSL 0/0/0
dsl-group auto
shdsl 4-wire mode enhanced
!
!
interface ATM0/0/0
no ip address
no atm ilmi-keepalive
pvc 0/35
encapsulation aal5snap
!
pvc 8/35
encapsulation aal5mux ppp dialer
dialer pool-member 2
!
!
interface Dialer1
ip unnumbered GigabitEthernet0/0
encapsulation ppp
dialer pool 2
dialer-group 2
!
This setup is pretty similar to that before apart from how the SHDSL controller is configured and there is also (due to this configuration) only 1 ATM interface (so also only Dialer interface needed). However, as the controller is changed it is required to issue the following command to remove the previous dsl-group definitions to switch from the previous setup:
controller SHDSL 0/0/0
no dsl-group 0
no dsl-group 1
Before this new configuration can be entered, which also removes all configuration of the ATM interfaces (the Dial interfaces are unaffected). The corresponding configuration on the c1841 is:
c1841 4-wire DSL connections to c1921:
controller DSL 0/0/0
mode atm
line-term co
line-mode 4-wire enhanced
dsl-mode shdsl symmetric annex B
!
interface ATM0/0/0
no ip address
no atm ilmi-keepalive
pvc 0/35
encapsulation aal5snap
!
pvc 8/35
encapsulation aal5mux ppp dialer
dialer pool-member 1
!
!
interface Dialer0
ip unnumbered Loopback0
encapsulation ppp
dialer pool 1
dialer-group 1
!
ip route 0.0.0.0 0.0.0.0 Dialer 0
!
dialer-list 1 protocol ip permit
!
With this new configuration early measurements indeed confirm the bandwith is higher, but not really doubled. As this is a connection to my test network, the performance is not really an issue but I like the idea of having physical cable that I can remove easily and the fact that my test network is not connected to my main switch / network in any way.
For some time now I am using a borrowed Cisco 881 router as router/firewall for my internet connection. The box is stable and configured as I want, but unlike with the Linux and Fritz!Box routers I used before, the Cisco does not allow to connect to forwarded IPv4 ports on its external address. This is inconvenient in my situation as this means that I am unable to reach some services from my internal network (i.e. I cannot reach websites I host). So far the only way around this was using split DNS and double administration, which is quite tedious and inconvenient.
Some time ago when looking how to set this up, I bumped into this article: NAT: access outside global address from the inside (this site seems to be down at the moment, but it’s content is still available through here thanks to the Internet Archive). This describes an alternative way to setup the Cisco NAT rules using the NAT Virtual Interface (NVI),which decouples them from the specific interface in a specific direction. Today I have tested this approach.
Setup
To setup the new NAT approach, change the existing NAT rules:
ip nat inside source static tcp 192.168.0.100 80 WW.XX.YY.ZZ 80
into something that looks like the next line:
ip nat source static tcp 192.168.0.100 80 WW.XX.YY.ZZ 80
ip access-list extended NAT-INSIDE-ADDRESSES
permit ip 192.168.0.0 0.0.0.255 any
!
ip nat source list NAT-INSIDE-ADDRESSES interface FastEthernet0/1 overload
(basically remove the inside clause in the statement). In my setup 192.168.0.100 is the internal IP address of my web server and WW.XX.YY.ZZ represents my external IP address. In this example I forwarded port 80 (HTTP). The last part is required to make sure that also internal traffic on FastEthernet0/1 will be NATted properly to avoid asynchronous data flows.
Testing it
The first basis tests of this new setup were promising. Indeed, after these changes I could access my external sites also from internal addresses. However, when downloading something from an internal site I noticed that the performance was not very good. This was something I definitely could live with as the traffic would not be massive. However, due to this change in config, all NAT traffic turned out to be slower and effectively the performance of my network connection was about half of what it used to be. Before this change the Cisco 881 was capable of streaming about 38 – 43 Mbit, which was not my full 50Mbit bandwith, but close enough. With this (NVI) setup, I noticed that my max. network bandwith using SpeedTest.NET dropped to 20Mbit and below. With the command show processes cpu history
on the router I noticed that the poor Cisco 881 was at 100% CPU load/utilization during the downloads. I suspect that the old Cisco 881 (which does not support 50Mbit in the first place) is CPU-bound when using NAT Virtual Interfaces and not capable of handling this at higher speeds.
Conclusion
Technically, the approach to use the NAT Virtual Interface (NVI) feature of IOS works to enable access to NAT forwarded external ports from the inside. However, since this appears to be very CPU intensive, it is not a good solution for now as the Cisco 881 cannot cope with the load and the internet bandwith is effectively reduced to only 50%. I think need to revisit this approach once I have acquired a router that is capable to support the bandwith I have and see if then can handle the CPU load.
