guinix TechNote: DSL in Portland, Part 2

PPPoE through the Cisco 678 DSL router

Introduction

In Part 1 of our story, we describe our recent return from Africa to Portland, Oregon, and our happy anticipation of great broadband.

But then we hit an annoying snag: the discovery of a surprising flaw in the network address translation (NAT) implementation on our Cisco 678 DSL router.

Although we did figure out a workaround specifically for a problem with DNS traffic, the experience left us feeling disenchanted with the packet filtering competency of the Cisco device. We simply didn't trust it anymore.

Ever since our first DSL connection with Qwest in Butte, Montana, we have wished for the possibility of hosting our public IP address directly on our own firewall, running OpenBSD. This would allow us to use the excellent and trusted PF packet filter on all traffic in and out of the network, not just those packets making their way through the DSL router. And compared to a closed, "black-box" device like the Cisco 67x, an OpenBSD/PF combination would offer many advantages:

• extremely secure and robust
• PF rules and syntax are powerful and flexible
• easy to configure and reload PF rulesets
• full logging and packet analysis via tcpdump on /dev/pflog0
• under ongoing development and improvement

Despite these advantages, it seemed that this would never be possible, at least not with our Cisco 67x hardware and current DSL provider. On Qwest provisioned lines, all instructions pertain to configuring the PPPoA built in to the Cisco 678 device. That is, you load your user name and password into the Cisco, and it performs the peer-to-peer link negotiation and network connection with the ISP's router over an ATM transport layer.

But our NAT experience so niggled us, we renewed our effort to find a way to bypass the device.

It turns out this is really pretty easy after all. Just setup the Cisco as a transparent bridge, and use PPPoE from the OpenBSD host for authentication and link negotiation. This gets the public IP address on our own gateway, and we can finally cut the Cisco NAT out of the picture entirely.

Diagram

Here is a crude sketch of the connection principle:

+----------+
|          |                           +-------+
| TCP      |                           |       |
| IP       |     +---------------+     | IP    |---[internet]
| PPP      |     |               |     | PPP   |
| ethernet-|-----|-ethernet  atm-|-----|-atm   |
|          |     |               |     |       |
+----------+     +---------------+     +-------+
 PPPoE on          Cisco 678 as         ISP
 OpenBSD           rfc1483 bridge       router 

With the Cisco 678 set to bridging mode, we physically connect its ethernet port to an ethernet port on our OpenBSD firewall. Then we run the PPP connection dialogue on the firewall, instead of on the Cisco. Because the Cisco acts as a bridge, the PPP dialogue running on the OpenBSD box is now channelled through on the ATM transport layer directly to the ISP's router.

Conceptually, it is all quite simple. The Cisco 678 just functions as a straight pipeline to the ISP; all other processing and packet filtering is now done on the OpenBSD box. And, as we will see, OpenBSD gives us all the tools we need to do this with the standard installation. No special tricks, extra packages, or kernel builds are required.

Cisco Bridging

First we'll set up the Cisco 678 in bridging mode. Note that this is done over the serial interface, that is, with the pale blue management cable connected to the serial port of another system. That's because --once configured for bridging mode-- the Cisco will no longer have an addressable network interface of its own.

From a BSD system, it's easy to use the tip(1) command:

# tip tty00

Alternatively, use a full featured communication program, such as minicom or kermit.

Then, in the terminal session, here is a complete sequence of commented CBOS commands to reset the device and configure it for bridging mode:

## reset the device to factory defaults:
enable
set nvram erase
write
reboot
## after reboot, configure for bridging:
enable
set bridging rfc1483 enabled
write
## configure DMT parameters:
set interface wan0-0 close
write
set interface wan0-0 vpi 0
set interface wan0-0 vci 32
set interface wan0-0 open
write
## some security/paranoia:
set tftp disabled
set telnet disabled
set web disabled
set web port 8888
set web remote 255.255.255.255
set password exec somepass
set password enable somepass
write
reboot

That's all there is to it for the Cisco. Now on to the OpenBSD firewall.

PPPoE on OpenBSD

First, the OpenBSD box should be configured to permit routing of packets between interfaces. Make sure there is an uncommented line like this in the file /etc/sysctl.conf:

net.inet.ip.forwarding=1

This system will usually have at least two network interfaces: one ethernet port connected to the Cisco device, another connected to a switch/hub for your local network. In the example here, we will consider our OpenBSD gateway on Soekris hardware, which actually has three physical network interfaces. Here is a table:

The first two interfaces, sis0 and sis1, are configured in the usual way, with the arguments for ifconfig as shown in hostname.sis0 and hostname.sis1.

The third interface, sis2, is the one physically connected to the DSL router. When we use PPPoE, it just provides for the physical transport, and isn't assigned an IP address itself. All we need to do in the way of configuration is to bring it up. And so the single word "up" in the hostname.sis2 file.

The last interface here, tun0, is the one that will receive the public IP address assigned during the PPP connection dialogue with the ISP. It is through tun0 that we will have our gateway to the 'net. Note that no hostname.if file is used for configuration of the interface; the tun0 configuration is all done through PPP.

The PPP we use in this case is the excellent "user ppp" available on BSD systems. Its configuration file is found in /etc/ppp/ppp.conf and for PPPoE will look like this:

# /etc/ppp/ppp.conf
# config for user ppp, with pppoe for dsl connection
# ===
default:
  set log Phase Chat IPCP CCP tun command 
  set redial 15 0
  set reconnect 15 10000

## qwest dsl connection:
qwest:
  set device "!/usr/sbin/pppoe -i sis2"
  disable acfcomp protocomp
  deny acfcomp
  set mtu max 1492
  set mru max 1492
  enable mssfixup
  set speed sync
  enable lqr
  set lqrperiod 5
  set ctsrts off
  disable ipv6cp
  set dial
  set login
  set timeout 0
  set authname yourname
  set authkey  yourpass
  add! default HISADDR

### that's all, folks!

This is very similar to a PPP configuration you might see for a dial-up connection. The main difference is that the set device isn't running on a serial line. Instead the special pppoe(8) helper application is used over the sis2 interface. Other parameters, such as the mru and mtu, are set as required for the particular characteristics of the PPPoE session. Note also that set timeout 0 disables the idle timer on the link, which should be kept up even without any traffic.

To test and debug the setup, you can run ppp(8) in interactive mode. A sample session would look like this:

# ppp 
Working in interactive mode
Using interface: tun0
ppp ON soekris> dial qwest
ppp ON soekris> Warning: deflink: Reducing configured MRU from 1500 to 1492
Ppp ON soekris> 
PPP ON soekris>

When the "PPP" in the prompt goes all uppercase, the connection is up. Normally, this happens almost instantly. From another terminal, check the tun0 interface and the routing tables to see the IP configuration is as expected:

# ifconfig tun0
tun0: flags=8011 mtu 1492
        inet 209.180.174.155 --> 207.225.84.1 netmask 0xffffffff 
# route -n show
Routing tables

Internet:
Destination      Gateway            Flags 
default          207.225.84.1       UG     
10.0.1.0         link#1             U      
10.0.1.254       127.0.0.1          UGH    
127.0.0.0        127.0.0.1          UG     
127.0.0.1        127.0.0.1          UH     
192.168.1.0      link#2             U      
207.225.84.1     209.180.174.155    UH     

Connectivity with the 'net is through the tun0 interface, configured with an IP address of 209.180.174.155, routing to 201.225.84.1, now set as the default gateway for the host.

For any trouble-shooting, check the logs (/var/log/daemon). When the PPP configuration is working as expected, bring down the link in interactive mode:

PPP ON soekris> close
ppp ON soekris> 
ppp ON soekris> quit
#

From here on, ppp can be started in -ddial mode, making sure the connection is restarted if it goes down for any reason:

# ppp -ddial qwest

This command can be added to the system start-up script /etc/rc.local. Of course, we prefer to run ppp under daemontools' svscan. (A discussion of a ppp service may be found on the djb way.) The run script for the service is simply:

#!/bin/sh
# ppp/run
# pppoe for qwest dsl
# ===
SYSTEM="qwest"
exec 2>&1
exec /usr/sbin/ppp -foreground ${SYSTEM}
### that's all, folks!

The use of -foreground mode lets svscan reliably monitor the ppp process, restarting it automatically if it should ever go down. By running under daemontools, the ppp service is also easily and gracefully signaled with the svc utility. For example, to bring the ppp service down:

# svc -d /service/ppp

And bring it back up again:

# svc -u /service/ppp

PF Configuration

One of the main reasons in using PPPoE with the Cisco 678 is to use the PF packet filter directly on our public IP address. To make sure PF kicks off afresh whenever the link comes up, put this pfctl(8) stuff in the file /etc/ppp/ppp.linkup:

# /etc/ppp/ppp.linkup
# linkup for dsl/pppoe connection
# ===
MYADDR:
  !bg sh -c "/sbin/pfctl -e -F all -f /etc/pf.conf"

### that's all, folks!

Otherwise it is just a matter of setting up the PF rules for your network as usual. As a starting point, here is a basic sample /etc/pf.conf for the network described here. Read the OpenBSD PF User's Guide for complete information. The main thing to remember when using PPPoE is that the external interface will be now be on tun0.

Conclusions

In our research for this setup, we found some older references describing performance degradation when using the PPPoE on OpenBSD.

These references may be outdated: on our modest 586-class, 266mhz Soekris box running OpenBSD 3.5, the overhead of pppoe appears to be negligible. In fact, we don't notice any difference in CPU or memory usage in the use of PPPoE with the Cisco as a bridge, compared to the original Cisco PPPoA setup.

If anything, the PPPoE configuration may even be faster. That is, by offloading the NAT filtering from the relatively slow Cisco CPU, and by doing all packet filtering on the more powerful OpenBSD gateway, overall throughput may even be improved.

The most important thing is that we have at last achieved our goal, hosting our own public IP address directly on our own packet filtering gateway. We now have all the security and control we like, and the bandwidth is fantastic. Our DSL connection is fast and sweet, and we are really enjoying our new life in Portland!