guinix TechNote: Remote Mail

Network design for "bandwidth-challenged" environments

Introduction

Electronic mail is the "classic" network application. Originally developed over 20 years ago --the Jurassic period of the Internet-- it continues to survive and thrive today as the single most essential, must-have communication technology. No matter who you are, where you are, you gotta have email.

In fact, just like the air we breath and the water we drink, email is now expected as a given in every habitable environment worldwide.

Individuals and organizations have grown to desire and depend on email for many very good reasons:

• efficient: high content to bandwidth
• accurate: good signal-to-noise ratio
• timely: leaps distances and transcends timezones
• easy to use: type and click

So, naturally, individuals and organizations seek to take advantage of electronic mail wherever possible. More and more, this includes remote deployments beyond the reach of conventional infrastructure. We refer to these as "bandwidth-challenged" environments, where Internet access is limited and/or very expensive.

While there are any number of possible scenarios calling for a remote mail deployment, we will describe the scenario we are most familiar with. That is, the "upcountry" field operation typical of a humanitarian relief and development effort in rural Africa. We are talking specifically about networking in the "bush".

Naive Deployment

The first stage of a humanitarian intervention is usually the preliminary field assessment. An individual or small team visit the affected region on a short term assignment to determine the feasibility and logistics outline for the longer term effort to follow.

This stage is marked by a high degree of mobility and rapid information collection. It involves frequent escorted travel within the affected region, conducting on-the-ground interviews with local officials, making health and environmental assessments within the target area, and getting a quick grasp of both the resources and problems at hand.

This first stage response generally has minimal infrastructure requirements. It usually calls for traveling light, carrying the minimum burden of equipment and supplies. In the way of communications technologies, the most one usually needs is a satellite phone (eg, Thuraya, Motorola/Iridium) for emergency voice communications, and possibly a small laptop computer for note-taking and data collection. Even this minimum of electronic equipment can be problematic and impractical in terms of power sources and recharging. For these reasons, full assessment reporting is usually deferred until after return from the field, such as upon arrival at the nearest major city having adequate infrastructure support: fax, phone, email.

Other issues also come into play. Because first stage assessments are often conducted under official escort, and in politically and often militarily sensitive circumstances, discretion is essential. In particular, special care must be taken with respect to the perception of the mission. For this reason, certain types of electronics and communication equipment may not only be undesirable, they may not even be permitted at all. In southern Sudan, for example, all equipment and personal belongings are searched meticulously by heavily armed soldiers at every frontier. Communications equipment in particular will receive extreme scrutiny, suspicion, and unwanted attention, with arbitrary, unpredictable results. And nothing will more cause more problems and embarrassment for a relief effort than a charge the team is engaged in covert reconnaissance or espionage.

Nevertheless, an assessment may be of sufficient duration that an attempt to support remote mail access is deemed both acceptable and desireable. In these cases, the individual or small team may be sent to the field with support for data communications via their Thuraya, Iridium, or Inmarsat equipment. This will generally be configured for what we will call the "naive deployment", and sketched in the diagram below:

{ internet }  <~~~~~>  (.)
                        |
                        |      +--------+
                      ===== ---| laptop |
                               +--------+
                               +========+

In this case, periodic connections are made --such as via satellite-- between the Internet and user's laptop in the field.

In other words, this configuration is designed to support only a single user at a time. If there is more than one user in the team, each will either share a single laptop for their email, or take turns connecting each personal laptop to the access device, each of which would need to be suitably configured.

While simple in concept, and possibly appropriate for single users mobilized quickly for short-term assignments, this architecture has a number of limitations and drawbacks:

• supports only one user at a time
• lacks packet filtering for network security and bandwidth control
• no access control --> no cost control

Most problematic is the lack of restrictions on type and volume of network traffic going over the connection, each kilobyte of which can result in rapidly escalating costs. No matter what network traffic the laptop is generating, and no matter whether the user is aware of it or not (and the typical Windows OS will generate voluminous unseen/unwanted traffic), it will go out unimpeded and billable over the metered link.

And it gets worse. In the most naive of the naive deployments, users will not be set up to use mail at all. Rather, they will expect to use webmail.

Mail vs Webmail

For many users, electronic mail has become synonymous with webmail --that is, the free web-based mail services from providers like Yahoo! and Hotmail. This confusion is regrettable, because webmail is nothing like regular email. In terms of network traffic and suitability for remote deployment, the two could not be more different:

	Mail	Webmail
upload protocol/port	SMTP/25	HTTP/80
download protocol/port	POP3/110	HTTP/80
user interface	email client	web browser
apparent speed	decent	SLOW!
bandwidth consumption	low	HIGH!

For the same message content, the use of webmail will consume 25 to 50 times the network bandwidth of regular email. This means that 10 kilobytes in messages of regular email --say a couple messages, one received and one reply-- will consume as much as 250k to 500k in bandwidth with webmail.

If bandwidth usage is metered, whether by connection time or in kilobytes, the costs will increase proportionally. Given a remote service plan with metered usage, such as an Inmarsat/RBGAN plan charging $.01/kilobyte, the cost differential is dramatic:

	Mail	Webmail
cost per 10k message	$.10	$2.50 - $5.00

And not only is webmail far more expensive than regular email, it is much more "painful" to the field user as well. In practice, links will never be optimal, the network will be slow, browsers will take seemingly forever to refresh pages, and connections will timeout and fail. Webmail is horrid for field deployments. So, this simple rule for your organization's mail support: don't ever use webmail!. Please.

On the other hand, as unsuited as webmail is for remote deployments, good ol' regular email is just about perfect. This is because the electronic mail protocols were originally developed at a time when networks were much slower and most links were not persistent. In other words, the conditions under which email was originally developed 20 years ago are now replicated by the "bandwidth-challenged" environments most often found in developing countries today. Regular email is the best example of a "store-and-forward" network application. It enables full-featured message reading and composition off-line, with minimal connect times required for message transfer, while providing highly reliable message delivery.

It should be a no-brainer, to always go with support for regular mail. If you need to set up a remote deployment, but are still fuzzy about the differences between mail and webmail, please see the resources section at the end of this article.

Optimal Deployment

After the initial field assessment, an operation will then move into mission delivery. This involves getting personnel and materiel on the ground at the affected site, with the necessary infrastructure to support the operation at all levels. Program staff, technicians, finance, logistics, administration: all the practiced machinery of intervention will be mobilized for the response. The deployment will now require food, shelter, sanitation, security and communications.

Unlike the short-term initial assessment, the mission delivery phase of the operation may be of indeterminate duration. It may range anywhere from several weeks to several years. We have even worked with operations that have been in place for decades. If there is a general rule, it is that deployments will be longer than originally anticipated.

Because of the importance and utility of electronic communications, it is worthwhile to get the email setup right from the start. And for existing installations, make it right now. That is, provide support for direct electronic mail to/from the compound in the field, for all key staff involved in the operation, while maintaining control over security, access, and bandwidth usage. The network design for an optimal deployment may be characterized by the following sketch:

{ internet } 
     |
     |
     v
   +---+
   |   |
   |   |
   |   |\
   +---+ \
   mail   \
   drop    \
            ~~~~~~~>  (.)
                       |      +---+
            <~~~~~~~   |      |   |
           /         ===== ---|   | <----> [ field lan ]
          /                   |   |
   smart /                    +---+
   relay/                     field
   +---+                      server
   |   |
   |   |
   |   |
   +---+
     |
     |
     v
{ internet }

In this case, all users in the compound are supported by a local area network, just as they would be in the office at headquarters. This field LAN can be wired --by using a hub and stringing some CAT5 cable-- or wireless --by popping a wireless card into the field server in AP mode.

The sketch also shows how field operations should be supported by "mail drop" and "smart relay" services hosted on a well-connected network elsewhere, such as in the USA or Europe, or a better-connected host, such as in the capital city of the country of operations. Although shown separately in the diagram, these services may easily be provided by a single host, either on the organization's network at headquarters, or by contract with a third party service provider.

The remote connectivity itself can be supported by any available cost-effective technology, from wireless/satellite communications provided by Thuraya, Inmarsat, DirecPC or VSAT, to HF radio on Codan equipment, to dial-up connections over local landlines to local ISPs. It doesn't matter to the design; just use whatever is available, proven, practical and reasonably priced.

Also, don't worry about getting broadband for remote operations. Even if you do find a broadband provider for the area, the service may be cost-prohibitive and is simply not necessary. (We have even set up these networks on the "slim-to-none" bandwidth of HF radio; see the Radio Email project listed in the resources section below.)

What is important here is the "field server", a small piece of hardware that provides the essential central component of an optimal deployment. The field server is configured with standard networking applications to provide the following services:

• local mail server:
• receives mail from the remote mail drop
• batches/queues outgoing mail to the remote smart relay
• serves mail to/from local users
• mail filtering according to policy: control viruses, spam, message size, etc.
• link management:
• automated control of access device
• configured to make periodic connections or to sustain full-time connections
• packet filter:
• ingress control, restrict incoming connections for security
• egress control, restrict outgoing connections for bandwidth/cost control
• network address translation (NAT), share connection among authorized LAN clients
• other services:
• DHCP, automatic configuration of LAN clients
• DNS cache, make DNS lookups efficient for all clients
• optional services:
• local intranet server
• local HTTP mirror for headquarters documents and other content
• secure, access-controlled FTP area
• local webmail
• time server
• captive portal, for offering "cyber cafe" to some users
• etc...

Note that none of the capabilities outlined here is available with the "naive deployment." Rather, the optimal deployment is designed specifically for servicing the needs and constraints of any operation located in a "bandwidth-challenged" environment.

Beyond the basics, the small server-based solution --in a well-considered, full-service deployment-- can be further optimized to both minimize bandwidth usage and maximize efficiency. For example, we configure these systems to use QMTP in place of SMTP, the Quick Mail Transport Protocol developed by Daniel Bernstein. QMTP reduces extraneous protocol overhead to an absolute minimum, and assures reliable delivery over even the slowest network connections. (QMTP is a freely available, included in every qmail installation. See the resources section below for more information on "the djb way".)

How Small?

Until recently, the idea of fielding a server-based solution into remote environments may have seemed impractical. That is, the equipment costs, power-requirements, and transport logistics may have been considered too high to make such a solution seem feasible or worthwhile. That's because we tend to think of servers as being big, complicated, power-hungry boxes, perhaps needing climate-controlled environments, and costing a lot of money.

All that has changed within the past year. The small servers we now deploy for field use have the following characteristics:

• small: about the size of a John Grisham paperback!
• reliable: solid state, fanless, industrial grade, cool running, quiet
• powerful: pentium-class processor
• low power: 15 watts maximum draw, easily solar/battery operated
• inexpensive: well under $500 per unit

These units are small, discrete, and can be located anywhere. You can pack 10 inside a suitcase, with plenty of room left over for your laptop and a change of clothes. They easily run the world-class, server-grade software standard on the most powerful of heavy-duty Unix systems, and one unit can easily support a remote compound with many dozens of users.

Add a wireless card and antenna for the instant wireless compound. Once configured and installed, they are readily maintained and upgraded by remote administration, from anywhere in the world. These small servers represent the state-of-the-art in appropriate technology, and are locally sustainable with a small investment in hands-on training. By leaving the server behind when the operation withdraws, an organization may thus help the local community bridge the "digital divide".

Summary

This article has described some of the key considerations and constraints for supporting remote mail in the "bandwidth-challenged" environments typical of a humanitarian assistance operation in rural Africa.

A naive deployment may seem expedient, but will give poor and costly results.

In contrast, the optimal deployment in such an environment will employ a small, server-based solution to provide efficient, equitable, secure, reliable, and cost-controlled access to scarce and/or expensive network resources.

By way of closing, and for whatever it's worth, we'll leave you here with some other random suggestions and opinions based on our own experience in the field:

1. Do not use webmail. Use tried-and-true, regular email. It is the perfect match for the constraints of bandwidth-challenged environments.

2. Deploy small server-based solutions. These provide access and cost control, network security, and can integrate the operations in the field with the network services of the home office.

3. Work smart! Invest in training and learn to use the technologies and resources appropriate to the operating environment.

4. Plain text is beautiful. Sending ".doc" word-processor documents as email is bad policy in any case; these bloated files are particularly devastating in bandwidth challenged environments. Train your staff to rediscover the beauty, consistency, and refreshing simplicity of mono-spaced, plain text communications, using the simplest of style guides:

   • *bold*
   • _italic_
   • * bullet

5. Consider the client applications. Disable all automatic updates, Real Player, all chats and instant messengers, no exceptions. And if you do nothing else, be sure to replace the standard Outlook/Explorer apps with the Thunderbird/Firefox suite on all systems in the organization. This one simple change will remedy a huge number of problems with security, errant traffic and system stability, both in the field and at headquarters.

Resources/Links

Radio Email Project

They said it couldn't be done; we proved them wrong. Read about our Radio Email project in West Africa, a wide area network based on HF radio, for the ultimate in bandwidth challenged environments. See also our write-up in the November 2002 Linux Journal, available online at http://www.linuxjournal.com/article.php?sid=6299.

http://thedjbway.org/

Our documentation project "the djb way" describes many server configurations designed especially for the bandwidth challenged environment. See for example:

• fastforward internationale
• tinydns internationale
• bushmail
• fetchmail

http://www.mozilla.org/

When you decide you have finally suffered enough from all the security and stability problems of Doubtlook/Exploder, just pull the plug. Installations of Thunderbird and Firefox are free and easy, perform beautifully, and will provide instant relief to an abundance of network misery.

http://www.guinix.com/

Need help with your remote operations? Still fuzzy about the problems with using webmail? Want the best small server-based solutions for your own optimal deployments?

We can help. Contact us at guinix international: info@guinix.com. We provide exceptional and experienced installations, support, training, and maintenance worldwide.