from the trenches

One ISP's Response to the Problem of Spam

by Nick Christenson
<npc@jetcafe.org>

Nick Christenson specializes in applied distributed computing and designing large and and scalable Internet systems. After two years as a senior architect at EarthLink Network, he is currently doing independent research.

and Dan Farmer
<zen@earthlink.net>

Dan Farmer does architecture and security research for EarthLink Network. He is interested in security and operations of large-scale/high-volume networks.

Introduction

As the Internet has become less of a research-oriented collection of computer-oriented acquaintances and more of a multinational business-driven communications medium, it has had to deal with problems that simply didn't exist before. One of these is "spam," the Internet equivalent of bulk mailings, junk faxes, and unsolicited telemarketing phone calls. This article details what we have employed in our effort to stop spam, with the obvious hope that the techniques used and the lessons learned will be of use to others.

Background

In 1994, two Arizona-based immigration attorneys, Canter and Segal, advertised their services by sending a message to each of the several thousand USENET newsgroups, whether their message was appropriate for the discussions taking place there or not. This unusual behavior earned them public scorn, and after they continued this practice, they were kicked off of one Internet service provider (ISP) after another. These days you can't enter a newsgroup without seeing such messages, typically called "spam," and in some newsgroups (especially the sexually oriented ones), they make up the majority of the traffic.

These sorts of tactics have not been reserved to USENET news. More recently, individuals have taken to harvesting email addresses from Web sites they maintain, open mailing lists, and USENET newsgroups so they might send unsolicited advertisements for various products and services. Officially known as UCE (Unsolicited Commercial Email) or UBE (Unsolicited Bulk Email), these practices have also been lumped under the general category of "spam," the definition of which has now been expanded to include essentially "all electronic garbage messages."

One of the more significant problems with spam is, unlike telemarketing or bulk postal mail, the sender pays very little of the cost of transporting the message. The spammer simply gives a mail host (often an ISP, due to its excellent connectivity, high-volume capacity, and a general difficulty keeping track of the huge amount of mail and news that passes through its system) a list of targets with a single message to send. The senders incur very little cost per message ­ essentially only their time and the cost to set up an account with an ISP. The host that relays the mail pays for the bulk of the transmission in bandwidth, service degradation, and cost of responding to the ensuing complaints. The target site also pays in loss of bandwidth, disk usage, connection costs, overflowing mailboxes, etc.

Of course, the cost that most people complain about is the expenditure of time and effort to sort, read, and delete the unwelcome communique. This can be especially painful when paying for access by the minute or by the byte. Most perceive the situation as unfair and feel that the costs of sending such messages should be paid by the initiator, not by the systems that are being abused or by the recipients.

Who We Are

EarthLink is one of the largest ISPs, serving approximately 450,000 customers, handling about two million pieces of email per day. By mid 1997, EarthLink was targeted by a large number of spammers, and the sheer volume of spam going through our networks was starting to have a significant impact on the performance of our services. At this point, we started taking legal action against the spammers and implemented the technical solutions that we present in this article. Our "zero tolerance" policy has the full backing of our upper management, and we go to great effort to ensure its implementation.

We are often asked if we claim to be so antispam, why don't we simply throw a switch and stop it from going through EarthLink's servers altogether? Unfortunately, there is no total solution. Although we have deployed a great many human and computing resources along with a large variety of technical and social tools, the spammers (new and old) keep misusing our resources almost as fast as we can stop them. The sheer amount of data flowing through our system defies implementation of a simple and all-inclusive mechanism by which to stop all network abuse.

We believe that the decision on what Internet traffic one wants to see or not see should be made by the end-user, if at all possible. And if one makes the decision to see or not see data coming from an individual source, one should be able to effect that decision. We also believe that anyone should have the right to refuse to provide service to those who use resources without paying for them and reserve the right to refuse to do business with anyone, especially those who consume a disproportionate amount of server, network, and human resources. People who use other people's resources to deliver their messages without express consent are, in essence, stealing. Those who deliver a message pretending to be someone they are not are, in essence, committing fraud. We do not support anyone's claim to the right to commit acts of this nature.

Caught in the Middle

At various times, the privilege of being the prime source of spam has made the rounds through the ISP community. AOL, Netcom, MCI, UUNet, EarthLink, Compuserve, the various regional Bell companies, and others have all held this distinction at one time or another. Not surprisingly, this title usually befalls an ISP during a time of extreme growth or other significant set of events in the history of the organization. ISPs are then forced to take their most extreme antispam measures when they have the fewest resources available. We believe that almost every entity that has been through this will recommend that organizations tighten up their systems and develop policies and procedures to deal with these situations before they arise. If an organization fails to do this when resources are available, it may be forced to face this problem when it is least able to.

Compounding the problem are the unscrupulous practices by many of the spam purveyors. Internet discussion groups are replete with stories of forged return addresses (so the advertising targets cannot complain to the true sources), hijacked servers (the spammers may use the computing resources of others to distribute their messages without permission), fraudulent identity claims (so that it is more difficult for filtering software to determine if the message comes from a source of spam), and procedures for removal of one's email address from these mailing lists that often do not work. It is no wonder that many have declared outright war on spam; consequently, spammers have had to resort to even more aggressive hit-and-run tactics to get their messages through.

Caught in the middle are the ISPs. The subscribers to these services generally don't want to see these messages, but if the ISP tries to filter these connections, it is accused of censorship. These organizations have the largest and most powerful email systems in the world, which their subscribers insist be more accepting than corporate servers (which can be restricted by policy and/or firewalled off), so they are natural targets for relaying by the spammers. Also, ISPs provide very inexpensive and convenient access as a jumping off point for the spammers to gain access to the Internet. These problems are not all easily solved, and it doesn't help, nor is it coincidental, that the tremendous increase in demand for Internet-capable professionals has been coincident with the time of ISPs' most rapid growth. Although this does not excuse an ISP that has ignored these issues, it is improper to believe that you can completely judge another situation without understanding the details of it.

The Current State of the Internet

On January 30, 1998, we decided to use a modified version of SATAN to conduct a quick-and-dirty technical survey of all the ISPs listed in CNET's "Ultimate Guide to Internet Service Providers" to get an idea of how many ISPs allowed unrestricted mail relaying. We decided to examine ISPs both because we are in the ISP business ourselves and most of the mail delivered on the Internet today is by ISPs. The results were staggering, quite simply, and go a long way to explain much of the reason why spam is such a problem on the Internet today:

number of ISPs checked:	597
number allowing unrestricted mail relaying:	320
% allowing unrestricted mail relaying:	53.6%

See the appendix for more on the details of the survey and the methodology used.

Although the final percentage of ISPs having open relays is only an approximate value, it's easy to see that the Internet is indeed a spammer's heaven. Even if a conscientious ISP turns off mail relaying or kicks a spammer off its systems, the miscreant can easily choose a different home or target to abuse. The story is analogous to security problems in general ­ the solutions are widely known, but apathy, the easy cushion of ignorance, the pain of change or implementation, the lack of auditing/verifying tools, or all of the above are preventing people from doing anything about it.

At the USENIX LISA 11 conference in San Diego in October of 1997, we held a BoF (Birds of a Feather) session on the problem of spam. It became apparent to us that the Internet community was extremely hungry for any advice on practical methods to reduce spam. In this article, we hope to provide practical information on how to help protect one's systems and to provide insight into how a site (be it an ISP or otherwise) might structure its response to these sorts of problems. Unfortunately there appears to be no single solution to this problem.

Technical Anti-Spam Methods

We can roughly divide our antispam efforts into technical and social methods. Although both have proved effective at reducing spam, the spammers are sometimes wily and always tenacious and have been very adaptive in combatting our efforts.

Technical efforts to stop spam are, of course, favored by us (being longtime Internet geeks). Realtime monitoring is fascinating, but very difficult, if for no other reason than the sheer volume of data flowing through our networks. So we've tried to focus on proactive methods whenever possible.

UNIX Mail Relay Filtering

This was part of the general philosophy of the early Internet: be a good neighbor; be generous in what one receives and restrictive in what one sends. Thus, if I sent an email message to <zen@trouble.org> from <npc@acm.org>, but sent it via the SMTP capable machine <mail.earthlink.net>, this server would have dutifully tried to deliver it to the <trouble.org> mail server for me.

This has been subverted by the spammers to (1) make somebody else do the hard part of delivering mail messages, (2) get around an administrative block of this spammer's organization, and/or (3) mask their culpability in this act. For example, let's assume I'm a spammer dialed up to my ISP, and I'm currently logged on to their service at <dialup666.faux_isp.net>. Now, I have a list of email addresses, maybe many thousands of them, to which I want to send an ad. I connect to <mail.good_isp.net>, claiming to be <niceguy@innocentcompany.com>. I then give a list of addresses to send my ad to, and the mail server will dutifully try to send the mail.

This open relaying policy is a friendly thing, in the best tradition of the Internet. On those rare occasions when an email message might get misrouted, machines will try to straighten everything out in a spirit of openness and cooperation. Before the rampant commercialization of the Internet, nobody thought twice about relaying mail for other sites, especially if they spanned networks. In fact, there were several sites that openly offered to do this as a public service. Unfortunately, this has been so badly abused by the spammers that the practice is on its way to being a distant memory on the Internet today. Here is how you can set up a system running the sendmail SMTP agent to prohibit unauthorized mail relaying for trivial and more complex cases.

Simple Case

The easiest way to prevent mail relaying is to simply disallow it altogether. The vast majority of hosts on the Internet can be set up this way. In fact, if the machine in question does not provide remote mail access (typically via the POP or IMAP protocols) or is not a central mail hub, this is undoubtedly the way the machine should be set up.

In order to block relaying in this manner, you need to be running the freely distributable version of sendmail, version 8.8 or higher. If you are not running at least this version, an upgrade is in order in any case because of the security problems associated with earlier versions.

In the sendmail.cf file, you simply need to add the following lines, stolen from the antispam rules at the sendmail Web site:

Scheck_rcpt
# anything terminating locally is ok
R< $+ @ $=w > $@ OK

# anything originating locally is ok
R$* $: $(dequote "" $&{client_name} $)
R$=w$@ OK
R$@ $@ OK

# anything else is bogus
R$* $#error $: "550 Relaying Denied"

These lines can be placed anywhere in the sendmail.cf file as long as they're not in the middle of another rule set. We like to put ours at the beginning of the file just before the "w" macro is defined.

You do not need to do anything more than add these lines and restart the sendmail daemon for the rules to take effect. These rules operate only on the envelope of the mail message, not the header, so that sendmail can't be fooled by forged headers. If the sendmail daemon receives email that either is not bound for the machine in question (that is, the machine in the "RCPT TO:" field of the envelope does not match the list of machines in the "w" macro of sendmail) or is not sent by itself, it rejects the connection with error 550 and the message "Relaying Denied." This is the way we recommend all machines that aren't mail hubs (e.g., desktop machines that need to run sendmail in daemon mode) be set up.

One final thing to note is that it's a bad idea for most hosts to run sendmail in daemon mode at all. Despite the fact that UNIX workstations come out of the box with sendmail installed (and almost always with relaying enabled), it is rarely necessary to run sendmail on more than a small fraction of computers on a given network. We urge system administrators to consider using the prospect of mail relaying as an impetus to rearchitect their mail systems, where appropriate. If your systems have been abused in this manner (if not by spam, it might be enough to remember that sendmail is one of the prime ways that intruders break into computers these days), you'll probably find this to be a relatively easy sell to your organization and/or management.

Advanced Case

What if a mail machine does act as a POP or IMAP server? In this case, there very well may be legitimate computers that need to use this machine to relay mail. You can specify a class ­ we use "W" ­ to be the hostnames of allowed relayers. If DNS isn't set up as well as we like, we additionally specify a class, "C" of all the Class C networks allowed to relay through us. The whole section to be added to the sendmail.cf looks like:

Scheck_rcpt
# anything terminating locally is ok
R< $- @ $=w > $@ OK

# anything originating locally is ok
R$* $: $(dequote ""$&{client_name} $)
R$=W $@ OK
R$+ . $=W $@ OK

# IP address ranges
R$* $: $(dequote "" $&{client_addr} $)
R$=C . $- $@ OK

R$@ $@ OK

# anything else is bogus
R$* $#error $: "550 Relaying Denied"

For this to work, we need to add the following definitions, also in the sendmail.cf file:

# file containing domains which are allowed to relay through us.
FW-o /etc/mail/sendmail.cW

# file containing legitimate client relayers by Class C prefix.
FC-o /etc/mail/sendmail.cC

The file /etc/mail/sendmail.cW might contain something like:

earthlink.net
trouble.org

and the file /etc/mail/sendmail.cC might look like:

208.197.253
207.217.91
207.217.118

That's all there is to it. Now mail will be rejected by this machine unless one of the following conditions holds:

• The destination host of the mail is in the contents of the "w" macro.
• The source host is in one of the DNS domains that are acceptable relays (from the file /etc/mail/sendmail.cW). As an example, the machine <trouble.trouble.org> would be allowed to relay; its domain, <trouble.org> is in the sendmail.cW file.
• The source host is in one of the Class Cs that are acceptable relays (from the file /etc/mail/sendmail.cC). As an example, the IP address 208.197.253.128, whose network is in the sendmail.cC file, would be allowed to relay.
• Mail is being sent from a process on this machine.

For those who aren't as familiar with the sendmail.cf file syntax, machines listed on a line that begins with Dw, Cw, or in a file called sendmail.cw make up the complete list of machines and domains for which the machine in question stores mail.

Of course, you can play with the rules, changing the Class C networks to Class Bs, removing the domain checking rules, or whatever is appropriate.

Testing these Rule Sets

Of course, you don't want to simply put these changes in and hope they work. They need to be tested. First, you should get an account on a machine from which relaying should not be allowed (not the machine that sendmail is running on!). For example, if the machine with the new relay rules is named <death.trouble.org>, you should Telnet to port 25 of this host from a disallowed host and verify that regular mail works but relaying doesn't by doing the following (the typed commands are in bold text):

fish.com % telnet death.trouble.org 25
Trying 208.197.253.134...
Connected to death.trouble.org.
Escape character is '^]'.
220 death.trouble.org ESMTP Sendmail 8.8.5/8.6.4 ready at Wed, 5            Nov 1997 14:52:55 -0800 (PST)
mail from: npc@acm.org
250 npc@acm.org... Sender ok
rcpt to: <npc@death.trouble.org>
250 <npc@death.trouble.org>... Recipient ok
rcpt to: <npc@acm.org>
550 <npc@acm.org>... Relaying Denied
quit

Mail is accepted for the local machine and denied for destinations not in the sendmail "w" macro.

Now we test from a machine that should be allowed to relay.

trouble.trouble.org % telnet death.trouble.org 25
Trying 208.197.253.134...
Connected to death.trouble.org.
Escape character is '^]'.
220 death.trouble.org ESMTP Sendmail 8.8.5/8.6.4 ready at Wed, 5            Nov 1997 14:52:55 -0800 (PST)
mail from: npc@death.trouble.org
250 npc@death.trouble.org... Sender ok
rcpt to: <npc@death.trouble.org>
250 <npc@death.trouble.org>... Recipient ok
rcpt to: <npc@acm.org>
250 <npc@acm.org>... Recipient ok

In this case, relaying was not denied whether the mail was to be delivered locally or not. These rules work and are probably safe to implement. As always, when making changes to the sendmail.cf you need to restart the sendmail daemon for them to take effect.

There are two things to note. First, the angle brackets when typing in the "rcpt to" line are mandatory. If these are omitted, you will always get "relaying denied." On machines without the Scheck_rcpt ruleset present, you will get "Sender ok" if they are omitted, but they are required by the SMTP protocol. Second, what is typed as an email address in the "mail from" line is irrelevant as long as it's a proper SMTP email address. This is never checked. Only the hostname/IP address of the sending host and the "rcpt to" line are ever checked by these relay rules.

You can do these things and more using the Scheck_relay rule set, but it's been our experience that using this rule set is buggier, slower, and rarely necessary. Nonetheless, information on these rules and others like them can be found at the sendmail Web site or in Sendmail , 2nd ed., by Bryan Costales with Eric Allman, published by O'Reilly & Associates. Both sources are highly recommended.

We also created a SATAN testing module that can be run on individual hosts or large networks. See "Auditing Tools" for more information on this.

News Backoff Algorithm

USENET news spam has been around longer than its email cousin, but it turns out to be fairly easy to implement a technical solution that greatly curtails it without serious side effects. The NNRP daemon is the process on an INN-based USENET news server that receives newsreader client connections; that is, this is the process on the news server to which the news client connects. The first thing you must do is make sure that posting is restricted to those hosts that should have access to the news server. The file that restricts access is called nnrp.access and is located with the rest of INN's configuration files. The exact location is operating system and version specific. Configuring this file is relatively simple; consult the nnrp.access(5) man page.

Additionally, what we've done is modify the NNRP daemon to keep track of how many posts come from a particular IP address in a period of time. If either the threshold for number of articles per unit time or the total number of articles is exceeded, the nnrpd daemon goes to sleep for a few seconds. The sleep time exponentially increases with each new successful post until a maximum value is reached; of course, if the posting attempts cease, nnrpd recognizes this and resets the counter after a period of time.

This algorithm has been very successful for us on our news service. We have drastically cut down the spam sent through our service without eliciting too many complaints. Indeed, we have found that if these (configurable) values are set properly, very few human posters will notice this policy change while any overly prolific automated posting program will quickly slow down to a crawl.

The backoff patches to INN's nnrpd (for INN version 1.4unoff4) are available to the public. Dave Hayes came up with the idea and wrote the patches while under contract by EarthLink Network. We expect these options to be added to the base INN distribution in the near future.

Of course, despite our best efforts and intentions, this can adversely affect some legitimate users. The first class of these is the frequent binary posters; their robot posting programs are, as far as this algorithm is concerned, indistinguishable from spammers. The second class of users that might notice this is those who use offline newsreaders. They slurp down piles of articles, read them offline, generate their responses, connect to the news server, and then send them up in one big batch. If the initial threshold is set to a number above what even an extreme news poster is likely to want to post in a single session, they won't be affected. Even if they are backed off, it may not be a problem for them because the postings will get through eventually, albeit slowly. If they are paying for connect time charges, though, this could be more than annoying.

The number of subscribers we have encountered who are legitimate users of the system but have been significantly affected by this change in service has been very small. For those who need to do robot posting, you could try to provide an authenticated NNRP service for them to post with. The details of such a news protocol have not been incorporated into an Internet standard, but the latest version of INN interoperates with several authenticating news clients.

Auditing Tools

We started by writing a simple script for parsing INN logs to assist humans in identifying spam. It examines news headers and reports on suspicious items ­ nonlocal email addresses, stereotypical spammy subject key words ("FREE," "PANTIES," and others too explicit to print in a family periodical), excessive cross-posting, a single person posting too many messages, etc. However, this is simply a reactive tool. Ideally, we want to stop the spam before it starts. Nonetheless, we believe it is impossible to stop completely, so maintaining a battery of reactive tools is necessary.

SATAN Module for Relay and News Checking

We also created a module for SATAN that can systematically walk through a network detecting if any hosts allow mail relaying or VRFY and EXPN queries or are running unrestricted NNTP servers that may need to be protected. This will be packaged with the next release of SATAN and will be available at <http://www.trouble.org/satan/spam.html>. It's remarkable, even with a fine system administration staff and a conscientious technical crew, how many systems continually keep cropping up with these sorts of problems.

Additional Logging ­ RADIUS Accounting

Another problem ISPs face is identifying service abusers in realtime. If caught "in the act," there is little room for argument as to whether they are responsible, and an immediate response can be taken. Making a mistake here is both unfair and bad for business; therefore, it is important to make this as accurate and efficient a process as possible.

Most dialup access equipment can be set up to use the RADIUS protocol to authenticate users' access to an ISP. An extension to this protocol, RADIUS Accounting, was designed to communicate accounting information between network access gear and an accounting server and is an exceedingly valuable tool that can greatly help in identifying a resource abuser, albeit after the fact. Unfortunately, this is still a problematic solution for us, primarily due to a lack of interoperability standardization and some very poor vendor implementations of this relatively new protocol. However, with the release of RFC2159, which standardizes RADIUS Accounting, we have further hope that sites will be able to support a RADIUS Accounting service stably across a variety of dialup access platforms.

If you have a large number of dialup ports, setting up a RADIUS Accounting server can require a significant amount of planning and resources. The service must be stable, accurate, and reasonably speedy, or it isn't going to do any good. Therefore, some thought and planning about how this service will be set up and maintained, as well as about what tools need to be written to access this information, need to be expended. If you are a smaller ISP or otherwise have a small dialup pool to maintain, the RADIUS Accounting logging code in the standard distribution can suffice, but larger services need to plan carefully for the very large volumes of data this service can generate.

As a final warning, even with RADIUS Accounting, we (like many other ISPs) have an additional logging problem. Because we lease POPs from other ISPs (primarily UUNET) and therefore don't own all the resources involved, our people trying to identify the abuse of our systems will not always be the ones who are able to identify the account. This discontinuity makes it doubly important to have a single point of contact internal to EarthLink to manage and facilitate all communications for any network abuse.

Social Antispam Methods

Spam Cowboy

Because this is a relatively new job description, it's nearly impossible to find people who have any experience to fill the position. Every organization must either develop these resources in-house or steal them from another ISP. Not only must candidates for the position have a good level of technical competence, a well-developed sense of ethics, and a good set of social skills (interacting with individuals at other ISPs and organizations, law enforcement personnel, and customers demands this), but they also must have a very thick skin. Complaints from the general public, griping from the subscribers, and telephone calls from the abusers (which can even take the form of death threats!) are a daily occurrence. It really is a tasking job, and it is difficult for those who haven't experienced it firsthand to understand its demands.

Punishment

One controversial measure we employed was to modify the EarthLink Subscriber Acceptable Use Policy (AUP) to include a provision to charge $200 to a subscriber who commits acts of network abuse, which include spamming as we have described it here. Employing this was not without considerable controversy within EarthLink, and we had to lobby our legal department and upper management to get it passed. Fortunately, having worked closely with them throughout the process, we haven't experienced any negative legal fallout from imposing these fines.

Collecting the fines turned out to be very simple: we charge the credit card used by us for billing. From the start, we consciously tried to reduce the number of "friendly fire" accidents by focusing on the more egregious offenses and using these as examples. This way we've managed to hit back hard against the really bad offenders while sending a message to casual spammers that they should think twice before using our service for these purposes.

As with any policy change, it was vital to get the message out to our subscribers. We modified the AUP on our Web page, sent email to all of our subscribers detailing the changes, and printed an article in our bimonthly newsletter, Blink, which is also online. The response we received from our subscribers regarding the changes we made in this regard have been overwhelmingly positive. The problem is well known and understood, and our candid description of what we were doing about it and how it would affect our customers was very well received.

This has been a big success for us, and we heartily recommend that other ISPs to consider adopting a similar measure. It does require some serious work to accomplish, but we have found it to be more than worth pursuing.

Negative Solutions

There are several sets of measures that folks have taken on the Internet in an attempt to deal with spam that weren't mentioned here because we do not like them. Some so-called solutions are, in our opinion, not solutions at all, for they advocate an eye for an eye (or worse) philosophy. We feel that in some cases these "solutions" are at least as worrisome as the spamming problem they're attempting to solve, and we do not recommend that they be adopted.

Terrorism

Foremost among these are what can only be described as terrorist attacks: Ping-of-Death, mailbombing, smurfing, hacking, and other denials of service or outright attacks against both the spam purveyors and the unwilling accessories to their offenses. These attacks are worse than the spammers, for although they are typically out for monetary gains, terrorists have real malice behind their actions with an intent to injure. In addition, these efforts can often have far-reaching and unintended consequences, not only to their target, but also to innocent victims along the path of destruction.

Black-Hole Routing

Another so-called solution that some folks have adopted to combat the spammers is to fail to route their networks; at the last LISA, one such group claimed to have a set of participants that could eliminate a target's capability to see about 20% of the Internet by blacklisting it. Although we believe that a terminal or endpoint network certainly has the right not to accept traffic from places it does not wish to communicate with, potential abuses have made this a practice we cannot support.

First, transit networks should not do this, only endpoint networks. As an ISP, we should not prevent the folks to whom we provide service from being able to contact anyone that they choose on the Internet. Under no circumstances should we censor their access without their express consent. If they ask us to filter, that is an entirely different matter and acceptable.

Second, on more than one occasion, legitimate users have been cut off from a significant portion of the Internet accidentally, despite their innocence of any form of network abuse. We cannot, in good conscience, support a system where this is such a strong probability.

Third, this solution, as it is currently implemented, bestows a great deal of power to an individual, so a potential for abuse is there. Even though we don't suspect that any unethical activity is likely, the mere possibility of this is distressing.

In addition, the misconduct of these individuals can make the spammers appear to be victims, rather than the network abusers that we believe they are.

We do not support these sorts of activities in any way, shape, or form, implore the employers of these methods to desist, and call for other legitimate organizations to decry these methods as well.

Future Work

There's a lot more still going on that will or may build on the efforts we have outlined here. Unfortunately, they aren't all positive or constructive efforts, in our opinion.

Sendmail's No-Relaying Default

Starting with sendmail 8.9, sendmail will have mail relaying off by default. This should cut down the amount of open relays by a considerable margin, because it is still by far the most popular mail delivery agent on the Internet.

SMTP Backoff

Since our USENET news backoff solution was so wildly successful, we've turned our attention to doing this for SMTP as well. We're currently talking with Eric Allman with the hope that he will add these capabilities to sendmail. We would like to see the mail local recipients stream through unaffected while outbound mail being relayed through the mail system is subject to the same basic kinds of backoff procedures we use for news. There's no completion date on this, but you might want to start looking for it sometime in late 1998.

Realtime Monitoring

We have the unenviable (from a security perspective) situation of having a large amount of network traffic and bandwidth that will only grow larger. Trying to monitor 50MB a second of email in realtime is a difficult task at best; we have yet to find something that can keep up with this volume of traffic. However, with the recent release of Network Flight Recorder, a programmable, high-speed, network-monitoring tool, we are hoping to put more significant effort into solving this problem. Having a tool that could warn us of network abuses as they occur could help us greatly mitigate our current dilemma. It remains to be seen whether this or any network-monitoring tool can keep up with present and future load.

IP Caller ID

We are envisioning a system whereby a unique identifier is handed out to a computer with a dynamic IP address when it signs on. This information can be used to grant or deny access to individual client machines within a single piece of dialup access gear. In this way, multiple ISPs could share a common dialup access provider without making themselves vulnerable to network abuse by the other ISP subscribers using IP-based authentication alone. This idea is hardly even in its infancy, but it is a technical possibility that might be worth pursuing.

ISP Version of NCTDE

Starting in December 1997, the telephone long distance phone companies (IXCs) put into full service a blind database maintained by an external entity for the purpose of coordinating information on households that are bad credit risks, that is, jump from one long distance provider to another without paying their bills. This database is known as NCTDE (National Consumer Telecommunications Data Exchange). Because of the way this database is structured, the phone companies have obtained an antitrust exemption from the Department of Justice. A nearly identical system, called NTDE (National Telecommunications Data Exchange), has been in service for over two years to track businesses in this matter.

No research or work that we're aware of has been done on this yet, but it seems reasonable that a similar service might be implemented in the ISP world and that this service might be expandable to track spammers.

Alternate Mail Delivery Systems

Although there have always been alternatives to sendmail, there has never been a serious challenge to its supremacy as the UNIX mailer of choice on the Internet. However, two mailers have stirred up quite a bit of interest and popularity: Qmail by Dan Bernstein and the upcoming VMailer by Wietse Venema. Both have various antispam features and, of course, have mail relaying off by default.

Resource Sharing Among ISPs

Resources can include realtime information, as well as personnel, hardware, and software. Rapid and easy communication among ISPs on resource abuse may have a great deal of promise in reducing the overall impact of the spammers on the Internet, although there are significant technical and legal barriers to making this happen. However, we hope groups like IOPS will help establish a dialog on how the ISP industry as a whole can cooperate to reduce the spam problem.

Laws

For better or for worse, it is primarily through legislation that governments have such an enormous impact on how the Internet functions. Opinion is currently split between those who believe that a legal approach would be a productive way to attack the spam problem and those who believe that government intervention is more to be feared than invited. We believe that both of these viewpoints are reasonable and are ourselves split on this issue.

Until now, the US Government has mostly let the Internet grow and evolve in a fairly unfettered state. This, combined with the overwhelming success of the Internet, has some fearing that if the government does intervene on the issue of spam, it will be an invitation for even more legislation on other issues that will have undesired consequences.

The often cited junk fax law (47 USC 227) has had a powerful and beneficial effect on curbing this nuisance in the fax world, and it's easy to understand why many folks believe that extending it to cover UCE and USENET news spam would be very beneficial. If an antispam law would have an analogous impact to the junk fax legislation, it would be hard for anyone who opposes spam, no matter how anarchistic he or she might be, not to concede that such legislation is a good thing. Indeed, even if some small interference by government in other Internet areas were a consequence, on balance, it might well be a price worth paying.

The bottom line is that any debate on whether the spam problem should be addressed via legislation leads to two key questions. First, would the legislation be effective in solving the problem? Second, is the price of the direct and indirect consequences of this legislation worth paying? Obviously and unfortunately, the answers to these questions are unknowable at the present time.

Virtually everyone does agree that it is by no means certain that good legislation will result from any governmental legislative efforts. However, because it is the nature (and, indeed, the vocation) of politicians and lawyers to legislate on topical issues, some laws are almost sure to be forthcoming. And if poor legislation does get passed, it would almost certainly be more difficult to undo this and obtain effective legal solutions in the future.

Almost no laws have been passed anywhere in the world to cover spam. It is absolutely in our best interests (out of self-preservation, if nothing else) both to try to understand the issues and to guide our legislators by written and spoken commentary. If the issues involving UCE are important to you, we urge you to educate both yourself and your legislators, regardless of your personal stance.

Here is a listing of the most prominent pieces of pending US legislation and some brief commentary on how we view their relative merits. Some good, if partisan, overview of these bills is available at the CAUCE (the Coalition Against Unsolicited Commercial Email) Web site. The focus of all these bills is on email, not USENET spam.

• US H.R. 1748 (the Smith bill). This is an attempt to extend the junk fax law to apply to UCE. This bill is the one that most antispam organizations (like CAUCE) support. Of the national legislation that has been proposed so far, this is by far the most virulently antispam, and the one we feel has the greatest merit. Read the text of this bill or read Representative Smith's commentary on his legislation.
• S. 771 (the Murkowski bill). The Murkowski bill would require that each UCE have the word "advertisement" as the first word in the subject line of each message and that ISPs install and filter messages for their customers or face legal actions and fines by the Federal Trade Commission. This means that ISPs would have to pay significant costs to support the spamming infrastructure. Although we don't particularly want to see this bill pass, at the very least, it would allow easy filtering of spam. On the negative side, it would have further deleterious effects on the performance of email, because of further processing, filtering, and increased volume of UCE due to legal sanctioning. Finally, it places the burden on the consumer to get off mailing lists (the so-called "opt out" provision, which requires a consumer to send mail to the advertiser to be removed from a mailing list), rather than having advertisers ask consumers if they want to receive their mailings. All considered, we cannot support this piece of legislation. Read the text of this bill or read Senator Murkowski's commentary on his legislation.
• US S. 875 (the Torricelli bill). This bill has the support of the Direct Marketing Association and other "legitimate" members of the marketing community (technical people always put quotation marks around the word "legitimate" when talking about marketing people) who claim that they don't condone the sort of spam that has gone on so far, but feel that UCE has enormous marketing potential and should remain open as an advertising option. It explicitly disallows the forging of headers, but includes the "opt out" provision that we are against. It appears to be slightly stronger than the Murkowski bill, but not enough so that it would be truly effective in combatting spam. Although it does have some merit, we oppose this bill. Read the text of this bill or read Senator Torricelli's commentary on his legislation.
• H.R. 2368 (the Tauzin bill).This bill is the weakest of the four. Like the Murkowski bill, it would require identifiable subject lines and have an "opt out" policy. The bill would also create a panel of people to specify responsible Internet marketing practices. The biggest downside of this bill is that adherence to its articles would be completely voluntary. Obviously, this does nothing to help stop spam. We strongly oppose this bill. Read the text of this bill. Currently there is no commentary, but you can examine Representative Tauzin's Web page.

Of course, even with good legislation, there are problems. First and foremost, these laws would be applicable only within the United States. A major reason there are currently few problems with junk faxes that originate outside the country is because sending these is prohibitively expensive (although perhaps junk faxers just haven't figured out how to use the Internet for this yet). Because the Internet currently employs a distance insensitive pricing model, legislative action in this country could simply prompt a migration of the spammers to offshore locations. Not only would this lead to further loading of already congested international links, but it could lead to either an international law enforcement nightmare or, more likely, a situation where nobody can take effective legal action.

In any case, unless the organizations or individuals sending the UCE are held accountable, there's not much anyone can do about overseas or domestic spamming. For all of these reasons and many more, technical solutions to UCE certainly are easier to implement! And although good legislation may help reduce spam on the Internet, it is our opinion that even very good new laws will not completely solve the problem, and strong technical mechanisms will still be the first line of defense.

Conclusion

For better or for worse, spam is here to stay. Electronic mail and news are each simply too effective a communication tool to be ignored by people wanting either to make money or to spread a message to the masses. However, we are not advocating total elimination of UCE; we simply want people to use responsible and acceptable distribution practices. Abusing resources is not acceptable, and that is what this article is trying to help prevent.

Unfortunately, even if everyone implemented all the solutions we discussed in this article, spam would continue. It is always going to be possible to misuse the Internet because its two main strengths, power and flexibility, are particularly easy to exploit. But we feel that some measures and practices are better than others, and if everyone (or even only ISPs) adopted them, they would definitely reduce the total amount of spam on the Internet. The most significant positive changes are:

• of open mail relays
• strict USENET news backoff algorithms that prevent posters from flooding the Internet
• a significant but realistic fine

The problems caused by spam and UCE are real and significant. The entire Internet has been affected by this malady, but there are things we can do to alleviate the problem. We believe that a combination of existing technical solutions, many of them described in this article, future technical work, and cooperation among Internet service providers can significantly impact network abusers without unduly affecting responsible Internet users. Much work still needs to be done on this, but we have shared our experiences and the techniques that EarthLink Network has found useful in combating these problems, and we hope to start a dialog on how we can further reduce the problems that spamming causes for the Internet as a whole.

Final Note

This article is a work in progress. It represents the version as of early February 1998. Updated versions can be found at <http://www.trouble.org/security/spam_war.html>.

Acknowledgments

We'd like to thank David Beckemeyer for information on his IP Caller ID proposal, Tim Bosserman for providing information on RADIUS Accounting, Dave Hayes for his work on the news backoff patches, Harris Schwartz for information on the results of our efforts, and Lisa Hoyt for her unceasing work in making the spammers leave EarthLink Network for some other provider, for now . . . .

Appendix

We utilized a new SATAN module that used a very simple method to determine if a host allowed unrestricted mail relaying. Our methodology:

• An untrusted/reasonably random system was used to do the testing. Trouble.org has no special privileges or status with any of the ISPs tested.
• Nslookup was used to examine the DNS records of the target ISP, and all MX hosts were collected.
• The MX host's SMTP daemon was connected, and the following commands were issued:
• helo tsunami.trouble.org
  mail from: <zen@tsunami.trouble.org>
  rcpt to: <zen%tsunami.trouble.org@target.host>

Note that the percent token (%) was used instead of the at sign (@) to determine if the system was a mail relay. Simple antimail relay rules in the SMTP daemon (and those proposed initially by sendmail.org) would allow this sort of mail to be delivered; we found several sites that blocked the latter method but not the former.

The return codes were then examined. If an appropriate response was received (250, etc.), the host was assumed to be an unrestricted mail relay. Obviously, it would be best to actually send the mail and see if it was delivered, rather than this partial test, but the difficulties of scanning arbitrarily large networks from arbitrary hosts make this a more palatable (at least, significantly easier to program) solution.

The method used is error prone in many ways, however. Although none of them are fatal, false positives could occur in numerous ways:

• Incorrect DNS information. Some hosts had no MX hosts listed at all.
• MX records might point to systems outside the ISPs control.
• Duplicate MX records. Smaller ISPs are sometimes merely repackaging of a larger ISP.
• The SMTP daemons could return a false positive or be non-RFC compliant.

False negatives are also possible, due to the following:

• Networks or individual hosts could have been down. We didn't ping the target systems to see if they were up because of the proliferation of packet screens that block ICMP requests.
• The ISP might not even exist. The list we got the targets from, like anything on the net, is virtually guaranteed to be out of date
• We examined only the MX mail hosts. If we had examined the entire ISP's network, we would presumably have found more mail relays than we found in this survey.

Ideally, we would hope that the errors would either not come up or simply cancel themselves out, but in practice, results found in this survey are likely an upper limit, probably within 10% of the final total (this is not an exact science!).