The FreeBSD Diary |
|
(TM) | Providing practical examples since 1998If you buy from Amazon USA, please support us by using this link. |
|
Fighting spam with pf
9 July 2007
|
|
This article was originally published at ONLamp as Greylisting with PF. This article will show you how I am using PF (the the Packet Filter from the OpenBSD project) and spamd (also from OpenBSD) to implement greylisting and greatly reduce incoming spam. This solution is completely MTA agnostic. You do not have to make any changes to your existing mail server configuration to use spamd/pf. In fact, you can easily use pf and spamd to guard any SMTP mail server. Yes, even MS Exchange. I first read about PF when reading Micheal Lucas's Absolute OpenBSD. If you have never read one of his books, I urge you to do so. His writing style is very clear and easy to follow. In the book, there are about thirty pages on PF, yet there is so much packed into that short chapter that I knew then and there that one day I'd start using PF. That day came a few weeks ago. My gateway at home has been happily running PF since early October. More important is my happiness. One of the best features of any software product is simplicity of use. PF is simple to get started with. And easy to extend once you need the more advanced features. A few weeks after implementing PF on my home gateway, I attended NYCBSDCon 2006 and listened to Bob Beck's talk on spamd. It was so easy that I just had to try it. So I did. On a related note, I'll also give you a short introduction to OS fingerprinting and how you can use pf to block/pass packets based up on the sending OS. |
|
TLS - be aware
|
|
If you are planning to use TLS (transport layer security) on your mail server, be aware that spamd has no TLS. If your MTA requires TLS, or communicates with other MTAs that require TLS, then you will probably have to whitelist those MTAs and perhaps make TLS optional on your MTA. |
|
A bit how greylisting works
|
|
spamd and PF work together. PF will direct any known good clients directly to the mail server. Similarly, known bad clients will be sent to the tarpit, a slow responding mail server, which does not add much load to your system. Finally, new clients, not known to be either good or bad, are asked to try again later. With the above, you have three lists of clients:
The key to greylisting is knowing that good and kind mail servers do not mind being politely asked to come back later. This is part of the STMP protocol. Spammers, however, are cheap and nasty. They don't like this. They probably won't come back later. Why? Queue management is tricky. If you're sending to millions of email addresses, why worry about a few messages that you cannot deliver? Just skip along to the next one. Spammers work on volume. We exploit that characteristic with greylisting. |
|
Implementing PF
|
|
My gateway was running FreeBSD 6.2 PRELREASE. PF is also available on DragonflyBSD, NetBSD, and OpenBSD (of course!). The FreeBSD Handbook has a good Introduction to PF. There are several ways to enable PF, including compiling it into the kernel, or loading the module. I added the following options to my /etc/rc.conf file to ensure PF starts up at boot time. It also starts the logging daemon. pf_enable="YES" pflog_enable="YES" pf_rules="/etc/pf.rules" NOTE that I have chosen a non-default value for my PF rules. The default value, as found in /etc/default/rc.conf is /etc/pf.conf. To avoid any merge conflicts with mergemaster(8), I chose to use a different file name. The default install comes with many fine examples in /etc/pf.conf and I urge you to read them. Designing your PF rule set is beyond the scope of this article. The OpenBSD project has a good example. |
|
Enabling PF
|
|
The primary interface between PF and the outside world is pfctl. To load PF on a running system, issue this command: # kldload pf # kldstat Id Refs Address Size Name 1 8 0xc0400000 6721fc kernel 2 1 0xc0a73000 58554 acpi.ko 3 1 0xc4eb5000 16000 linux.ko 4 1 0xc5e20000 2d000 pf.ko Now that you have PF loaded, let's look at what this gives you. The following command shows you all the filter parameters: # pfctl -s all No ALTQ support in kernel ALTQ related functions disabled FILTER RULES: INFO: Status: Disabled Debug: None Hostid: 0x595cedd1 State Table Total Rate current entries 0 searches 0 0.0/s inserts 0 0.0/s removals 0 0.0/s Counters match 0 0.0/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s TIMEOUTS: tcp.first 120s tcp.opening 30s tcp.established 86400s tcp.closing 900s tcp.finwait 45s tcp.closed 90s tcp.tsdiff 30s udp.first 60s udp.single 30s udp.multiple 60s icmp.first 20s icmp.error 10s other.first 60s other.single 30s other.multiple 60s frag 30s interval 10s adaptive.start 0 states adaptive.end 0 states src.track 0s LIMITS: states hard limit 10000 src-nodes hard limit 10000 frags hard limit 5000 Notice that the status indicates PF is disabled. To enable PF, issue this command: # pfctl -e No ALTQ support in kernel ALTQ related functions disabled pf enabled There are no filter rules loaded. To test your rules, without loading them, try this command: pfctl -n -f /etc/pf.rules Any syntax errors will be brought to your attention. The following loads the rules: pfctl -f /etc/pf.rules Read the man page for more -s options. You can pull out NAT, RDR, filtering, etc. |
|
Very simple rules
|
|
This section shows very simple rules. It is a cut down version of what I use at home. fxp0 faces my ISP. fxp1 talks to my home network. This setting in /etc/rc.conf allows the gateway to forward packets between the two NICs: gateway_enable="YES" And the rule set is: A few notes on specific lines above:
I hope that's enough to get you started with your own ruleset. |
|
Treating different OSes differently
|
|
Equality is a nice concept. The theory is great. The practice is not universal. Especially when it comes to operating systems. Why discriminate? I will answer that question in the form of a story related to me by someone who uses OS fingerprinting at home to lessen his domestic workload. To protect the guilty, this person will be know as Phil. Phil uses BSD in his everyday work. At home, his kids use BSD. Their machines also dual-boot with Windows in case they need to do specific things for homework etc. In general, they are to use BSD. When using Windows, they are only allowed to use the web, nothing else. Why do these rules exist? To lessen Phil's workload. He doesn't want to be removing viruses and spamware all the time. How do you keep them off the Windows computers? You keep the Windows computers off the Internet. Sure, rules are nice, but some kids, and some adults, are known not to follow the rules. It would be nice if there was some way to enforce this at the firewall. Enter OS fingerprinting. OS fingerprinting is not new. nmap, for example, uses it. From man pf.conf: Passive OS Fingerprinting is a mechanism to inspect nuances of a TCP connection's initial SYN packet and guess at the host's operating system. It is pf's ability to detect the Operating System at the other end of the TCP/IP connection that allows Phil to prevent his kids from breaking the rules. To illustrate this, I'll try blocking incoming queries based upon the OS you are using. On my webserver at home, I will create a website: http://pf-fingerprinting.example.org/. To demonstrate that the website is actually running, you will be able to browse freely to the above URL. But if you try http://pf-fingerprinting.example.org/:8080 you will be blocked if you are using Windows. NOTE: I have since taken down these example webpages. Sorry. :) How do I do this? I'll show you the pf rules, but the Apache setup is outside the scope of this article.Redirect incoming traffic to the webserverThese rules redirect incoming traffic to from the gateway to the webserver: rdr on $ext_if proto tcp from any to $external_addr port http -> $webserver rdr on $ext_if proto tcp from any to $external_addr port 8080 -> $webserver As you can see port 80 (http) and port 8080 are both redirected to my webserver. Block/pass traffic on those portsThese rules will pass or block the traffic based on port and OS: pass in quick on $ext_if inet proto tcp from any to $webserver port http flags S/SA synproxy state block in quick on $ext_if inet proto tcp from any os windows to $webserver port 8080 pass in quick on $ext_if inet proto tcp from any to $webserver port 8080 flags S/SA synproxy state The first line allows traffic to flow freely from my internal nic to the webserver, on port 80. The second line blocks all traffic from any Windows machine headed towards port 8080 on my webserver. The last line passes all traffic on port 8080. The above line contains a quick directive so if the client OS is Windows, subsequent filter rules have no effect on the packet. I originally wanted to redirect different OS connections to different webservers, but the OS directive is not available on the rdr statement. |
|
Getting spamd enabled
|
|
For FreeBSD, spamd comes as a port. The easiest way to install it is to have a fresh copy of the FreeBSD ports tree and issue these commands: cd /usr/ports/mail/spamd make install clean To enable spamd, get greylisting going, and get verbose logging, add these entries to /etc/rc.conf: pfspamd_enable="YES" pfspamd_flags="-g -v" See man spamd for more details on the various options you can specify. If you are using greylisting, you also need to issue this command: mount -t fdescfs fdescfs /dev/fd This mount allows spamlogd to update the spamd table. To ensure this is mounted at boot time, add the following to /etc/fstab: fdescfs /dev/fd fdescfs rw 0 0 To ensure you have the latest versions of the spam blacklists, you can refresh them once per hour with this line in /etc/crontab: 48 * * * * /usr/local/sbin/spamd-setup The spamd-setup utility adds blacklists by adding addresses to the pf table <spamd> according to the instructions in /usr/local/etc/spamd.conf. To distribute the load a bit, and not have everyone hitting the servers at the same time (e.g. 48 minutes past the hour, or at the top of the hour), change 48 to whatever minute is is when you enter the crontab entry. You'll need a copy of spamd.conf: cp /usr/local/etc/spamd.conf.sample /usr/local/etc/spamd.conf You may wish to amend spamd.conf according to your needs. Personally, I changed the following: I also added this to /etc/syslog.conf so I could see the log from spamd: !spamd daemon.err;daemon.warn;daemon.info /var/log/spamd I also issued this command to create the file: touch /var/log/spamd Remember to HUP syslogd so it reads your changes and takes appropriate action: kill -HUP `cat /var/run/syslog.pid` Although your log file will be empty at this point, here are a few entries that appeared after it had been running for a while. No, those aren't the real email addresses from my logs, but they are close. |
|
Known good mailers that have trouble with greylisting
|
|
There are some problems with greylisting. Have a read of the whitelisting section at greylisting.org. I have taken their whitelist and added it to my whitelist (/usr/local/etc/spamd-mywhite). Also in that file are my own mailservers and any special places which are immune from any spamd intervention. |
|
Directing things around the tarpit
|
|
Here are the rules I added to /etc/pf.rules:
Notes for the above:
NOTE: The use of pass on the RDR rules is significant. The following is borrowed from the OpenBSD PF FAQ
In short, if you're on a whitelist (either spamd's whitelist or my whitelist), you go straight to the mail server. Everyone else, goes to spamd. On your first visit to spamd, you are asked to come back later. If you do, then you're asked to try again, and you are added to the whitelist. By the way, after making changes to /usr/local/etc/spamd-mywhite, to tell PF to take notice of those changes, I issue this command: $ pfctl -t spamd-mywhite -T replace -f /usr/local/etc/spamd-mywhite No ALTQ support in kernel ALTQ related functions disabled 24 addresses added. 39 addresses deleted. |
|
Starting spamd
|
|
To start spamd manually, issue this command: /usr/local/etc/rc.d/pfspamd start The first time you run spamd, it may take 10 or 20 seconds for it to come back to the command line. Be patient. If you get the following error message: then you probably forget to create /usr/local/etc/spamd.conf (see above) or you removed the# /usr/local/etc/rc.d/pfspamd start Starting pfspamd. spamd-setup: Can't find "all" in spamd config: No such file or directory all section from it.
You will also want to start pflogd, so that the logging works: /etc/rc.d/pflog start After successfully starting spamd, you should see stuff like this: # ps auwx | grep pf nobody 94067 ?? Ss 11:20AM 0:00.03 spamd: (pf <spamd-white> update) (spamd) root 94282 ?? Is 11:31AM 0:00.00 pflogd: [priv] (pflogd) _pflogd 94286 ?? S 11:31AM 0:00.00 pflogd: [running] -s 116 -f /var/log/pflog (pflogd) I have removed some of the columns from the above display to make it fit better on the page. On a side note, I'd like to see spamd running as something other than nobody. Perhaps I'll work on that later. spamd-setup maintains the <spamd> table shown on line 1 of the PF ruleset found in a later section of this article. To view the contents of this table, issue this command: pfctl -t spamd -T show |
|
Getting things from the greylist into the whitelist
|
|
You don't have to worry about moving items from the greylist to the whitelist. spamlogd will take care of that for you. If you're setting this up for the first, time, you can get spamlogd running with this command: /usr/local/libexec/spamlogd To ensure spamlogd starts at boot time, include this in /etc/rc.conf: pfspamlogd_enable="YES" spamlogd updates the spamd database (/var/db/spamd). When it sees a successful connection spamd, in turn, uses this database to decide whether someone is on the whitelist or greylist. In order to provide spamlogd with the information it needs, you must log your mail server activity. See lines 13 & 14 above. Read all the details in man spamlogd. If spamlogd does not start, it is probably because pflogd is not running. See above for starting pflogd. This is what spamlogd looks like when it is running: # ps auwx | grep spamlogd root 94345 ?? Ss 11:36AM 0:00.00 /usr/local/libexec/spamlogd root 94349 p2 S+ 11:36AM 0:00.00 grep spamlogd |
|
A sample greylisting
|
|
In this section, I will send a message from a non-whitelisted server and demonstrate how the server moves from the greylist to the whitelist. I will be sending from dan@zip.example.org to dan@nyi.example.org. For your information, zip is running sendmail, and nyi is running postfix. In both cases, that is completely irrelevant to greylisting. Here is an extract from the sending mailserver. I guess I should point out that this server is in New Zealand and the one I'm sending to is in New York. A new host, zip.example.org, previously unknown to nyi.example.org, attempted to send email. spamd on nyi correctly asked zip to try again. Checking the mail queue on zip, you should see something like this: Looking at the logs on nyi, I see this in /var/log/spamd: Furthermore, you can see that zip is greylisted by issuing this command: I waited. Shortly thereafter, zip tries again, and again spamd asks it to try again. Here is the log entry from zip, the sending mailserver: Checking on nyi, I looked in the spamd database again: There it is, clear as day. The entry has been greylisted. The three numeric fields indicate timestamps related to this host. The 2 means the host has attempted delivery twice. The zero means the host has not yet delivered any mail. Why was the second attempt not allowed? spamd has three time parameters related to greylisting. See the man page for better definitions. The values shown are the defaults.
So now I waited, for another attempt. And there it was: Remember how I said spamlogd will monitor the mail logs and move entries to the white list? Look here on nyi: $ spamdb | grep 203.118.144.46 WHITE|203.118.144.46|||1163006999|1163009572|1166119999|3|0 There you go, the entry has been whitelisted. Three delivery attempts have been made, and none have succeeded. Actually, as I type this, the mail is still sitting on zip waiting to attempt another delivery. This time, PF will redirect the incoming connection directly to my mail server, and not to spamd. Coincident with the above was this log entry: *** /var/log/debug.log *** Nov 8 13:13:19 nyi spamd[27526]: whitelisting 203.118.144.46 in /var/db/spamd That is spamlogd reporting that it has whitelisted the client. On the next delivery attempt, the message should go straight through. And here it is: The host is now in the spamd database as this: $ spamdb | grep 203.118.144.46 WHITE|203.118.144.46|||1163006999|1163009572|1166121773|3|1 203.118.144.46 will remain on the whitelist until 36 days of no sent email. While on the whitelist, it will not be subject to greylisting. If if falls off the whitelist, it will go through the above greylisting process again. |
|
Problems I encountered
|
|
Yes, I had a problem. The <spamd-white> table was always empty. Even after spamlogd moved something from the greylist to the whitelist. This thread in the FreeBSD PF mailing list found me the missing piece: mount -t fdescfs fdescfs /dev/fd Now the whitelist table has some entries! # pfctl -t spamd-white -T show No ALTQ support in kernel ALTQ related functions disabled 12.152.184.25 66.35.250.206 205.150.199.217 216.136.204.119 # My thanks to LI Xin. Now, an even bigger problem. Despite having the whitelist updated, my email still isn't getting through. Whitelisted clients were still being subjected to greylisting. delo found the answer. My rules were wrong. I had this:
See the problem? Line 1. That table name is wrong. It should be <spamd-white>, the table maintained by spamlogd. DOH! When I found this problem, I corrected /etc/pf.rules and updated PF by issuing this command: pfctl -f /etc/pf.rules Later on, I found out about this option: Once I fixed the NAT rules, I went to m21 and tried to connect. I got straight through to the real smtp server: dan@m21:~$ telnet nyi 25 Trying 64.147.113.42... Connected to nyi.example.org. Escape character is '^]'. 220 nyi.example.org ESMTP Postfix QUIT 221 2.0.0 Bye Connection closed by foreign host. dan@m21:~$ Good, that proves the whitelisting is working. Then I flushed the Postfix mail queue, and the mail message went straight through. Yes, I missed this entirely during the port install: |
|
What is in my spambd right now?
|
|
Yes, the domain names have been slightly obscured, but you should be able to see who is sending to what. For the record, the MX server in question is not an MX for langille.org or freebsddiary.org... but that's not stopping the spammers from trying. At present, only bsdcan.org uses this greylisting server as an MX. I'm about to add more domains to it and implement greylisting on my other servers. As I type this additional note on 24 November, about 3 weeks after the above, here are the stats of each of my three mail servers:
It is interesting to see that one machine has whitelisted nearly 4500 servers in about 9 days... |
|
Greytrapping
|
|
I'm sure all of this sounds great. It can be better. Greytrapping is one step further than greylisting. No doubt, you have an email address that is no longer used, but you still get email sent to it. It's probably been on spamming lists for years. If someone is sending email to that address, it's bound to be spam. What you can do is add that address to spamdb as a spamtrap address. See man spamdb for details. For example, if you want to designate anyone sending to yourname@example.org, then you can issue this command: spamdb -T -a "<yourname@example.org>" I have a list of 24592 such email addresses. Why? Well, they aren't really addresses. They are Message-ID: values from FreshPorts. Background: FreshPorts didn't always store Message-ID:. When that attribute was added, I need to come up with a value for the existing commits stored in the database. Unfortunately, I select something like fp1.12345@example.org (s/example/FreshPorts). Spammers grabbed all those addresses, and I started to see huge spam attempts. All bounced of course, because they were not valid addresses. I have since changed those Message-ID:s to @dev.null.example.org (s/example/FreshPorts). But the spammers continue. So how do I get the email addresses into spamdb? They are all in a file named greytrap. This command loads them. It takes a few minutes to complete. cat greytrap | xargs -n1 spamdb -T -a That's all there is to it... |
|
Greyscanning
|
|
With newer versions of spamd (not available in the FreeBSD Ports tree at the time of writing), you can take advantage of the greylisting period to scan your logs and take appropriate action. The greyscanner script will scan the spamdb output and look for patterns and blacklist that IP address for 24 hours. If it's not spam, it will come through later. If it is spam, well, you've delayed it. This script can validate address, check for an MX or A record for the source address, etc. Look here for details: http://www.ualberta.ca/~beck/nycbug06/scripts/ |
|
Things to think about
|
|
Greylisting can delay mail. Greylisting can block mail, but only if you continuously redirect the connection to the tarpit. However, it does greatly reduce the amount of incoming spam. I have no comparative statistics to show you. All I know is I like it and it reduces the amount of crap in my mail box. :) |