Latency on busy RBL servers

RBL or Realtime Blacklist Lookups can introduce some latency or delivery delay in a busy mail system, or slow, problematic network. Each RBL lookup constitutes a DNS lookup and introduces some level of network latency, depending on the service providers network speed, the load on any particular RBL and the lack of caching on the part of the requesting clients.

Open RBL servers are free for use by anyone, and predictably busy. As a blacklist becomes more and more busy, it becomes simple to understand that latency can increase. An open blacklist server must service countless numbers of SMTP servers wanting to reduce and even eliminate spam in their spam filtering efforts.

A mail server will reject any email that is not destine for a valid mailbox. Emails destine for non existent mailboxes are 100% spams except for the occasional typo by the sender. In the case of a typo, the sender is notified of delivery failure. Since spams destined for invalid mailboxes are blocked at the front door, email servers tend to have a lower load than do gateways, which must accept all email for a given domain, without regard for the validity of a recipient - though LDAP verification is common for gateways, more on that in other chapters.

This is where RBL and latency comes in. Gateway filters handle a much greater load than do email servers because a gateway is unaware of actual individual mailboxes. Because gateways are unaware, they must accept all email for a hosted domain, and thus the load can be much higher for a gateway, than a server.

This higher load translates directly into a higher number of lookups to RBL servers, reverse DNS lookups and any number of other spam identification tools that perform DNS lookups over the network. A busy gateway performing RBL lookups on a handful of RBLs can pose a significant amount of latency.

There are some ways to lighten the load on a gateway spam filter suffering from latency issues. For example:

• A full featured mail server will cache RBL lookups. Each time an RBL lookup is done on an IP address the results are stored for X period of time. By caching the RBL lookups in this way, latency is greatly reduced since the same RBL lookups do not have to go out over the wire to the RBL server for every single email, every single IP address.
• RBL latency can also be reduced or cached by passing requests through a local DNS server inside the network. DNS servers will cache results, passing requests upstream only when there is no cached result, or when a cached result has expired, latency can be reduced by configuring your system this way.
• Use of a proxy DNS server. Not entirely different from the previous example, DNS requests can be sent through a proxy DNS server either inside or outside the network. Here again, caching results for quicker repeat lookups is the goal, and reduced RBL lookup latency the end result.

RBL lookup latency will typically make itself apparent with backed up queues. If log files for the SMTP-In sessions show no other problems, Spampit folders are not overly full, and queues are still backing up, turning off RBL lookups for a brief period of time will allow more spam through your spam filter, but it's the first step in identifying RBL latency as the cause of backed up queues.