Scot’s Newsletter Firewall Test Suite and Methodology

Every hardware or software firewall product Scot’s Newsletter reviews is subjected to a battery of vulnerability tests: GRC's Shields Up! and NanoProbe; GRC's LeakTest; HackerWhacker.com's default port-probing and Trojan Horse tests; PC Flank's Stealth, Trojan, and Exploits tests; AuditMyPC port probe and Trojan tests; and the exhaustive Standard Audit from E-Soft's SecuritySpace -- the most grueling examination of all.

How I Test
Because I don't believe in ideal test situations, I use every firewall product I test for as long as it takes (usually at least two weeks) in advance of the vulnerability tests, configuring the firewall in a way I consider to be typical use. That includes testing with a long list of Internet-oriented programs:

  • HTTP (Internet Explorer 5.5 or 6.0, Mozilla 1.x, and Opera 7.x)
  • FTP (CuteFTP 3.5)
  • SMTP and POP3 email (either Outlook Express 6 or Eudora 5.2)
  • Instant messaging (AOL Instant Messenger 4.3)
  • NNTP (Outlook Express 6 newsgroup access)
  • Windows PPTP-style VPN client access
  • Application updating (Norton LiveUpdate and several others)
  • NTP (Tardis 2000 1.4 Internet clock or Windows XP's built-in clock check)
  • Windows Remote Desktop Connection
  • Ping (CyberKit 2.5)
  • America Online 6.0
  • Symantec Norton AntiVirus 2003 (or similar antivirus program)
  • Microsoft Office 2000 or higher

    I do not test Internet phones, two-way Internet audio-video realtime communication, or online gaming. Only after the firewall is working for my average Internet protocols and online activities do I run the Scot’s Newsletter Firewall Test Suite.

    My benchmark testbed has shifted from a Windows 98 system to a Windows XP system behind a static IP DSL connection. Additionally, all software firewalls and broadband routers or firewalls are evaluated with both Win9x and XP, and also with two different broadband connections, cable Internet and DSL. Vendors are always welcome to request more detail about my testing procedures.

    Because I test a lot of broadband hardware and software, it's important to note that any test of a software firewall is conducted with a straight connection to the broadband modem. No other networking intelligence is involved (including NAT, DHCP, or obviously hardware-based firewall functionality). Hardware router/firewall tests are conducted in a networked environment, since they usually involve DHCP services, but without any software firewall services. The DSL connection I use to benchmark with is very standard. It requires no software running on the PC and it is not behind a proxy server.

    In addition to formally testing a security product on a testbed, I also use it on or with several PCs for weeks (sometimes months) before I write a review. I'm not noted for being out there with the earliest review. Getting it right is more important to me.

    Your Input Desired
    I'm always looking to improve my tests or add to them. If you have suggestions for how I might do that, or would like to suggest other network monitoring services, online scanners, or security testing software, please send your comments my way. I welcome them.