How I Test
Because I don't believe in ideal test situations, I use every firewall product I test for as long as it takes (usually at least two weeks) in advance of the vulnerability tests, configuring the firewall in a way I consider to be typical use. That includes testing with a long list of Internet-oriented programs:
I do not test Internet phones, two-way Internet audio-video realtime communication, or online gaming. Only after the firewall is working for my average Internet protocols and online activities do I run the Scot’s Newsletter Firewall Test Suite.
My benchmark testbed has shifted from a Windows 98 system to a Windows XP system behind a static IP DSL connection. Additionally, all software firewalls and broadband routers or firewalls are evaluated with both Win9x and XP, and also with two different broadband connections, cable Internet and DSL. Vendors are always welcome to request more detail about my testing procedures.
Because I test a lot of broadband hardware and software, it's important to note that any test of a software firewall is conducted with a straight connection to the broadband modem. No other networking intelligence is involved (including NAT, DHCP, or obviously hardware-based firewall functionality). Hardware router/firewall tests are conducted in a networked environment, since they usually involve DHCP services, but without any software firewall services. The DSL connection I use to benchmark with is very standard. It requires no software running on the PC and it is not behind a proxy server.
In addition to formally testing a security product on a testbed, I also use it on or with several PCs for weeks (sometimes months) before I write a review. I'm not noted for being out there with the earliest review. Getting it right is more important to me.
Your Input Desired
I'm always looking to improve my tests or add to them. If you have suggestions for how I might do that, or would like to suggest other network monitoring services, online scanners, or security testing software, please send your comments my way. I welcome them.