6. Firewall Testing

When you have put time and money into a range of measures for protecting your data it makes sense to test them. Ongoing testing to make sure the firewall continues to work as intended is highly recommended (inspiration for this discussion of firewall self-testing comes, in part, from the paper by Wietse Venema and Dan Farmer, "Improving your site's security by breaking into it"). Consistent, periodic testing is an important part of maintaining effective firewall security. Placing trust in an un-validated configuration, or in a configuration which was only validated at installation is dangerous. For example, any quick change to a firewall configuration to support a special project or one-time access may have unforeseen effects on the security of the entire configuration. While self-testing is no guarantee of invulnerability, it ensures that the castle walls are not crumbling, the gates are closed, and that the moat is full of water.

Black Box Testing -- What we look like from the outside
What can we learn by probing our firewall? This type of testing can be divided into two parts, port scanning and on-the-wire observation. A port scanner systematically works its way across all of the sixty-five thousand possible service-connection ports on a TCP/IP connected host or device. This is a valuable service to both the hacker-of-systems and to the maintainer-of-systems, as it reveals what avenues of attack and legitimate services may be available on the target.

Standalone scanning software is widely available for Unix-based platforms as well as for Windows workstations. Scanning capabilities are also incorporated into automated testing tools (more on these later). The useful part of scanning is accounting for what the scan turns up, and making sure that only things which are authorized under policy are present. Firewalls vary widely in how they implement services. Likewise, the operating system platforms upon which they are installed have sundry built-in or necessary services which cannot be disabled. However, it is possible to establish some generic guidelines for self-test scanning:

a. Baseline scans
Run a baseline scan on any configuration before connecting it to the internet. Having pristine-state scans of firewalls and any other network-attached devices is useful in tracking down things that change -- due to causes legitimate or nefarious.

b. Scan from multiple locations
It makes sense to scan from numerous locations. At a minimum these include scanning from:
- a "typical user" location on the protected network
- the DMZ (network immediately outside of the firewall)
- a "foreign" external site (perhaps a dial-up ISP account)

Note that correlating output from these scans should provide a clear picture of the network security policy. Differences between DMZ and foreign scans may indicate unwarranted trust of the DMZ, that should be investigated, or the effect of filtering at the router. Scanning from inside the protected net also provides a picture of what an attacker who managed to dial-up, or gained physical access, would see. This often provides interesting food for thought.

Some recommendations:

· Despite the time that it may take, ensure that scanning attempts all ports from 0 to 65535. Stopping a scan at port 10000 (as some automatically do) leaves a large section of unmonitored perimeter.

· Scan everything. Although it spills over into the policy area, ensuring that the service offerings of all systems, even inside the protected network, are recorded and understood contributes greatly to site security. (see point b above concerning attackers inside the perimeter).

· Review the output. Scripting of scan tests is a great time saver, however, reviewing the output for anomalies requires some clever automation or a diligent administrator (organizations are urged to acquire one or the other, if not both).

· A most pernicious work of Trojaneering is a tool which tells you that things are just fine (regardless of what is actually there). Likewise having baseline or periodic reports "adjusted" by unfriendlies has no good outcome. Techniques for protecting tools range from read-only file systems and removable media to air-gapped (unplugged from the network) systems and beyond.

· You want to keep your test records out of harm's way. Once again, removable media, or even paper copies, safely locked away, may prove to be invaluable should things go awry.

On-the-Wire Observation
In the not too distant past, network analyzers were bulky, expensive devices which almost required a pilot's license to operate properly. The top end of the product category is still expensive and complex, but there are now cheap and even free products for monitoring network traffic. Most network monitors provide a semi-interpreted play-by-play of what is going past the monitoring system in real time. Note that, as in the case of port scanners, the network monitor (commonly called a sniffer, which is a trademark of Network General Corporation) is a tool which can be used effectively by security personnel and also, unfortunately, by less honorably intentioned persons.

Network monitor software is available as part of some operating system distributions (such as snoop which ships with Sun's Solaris). A popular free product, tcpdump, is available as source code for many other Unix platforms. There are also a number of PC-based products which work with packet drivers and allow the use of retired 286 or 386 PCs as dedicated network monitoring stations (a noble and cost-effective alternative to the dust-collector-and-doorstop retirement plan).

Please note that the cheap-if-not-free network monitoring products are generally applicable to non-switched Ethernet networks. If the configuration under test utilizes less common network media, the type of monitoring described may well require special equipment. Also note that a tutorial on the details of TCP/IP diagnosis is beyond the scope of this document but a number of works on the subject are available (see Reference section at the end of this document).

a. "Quiet Wire" Observation
It is important to be familiar with the "signature" of the network when quiescent. In addition to the ports which are active (ready to respond to incoming calls), services which broadcast information are a concern worth noting. Depending upon the configuration and point of connection, relative to hosts/router/firewall, different types of traffic will be visible.

With the DMZ disconnected from the Internet router, and no clients initiating traffic, a network monitor should see relatively little (if any) traffic. While it is of little utility to save trace files in this mode, it is very helpful to have notes of the type of broadcasts or session requests seen, and which devices originate requests.

b. Control Testing
As a control, it is useful to observe the trace of a connection originating on internal clients leaving the firewall bound for an external destination. The source address will vary depending upon the particular firewall implementation, as some products cause all sessions to appear to be originating from the firewall's own IP address. Products which implement Network Address Translation (NAT) by definition may not allow the client's true IP address to be seen on the DMZ.

Observing the behavior of inbound connections, both to provided services and to services not permitted by the firewall, improves familiarity with how normal production traffic on the network will appear. A worthwhile control exercise is to observe a network monitor while running a port scan. Scanners which run high-speed `linear' scans are particularly visible to network monitor tools. Other more stealthy scanners insert delays between probes and target ports in a semi-random order (depending upon the type of network monitor being used, some scanning techniques are not visible).

c. "Live Wire" Observation
On a busy network, a wide-open network monitor can scroll information past at an incredible rate. The utility of watching everything is limited, with a couple of exceptions. In the event that something goes wrong at the ISP or backbone level, being able to note the direction and type of traffic passing through (or to) the firewall is an aid to diagnosis. Port scans, as mentioned earlier, may be observed as well. In the event that the firewall's logging/alert system reports unusual activity, it is very handy to have a network monitor available to observe, decode, and possibly record it.

System Logging Verification - Is someone knocking at the door?
It is critical that the personnel supporting a firewall be familiar with the outputs of the logging/alerts system. Configurability of logging/alert facilities is a great help in reducing the amount of material to be reviewed by the administrator; however, it is important to verify the kinds of event logged by the system and the format in which they appear. Since logging facilities vary considerably by product vendor, a meaningful universal list of what ought to be logged is difficult to establish. The point is to attempt violations of the security policy and to note how (or if) the attempt is logged. Here are some tactics to try:

a. Log on to the FTP and Telnet proxies
Try logging on to the FTP and Telnet proxies (if the firewall has them and they are enabled) numerous times as a non-existent user name. Note whether the events are logged singly as failed access attempts or if each attempt is logged separately. This behavior reveals how brute-force password guessing attempts may show up in the logs. Likewise note whether the IP address of the offender is listed.

Perform the same test from the DMZ, a foreign site, and the internal network. On non-NAT systems attempt the test from a foreign network to the IP address of the firewall located on the protected network.

b. Test unsupported protocols
Attempt to use a protocol not supported by the security policy to connect to the firewall, possibly rlogin or TFTP. Many systems reject such attempts without logging; however, on systems which do log the attempts, it is a useful indicator. Perform the same test from the DMZ, from a foreign site, and from the internal network.

c. Test token authentication
In configurations which utilize token authenticators, it is helpful to note what occurs when an invalid challenge/response cycle occurs on a legitimate user account.

d. Test mail
On configuration which have SMTP proxies, the following dialogue should produce a number of log entries (system responses are in bold):
telnet IP-OF-FIREWALL 25
200 <welcome message>
HELO foo.bar.com
220 <hello message, likely with your real IP or host name listed>
WIZ
500 <unknown command, possibly an insulting message>
DEBUG
500 <unknown command, likely a further insulting message>
QUIT
This dialogue is a blatant attempt to lie about one's mail address as well as to exploit two hoary old back doors in the Sendmail SMTP software.

e. Test automated tools
Automated testing tools (see the later section) will certainly exercise a firewall's logging capability in the extreme, and may generate such copious output as to swamp the firewall. If you have access to such tools it is useful to note what shows up in the logs (or possibly what the firewall does) during an all-out non-stealthy automated probe.

Configuration Testing - What did we type for that parameter ?
The point of configuration testing is to validate that the detail components of the configuration were correctly entered as the firewall was installed. The examples listed are by no means exhaustive, but should serve as a basis for devising tests appropriate for the installed configuration.

Misplaced trust of systems due to misconfiguration is a common (and dangerous) situation. In testing for misplaced trust, do not underestimate the risk of trusting IP addresses located on the DMZ network. Trust of DMZ IP addresses may increase susceptibility to address spoofing attacks, and widens the zone of exposure should any other hosts/devices on the DMZ be compromised.

a. SOCKS
In a firewall configuration which uses SOCKS, the facility incorporates trust of certain clients based on IP address of origin. SOCKS incorporates address-spoofing protection, however, misconfiguration can result in external SOCKS clients being able to pass the firewall.

Using SOCKS-ified clients (such as Telnet and FTP) from a foreign network, configure the SOCKS HOST parameter to point to the external IP address of the firewall. Then attempt to access an IP address inside the firewall using the client software. Referencing by IP is more reliable in this situation than using a fully-qualified-domain-name. Attempt the same test specifying the IP address of the firewall's interface on the protected network as the SOCKS HOST. Repeat the test from the DMZ.

b. HTTP Proxies
In the same manner as the SOCKS test, configure a Web browser to use the firewall's Web proxy (if one exists). Attempt to access a Web server inside the protected network. Also attempt to access a Web server outside the protected network. Success with either indicates a misconfigured HTTP Proxy. Attempt the same test specifying the firewalls IP address on the protected network, also repeat the test from the DMZ.

c. DNS
Since numerous schemes for supporting DNS with varying degrees of opacity are common, a full DNS test is highly site specific (and also policy specific). Ensure that fully-qualified-domain-names (FQDNs) are resolvable from the internal network. The ability to Telnet to an external site by IP address, but not by FQDN indicates a DNS configuration problem.

d. User Services
Though pedestrian, formally exercising all of the end-user software in accessing external services is a reasonable validation as to which services the firewall does actually pass to the outside (and that users will be able to use the software with the expected results). Ensuring that services provided (or denied) to external hosts function (or fail) in accordance with policy is likewise helpful as a validation.

Third Party Validations
In addition to self-testing, a range of security assessment services are available from consulting and auditing firms. Third party validation services offer a valuable cross-check of implemented security policy, as well as providing additional assurance for customers, stockholders or management. Options to consider when arranging for such services include:

Scheduled/Periodic
The idea of periodic testing is much like scheduled tune-ups for a car. They serve to make sure everything is running smoothly and discover undesirable changes you might not have detected.

Unscheduled (surprise!)
This type of test is designed to keep security staff on their toes and is sometimes incorporated in broader company audits. The idea is to see what happens when attacks take place unannounced and without warning (just like in the real world).

Espionage/Social Engineering
Anyone familiar with Ira Winkler's papers and book on social engineering [Wink] will know that the best-laid plans of network administrators and MIS professionals can often be undermined by poor personnel practices and the natural human tendency to under-estimate the duplicity of others. Tests in this realm should be considered if you have any doubts about any of your people or suspect that any of your competitors are developing a taste for espionage. However, such tests require extensive and careful preparation to avoid damaging morale and even causing lawsuits for entrapment and harassment.

Customized Testing
You may have specialized needs that require custom test scenarios. Try to select a third party that can demonstrate not only the technical skills required to devise the test, but also a good understanding of your business and the likely threats.

Resources
· Bellcore: Pingware
· Dan Farmer: System Administrator's Tool for Analyzing Networks "SATAN"
· DoD's SPI Package (available only to DoD/DoE sites... you know who and how)
· SNI Ballista
· Internet Security Systems: Internet Scanner
· Wheelgroup NetSonar
· Netect Scanners
· Miora Systems Consulting