For years, Snort [developed and maintained by SourceFire] has been the de facto standard for open source Intrusion Detection/Prevention Systems [IDS/IPS]. Its engine combines the benefits of signatures, protocols, and anomaly-based inspection and has become the most widely deployed IDS/IPS in the world.
Suricata, a new and less widespread product developed by the Open Information Security Foundation [OISF], has recently appeared, and seems really promising. It is also based on signatures but integrates revolutionary techniques. This engine embeds a HTTP normalizer and parser [HTP library] that provides very advanced processing of HTTP streams, enabling the understanding of traffic on the 7th level of the OSI model.
Tests have been conducted against two platforms receiving the same payloads. Based on these tests, conclusions will be discussed to present the advantages and limitations of these two products.
I'd like to give a special thank to Joel Esler for his very constructive review on the write-up. I've been in touch with Joel to review Snort configuration files and make the results as accurate as possible.
Global overview
Param Suricata Snort IPS feature optional while compiling [--enable-nfqueue] Snort_inline or snort used with -Q option Rules
- VRT::Snort rules
- EmergingThreats rules
- VRT::Snort rules
- SO rules
- EmergingThreats rules Threads Multi-thread Single-thread Ease of install Not available from packages. Manual installation. Relatively straightforward. Installation also available from packages. Documentation Few resources on the Internet Well documented on the official website and over the Internet Event logging Flat file, database, unified2 logs for barnyard IPv6 support Fully supported Supported when compiled with --enable-ipv6 option. Capture accelerators PF_RING, packet capture accelerator None, use of libpcap Configuration file suricata.yaml, classification.config, reference.config, threshold.config snort.conf, threshold.conf Offline analysis [pcap file] yes Frontends Sguil, Aanval, BASE, FPCGUI [Full Packet Capture GUI], Snortsnarf
Environment & methodology
Platforms
Tests have been conducted against two identical platforms based on a Debian 5 Lenny distribution hosted on a ESX VMWare server. In addition, following prerequisites have been installed on the 2 test plateforms:
- libpcap version 1.1.1
- PCRE version: 7.6 2008-01-28
- ZLIB version: 1.2.3.3
Versions
Last available versions in the time of this writing have been tested:
- Suricata: 1.1beta1 [rev a96eae0]
- Snort: 2.9.0.4 IPv6 [Build 111]
Rules
Three sets of rules have been used as follows:
Rule set Suricata Snort VRT::Snort rules [free version] x x SO rules [Shared Object] x EmergingThreats rules [free version] x
All rules have been activated [even those commented out by default]:
for x in ls *.rules | grep -v 'local.rules'
do
sed -i -e 's/^#\s*alert/alert/i' $x
done
Configuration files
Configuration files used for the tests are available here:
- Suricata: suricata.yaml
- Snort: snort.conf
Methodology
Scoring methodology
Following scoring has been used to evaluate test results:
- 0: Does not meet criteria or meets them minimally
- 1: Meets criteria
- 2: Exceed criteria [essentially used in the shape of evasion tests]
In addition, a priority has been associated to each group of test. A total is computed by summing weighted totals [SUM[sum of the scores for a given group * priority for the group]].
Captures
Log files have been analyzed "live":
- Suricata: tail -f /var/log/suricata/fast.log
- Snort: tail -f /var/log/snort/alert
Tcpdump / Wireshark have been used to track malware & viruses.
Tools used for the tests
For the tests, following tools have been used:
- Scapy: to forge packets from any type. Also Scapytain has been used to save payloads and for easy replay.
- Netcat: to send payloads on a specific host/port in conjunction with echo
- Hping: to perform DoS
- Nmap: to perform ports scans from many types
- tcpdump: to capture traffic and analyze it [e.g. filter]
- tcpreplay: to replay capture files
- Hydra: to brute force services
Test results
Synthesis
[email protected]$ 0x0010: c0a8 6412 0015 8044 8c27 3997 600b 8d62 ..d....D.'9.`..b 0x0020: 8018 016a 49c4 0000 0101 080a 0be2 9c7f ...jI........... 0x0030: 002e cb9d 3533 3020 4c6f 6769 6e20 696e ....530.Login.in 0x0040: 636f 7272 6563 742e 0d0a correct...
In addition, the PCRE engine has been successfully tested:
$ pcretest PCRE version 7.6 2008-01-28 re> /^530\s+[Login|User]/smi data> 530 Login incorrect 0: 530 Login 1: Login
At last, the rule itself has been isolated to the local.rules file and has been successfully loaded by Snort.
A bug has been reported to Snort to understand why the rule doesn't work.
Evasion techniques
These tests aim at testing the ability of the engines to detect attack attempts with evasion techniques such as fragmentation, encryption, ...
Fifteen evasion techniques have been tested.
Both Snort and Suricata have demonstrated their ability to detect the attacker on decoy attacks, even on the 7th position, as well as Nmap scans with fragmentation.
Snort seems to be better than Suricata at detecting certain evasion techniques, especially the following ones:
- Random URI encoding [non-UTF8]
- Directory self-reference [/./]
- Prepend long random string
- Fake parameter
- Change the case of URL
- Use Windows directory separator [\]
- Use binary value 0x0b as a request spacer
In addition, JavaScript obfuscation hasn't been detected by Suricata in our test campaign.
Malware & viruses
These tests aim at testing the ability of the engines to detect malware and viruses.
The tests have been conducted on 14 malware and viruses. If Suricata has a better detection level than Snort, both Suricata and Snort have demonstrated their ability to detect viruses.
Shellcodes
These tests aim at testing the ability of the engines to detect shellcodes.
On a set of 11 shellcodes, Suricata has detected 9 shellcodes and Snort has detected 7 shellcodes.
Suricata is better at detecting shellcodes.
Denial of Service
These tests aim at testing statefull inspection capabilities of the engines face to Denial of Service attempts.
On a set of 3 tests, both Suricata and Snort have detected the 3 DoS attempts against SSH and MSSQL services. To notice that the alerts that have been triggered mainly come from Emerging Threats.
Client-side attacks
These tests consists in sending malicious documents commonly used for client-side attacks to test the ability of the engines to trigger alerts for client-side attacks.
The tests have demonstrated that Snort is better than Snort to detect client side attacks, with a detection rate of 82% against 49% for Suricata.
Performance
We did not test this due to not having any hardware that had multiple CPU's, but according to this article: //holisticinfosec.blogspot.com/2010/08/suricata-in-toolsmith-meet-meerkat.html "Suricata has a noticeable performance improvement with hardware running multiple CPU's".
Inline/blocking capabilities
This point has not been tested. Nevertheless, according to AlienVault, both Suricata and Snort are compliant and have similar blocking capabilities.
Conclusion
More than 300 tests have been conducted against Suricata and Snort. Not every feature has been tested [IP/DNS reputation, performance, ...] but the tests were mainly aimed at testing the detection capabilities of the engines.
Both Snort and Suricata are based on sets of rules. Most of the tests have shown that VRT::Snort and EmergingThreats rules are complementary and are both needed to optimize the detection of all attack types. In addition, both Snort and Suricata have demonstrated their ability to detect attacks based on signatures from rules.
Suricata offers new features that Snort could implement in the future: multi-threading support, capture accelerators but suffers from a lack of documentation [few documentation on the Internet and outdated one on the official website]. In addition, Suricata doesn't accept some rules from VRT::Snort and EmergingThreats due to incompatibilities [no support of certain keywords]. The support of these missing keywords should be implemented in future versions of Suricata.
On the other hand, Snort is mature. It remains a very powerful IDS/IPS, very well documented over the Internet and that properly detects most of the malware and evasion techniques. Its preprocessors are very usefull for reassembling fragmented packets.
The comparison of stateful inspection features show that Snort and Suricata have different approaches. Snort bases the detection on rules and thresholds to track the number of time a rule is triggered whereas Suricata introduces session variables [e.g. via flowint] enabling to create counters. These variables can then be used by manual rules [local.rules file] to trigger events. One advantage Suricata has is its ability to understand level 7 of the OSI model, which enhances its ability of detecting malware. Suricata has demonstrated that it is far more efficient than Snort for detecting malware, viruses and shellcodes.
As a conclusion, Snort remains the de facto standard for IDS/IPS in production environments. It is stable, easily configurable and very well documented. Nevertheless, Suricata is an emerging IDS/IPS that could revolution the detection techniques and Snort will certainly implement some of these features [support of multi-threading] in future releases. We would still recommend Snort for production environments but keep a close eye to Suricata since this conclusion could quickly be updated in a very near future.
16:10, 12 April 2011 [CEST]
Good Write-up!
17:19, 12 April 2011 [CEST]
18:15, 12 April 2011 [CEST]
19:20, 12 April 2011 [CEST]
17:00, 14 April 2011 [CEST]
18:27, 14 April 2011 [CEST]
20:34, 14 April 2011 [CEST]
20:47, 14 April 2011 [CEST]
@Mike: Many thanks for this very positive feedback. Your comments are really precise and constructive; I'll include them in my write-up.
19:21, 15 April 2011 [CEST]
After going through the 257 Client side samples that you have md5sums for, we have pulled 203 of them. When running these files through Snort we've alerted much much more than the post says we do. However, in order to replicate your results, we'd like to see if we can get copies of the other 54 samples from you. Is there some way I can provide you the list of the md5sum's that we don't have so that we may verify coverage?