While pfSense firewall offerings are based on the BSD packet filter (pf) functions and offer excellent performance and value, the current implementation my customers are running (2.1.5) outputs firewall rule logs in two syslog lines.  The skilled developers that maintain pfSense have indicated that in the 2.2 release they will likely move to a single line log format that is friendly to machine parsers, however I needed to provide some ArcSight parsing of this log data immediately.

Two approaches are possible here.  I could write a FlexConnector parser that expects and parses two consecutive syslog lines into one in order to pull out the relevant source, destination addresses, ports and device handling information (pass, block, reject, interface, protocol, flags, etc).  The other option I discovered was what I implemented since my customer was willing to allow a system patch to be applied to the firewalls that outputs log entries on a single line.  This will make a FlexConnector parser a breeze to create.

After looking through the pfSense support forums, I found this post by jimp (one of the developers) that provided a patch to enable single line logging.  To install the patch, you need to install the System Patch installation package on the firewall, since it’s not there by default.

Package install on pfSense Firewall

Enables installation of patches

From the Package Manager, install the System Patch package.  This enables downloading and automating the patch installation process.  This is a safe way to apply patches, since it will validate the patch can be applied and more importantly backed out if needed.

Select the Patches option in the System menu to start the patch application process. At this point you can (and probably should) just enter the URL to the patch versus manually entering the patch diff file in the text dialog box that comes up. Entering the URL (http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff) gives the ability to Fetch the contents of the patch (double click on the Fetch option).  The System Patch package will go download the patch at the given URL and validate it.  If the validation is successful (it can apply and also roll back the patch), then the Test and Apply links should automatically show up.  If not, only the Test link will show up .. click on the Test link to validate the patch can be applied and backed out.  This should happen automatically so the Apply link shows up.  After application of the patch there will be a new option that shows up in the Status > System Logs > Settings pane that allows the raw logs to be written out on a single line.

Now the logs are coming in on a single line, creation of an ArcSight FlexConnector is simple.  I first directed logs from the firewall to a standard syslog daemon SmartConnector (7.0.5) and enabled Preserve Raw so I can grab the original event and write the parser to grab all the relevant fields I’m looking to correlate.

Using the ArcSight FlexConnector tool kit I was able to get it to suggest a starting point for the regex patterns to parse the single line logs.  Launched the regex GUI with arcsight regex in any connector’s current/bin directory. I copied a FlexConnector config file from an iptables parser I wrote to use to start from .. and produced a quick and dirty mapping of the pfSense events with this parser in user/agent/flexagent/syslog/pfsense.subagent.sdkrfilereader.properties.  Note this is a work in progress, so no categorization map files have been provided yet.  That will come soon.