Installation notes for ArcSight ESM 6.9.1 on CentOS 7.1

Aside

Installation of HPE ArcSight Enterprise Security Manager (ESM) 6.9.1 on CentOS 7.1 is substantially easier with engineering adding a “pre-installation” setup script to this version.  For a smooth installation, there are still a few steps we need to take .. outlined below.

  1. Base install of CentOS 7.1, minimal packages but add Compatibility Libraries. Be sure you use the CentOS-7-x86_64-Minimal-1503-01.iso revision since more recent releases of CentOS have other quirks that may make the ESM install or execution fail. Ensure /tmp has at least 5GB of free space and /opt/arcsight has at least 50GB of usable space – I’d suggest going with at least:
    • /boot – 500MB
    • / – 8GB+
    • swap – 6GB+
    • /opt – 85GB+
  2. Ensure some needed (and helpful) utilities are installed, since the minimal distribution does not include these and unfortunately the ESM install script just assumes they are there .. if they aren’t, the install will eventually fail.
    • yum install -y bind-utils pciutils tzdata zip unzip
    • Edit /etc/selinux/config and disable (or set to permissive) .. the CORR storage engine install will fail with “enforcing” mode of SElinux.  I’ll update this at some point with how to leave SElinux in enforcing mode.
    • Disable the netfilter firewall (again, at some point I’ll update this with the rules needed to leave netfilter enabled).
    • systemctl disable firewalld;  systemctl mask firewalld
    • Install and configure NTP
    • yum install -y ntpdate ntp
    • (optionally edit /etc/ntp.conf to select the NTP servers you want your new ESM system to use)
    • systemctl enable ntpd; systemctl start ntpd
    • Edit /etc/rsyslog.conf and enable forwarding of syslog events to your friendly neighborhood syslog SmartConnector (optional, but otherwise how do you monitor your ESM installation?) .. you can typically just uncomment the log handling statements at the bottom of the file and fill in your syslog SmartConnector hostname or IP address. Note the forward statement I use only has a single at sign – indicating UDP versus TCP designated by two at signs:
    • $ActionQueueFileName fwdRule1 # unique name prefix for spool files
      $ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
      $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
      $ActionQueueType LinkedList   # run asynchronously
      $ActionResumeRetryCount -1    # infinite retries if host is down
      # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
      #*.* @@remote-host:514
      *.* @10.10.10.5:514
    • Restart rsyslog after updating the conf file
    • systemctl restart rsyslog
    • Optionally add some packages that support trouble shooting or other non-ESM functions you run on the ESM server, such as system monitoring
    • yum install -y mailx tcpdump
  3. Untar the ESM distribution tar ball, ensure the files are owned by the “arcsight” user, then run the Tools/prepare_system.sh to adjust the maximum open files and other requirements that we used to manually update in previous releases.  NOTE: in 6.9.1 there are some previous “shadow” requirements that are now enforced (eg. you don’t get to change) .. such as the application owner account must be “arcsight”, the installation directory must be “/opt/arcsight”.  The “prepare_system.sh” script will check to see if there already is an “arcsight” user and if not, will create it.  I usually manually create all the common users on my various systems since I want them to have the same uid / gid across all my systems.
  4. Run the Tools/prepare_system.sh script as “root” user
    • cd Tools
    • ./prepare_system.sh
  5. Run the ESM install as the “arcsight” user
    • ./ArcSightESMSuite.bin
  6. Download content from the HPE ArcSight Marketplace at https://saas.hpe.com/marketplace/arcsight
  7. Install your ESM 6.9.1 console on Windows, Linux or Mac OS X .. although the web interface is much richer in the last couple releases, you’ll still need to use the console for content creation and editing.
  8. Optionally extend the session timeout period for the web interface.  There still isn’t an easy setting to do this in the GUI, so get into command line on your ESM server and edit or add the following lines .. which indicate the timeout period in seconds.  The default is around five (5) minutes. You should be able to edit these configuration files as the “arcsight” user, but I typically restart the services as “root”.
    • Edit /opt/arcsight/manager/config/server.properties
    • service.session.timeout=28800
    • Edit /opt/arcsight/logger/userdata/logger/user/logger/logger.properties
    • server.search.timeout=28800
    • Restart the ESM services .. I typically run this as “root”
    • /etc/init.d/arcsight_services stop
    • /etc/init.d/arcsight_services start
  9. Optionally configure the manager to display a static banner at the top of each console interface so you can have multiple consoles open and know what manager each is connected to (cool!):
    • Edit /opt/arcsight/manager/config/server.properties and add server.staticbanner.* properties (backgroundcolor, textcolor, text). Both backgroundcolor and textcolor take black, blue, cyan, gray, green, magenta, orange, pink, red, white, yellow as acceptable arguments. Text is the identifier you would like that manager to display, such as “super-awesome-production-box”
    • server.staticbanner.textcolor=green
    • server.staticbanner.backgroundcolor=black
    • server.staticbanner.text=esm691
    • Restart the ESM manager service .. I typically run this as “root”
    • /etc/init.d/arcsight_services stop manager
    • /etc/init.d/arcsight_services start manager
  10. If you are going to install any SmartConnectors on the system hosting your Enterprise Security Manager, check out my post regarding required libraries for CentOS and RedHat, before you try to run the Linux SmartConnector install. This includes any Model Import Connectors (MIC) or forwarding connectors (SuperConnectors).

How to replay syslog events using the performance testing feature of ArcSight SmartConnectors

Aside

[Updated 2016/08/22]

For testing ArcSight SmartConnector settings or Logger and Enterprise Security Manager (ESM) content, it is quite useful to be able to replay previously captured syslog events.  The built in PerfTestSyslog class in ArcSight SmartConnectors make this easy.

There are several ways to capture syslog traffic into a text file for use in replay scenarios. Below are some methods that I have used – may not be the most elegant, but gets the job done.

Run a packet capture of syslog traffic

On the node that has inbound syslog traffic, run a packet capture using tcpdump:syslog-simulator

tcpdump -nn -i eth0 -s0 -w syslog-traffic.pcap port 514

where eth0 is the network interface receiving the syslog traffic, syslog-traffic.pcap is the resulting pcap format output file of captured events and 514 is the port that syslog traffic is expected to be received.

After capturing a suitable size of events, import the pcap file into Wireshark, click on one of the syslog packets, right click and select Follow UDP stream. A decoded content window will appear where you can select Save As .. and dump it to a sample events file. Ensure to select ASCII versus Raw format. This will be your event input file to feed the PerfTestSyslog function of the ArcSight SmartConnector.

Replaying the syslog events using an ArcSight SmartConnector is controlled via the GUI that is displayed when the PerfTestSyslog class is launched. In my example, I have a Test Connector installed on my current host (RedHat Enterprise Linux, however Windows, Solaris or AIX would work just as well) in the /opt/agents/syslog-udp-1514 directory. This connector is up and running listening on UDP 1514 for syslog messages, however we are also going to use it to feed the syslog event to the same connector. Just think of it in two separate unrelated processes, since you could just as easily use this to feed the syslog events to another host somewhere on the network.

cd /opt/agents/syslog-udp-1514/current/bin
./arcsight agent runjava com.arcsight.agent.loadable._PerfTestSyslog -H 127.0.0.1 -P 1514 -f ~arcsight/udp.txt -x 50

In this example, we are launching the connector framework (./arcsight) and telling the PerfTestSyslog class to read the ~arcsight/udp.txt file (our previously saved syslog events captured with tcpdump) and send them to Host 127.0.0.1 on Port 1514. The last argument is interesting – it configures a slider allowing the user to dynamically increase the Event Per Second (EPS) rate up to a maximum of (in our case) 50 EPS.

A sample capture file has events that look like:

<190>Jun 27 2012 12:16:53: %PIX-6-106015: Deny TCP (no connection) from 10.50.215.102/15603 to 2.3.4.5/80 flags FIN ACK  on interface outside

You can also eliminate the original timestamp if you chose:

%PIX-6-106015: Deny TCP (no connection) from 10.50.215.102/15605 to 204.110.227.10/80 flags FIN ACK  on interface outside

The PerfTestSyslog class has a number of pretty useful options, including -m to randomize the Device Address. This is really good for faking events from multiple firewalls.

Various configuration options exist on both the receiving SmartConnector (that is listening on UDP 1514) and the transmitting program, including the ability to keep the original timestamp intact or replace it with current time. This is especially useful for testing new content or performing historical analysis on previously saved event data where the original timestamp is needed.

Update:

For situations where you would like to run this without a GUI, you can add the -n option to start with No GUI.  In that case, although the rate is no longer dynamic, you do need to specify a starting event rate otherwise it appears the default is 0 .. eg. no events will be sent.  Instead of only specifying -x for max rate, also specify the starting rate with -r

./arcsight agent runjava com.arcsight.agent.loadable._PerfTestSyslog -H 127.0.0.1 -P 1514 -f ~arcsight/udp.txt -n -r 50 -x 50

See also: Common ArcSight Command Line Operations