Using the ArcSight ESM Console to Create Replay Files

HP ArcSight Enterprise Security Manager (ESM) has some built-in capabilities to generate event files suitable for use with the ArcSight Test SmartConnector.  These replay files can be used to test functioning of new ESM content (Dashboards, Datamonitors, Filters, Rules, Queries, Trends, Reports, etc).  The Test connector has some very powerful features including the ability to replay the captured data as is, or to update the date/time stamp on each event to make the data appear as current versus historical data.  The Test connector can also run multiple replay files into it’s configured destinations simultaneously and at a variable rate suitable to support initial content development as well as high speed, high volume performance testing.

Preparing to Generate Replay File

There are multiple ways to generate replay files, but in this post we will focus on use of the ESM console application software to generate the replay file from selected events already existing in the ESM instance.  In order to constrain the events to a selected subset, we need to have a filter prepared to chose the appropriate events.

1-ReplayFileGen  2-ReplayFileGen

For this example, a filter named router4 will be used, where it simply selects all events that have been generated by device name router4 or device address 10.20.1.27

Generating the Replay File

On the workstation or system where the ESM Console software is installed, start the replay file generator with a replayfilegen argument to the arcsight script in the bin directory.  If the console is installed on Linux or Mac OS X, simply use ./arcsight replayfilegen as the command.

0-ReplayFileGen

When the replayfilegen tool starts, it will display a GUI that allows the user to select the target filename to be generated, the timeframe to query and the filter to select the event data.

3-ReplayFileGen

Note that a relative time frame may be specified by using relative start and end time operators – these will calculate the absolute time frames needed.

4-ReplayFileGen

Once the collection has started, there will be a progress display showing the generation of the replay file.

5-ReplayFileGen

Deploying and Using the Replay File

Now the replay file has been generated, the user can simply copy the file to the current directory of the Test SmartConnector. There can be multiple replay files in the current directory and all will be displayed when the Test connector GUI starts.

6-ReplayFileGen

The user can select which replay files are to be read and events forwarded to the Test connector destinations.  Any or all of the replay files may be selected, making the Test connector ideal for assisting in content development for multiple use cases.

7-ReplayFileGen

Once the desired replay files are selected, the events will be replayed to the configured destinations at the rate specified by the user, as soon as the Continue button is pressed.

8-ReplayFileGen

The Test connector will run through all the event data in each selected replay file and stop. By default there will only be one pass through the data files and no event data is altered. ESM Manager Receipt Time will show the current date/time however the original timestamps will be present in the event data.  The event rate can be changed dynamically while the replay is in progress, so for example, some basic event data could be played to the destinations for some time then the user could adjust the event rate substantially higher to speed the event ingest to the destinations.  This is useful for testing use cases where there may be denial of service or worm outbreak detection that is sensitive to event rates.

There are many run-time options that can be set for the Test Connector, including the ability to loop on the replay files, replay the event data with current time stamps and other event handling options.