Trade offs of the terrible syslog protocol

Posted on August 15, 2013 by apomeroy

syslog is a very old message transmission protocol that transmits system messages across a network. The first versions of this protocol were drafted into RFC 5426. Some assumed updating the transmission to use TCP would make things better, and the IETF released RFC 6587 describing syslog over TCP. The problem is, that is inherently unreliable as well, since the application (syslog) has no mechanism to ensure that all messages transmitted were actually received, regardless of the network level transport protocol used to convey the messages.

Rainer Gerhards wrote a blog post on the unreliability of using plain TCP to transmit syslog event data.

An attempt to create a reliable syslog protocol is described in RFC 3195, the problem is that very few vendors have adopted that standard (BEEP).

There is a movement to find a more reliable system message delivery mechanism, as described in this Wikipedia post, however the problem is not only one of a technically feasible mechanism – one that relies on the application itself to validate and guarantee message integrity and completeness – but also on wide spread adoption by the 10’s or 100’s of millions of devices that send their system logs via syslog UDP.

That will take decades, so best is to use mechanisms that can collect the event messages in native syslog UDP format as close to the generating source as possible then use an application oriented framework to convey those messages to their destination. HP ArcSight SmartConnectors are a good way to accomplish this, with their application level event queuing on input, persistent caching output, compression, encryption, bandwidth throttling, filtering, aggregation and event QoS policies.

Libraries needed to install ArcSight SmartConnectors on RedHat Enterprise Linux and CentOS

Posted on August 7, 2013 by apomeroy

[Update 2016/03/11]:

Starting with SmartConnector 7.1.7 (I think, might be a rev or two earlier), there are a couple more libraries that are needed to successfully install the SmartConnector on Linux. Include libXrender.i686 libXrender.x86_64 libgcc.i686 libgcc.x86_64
yum install libXrender.i686 libXrender.x86_64 libgcc.i686 libgcc.x86_64

[Update 2014/02/04]:
Simpler syntax for the install, using yum to do the automatic dependency processing, and .. a update for CentOS 6.4 64-bit. I believe RHEL 6.4 64-bit would also need these libraries. This worked for installing ArcSight SmartConnector 6.0.7 on CentOS 6.4 64-bit.

glibc.i686 libX11.i686 libXext.i686 libXi.i686 libXtst.i686

You could install like:
yum install glibc.i686 libX11.i686 libXext.i686 libXi.i686 libXtst.i686

[Original post]
While installing an ArcSight SmartConnector 6.0.2 on RedHat Enterprise Linux 6.2 64-bit, the initial install runs successfully, however the connector configuration never kicks off, then install just claims it is done. runagentsetup.sh fails with Error occurred during initialization of VM .. java/lang/NoClassDefFoundError: java/lang/Object .. obviously a pretty major Java error.

Turns out there are some additional libraries that need to be loaded in addition to what is listed in the documentation.

Some research leads me to believe there were some base libraries that may be missing from the vanilla RHEL 6.2 64 bit install. Basic Server + Desktop configuration was selected and all libraries referenced in the ESM 6.0c Install Guide and SmartConnector User Guide were installed. Tracing through all the dependencies created this exact list of of libraries that are required to be installed on RHEL 6.2 64 bit:

glibc-2.12-1.47.el6.i686.rpm glibc-2.12-1.47.el6.x86_64.rpm glibc-common-2.12-1.47.el6.x86_64.rpm libX11-1.3-2.el6.i686.rpm libX11-1.3-2.el6.x86_64.rpm libX11-common-1.3-2.el6.noarch.rpm libXau-1.0.5-1.el6.i686.rpm libXau-1.0.5-1.el6.x86_64.rpm libxcb-1.5-1.el6.i686.rpm libxcb-1.5-1.el6.x86_64.rpm libXext-1.1-3.el6.i686.rpm libXext-1.1-3.el6.x86_64.rpm libXi-1.3-3.el6.i686.rpm libXi-1.3-3.el6.x86_64.rpm libXtst-1.0.99.2-3.el6.i686.rpm libXtst-1.0.99.2-3.el6.x86_64.rpm nss-softokn-freebl-3.12.9-11.el6.i686.rpm nss-softokn-freebl-3.12.9-11.el6.x86_64.rpm

Note the specific X libraries versus the generic list as shown in the connector user guide. What was interesting about these is that they did NOT all install when doing a wildcard rpm install, and additionally did not report any failures. After some trial and error, on my system it appears the 32 bit X libraries needed to be installed individually for some reason. You may want to use rpm -q -a to verify each of the libraries successfully installed. Once all the above libraries were installed, the connector installation worked as expected.

A tarball with the libraries can be downloaded from here.

Extract the libraries, change into the resulting directory, then you can use the following brute force syntax to determine which libraries are not installed and install them:

rpm -ivh `ls | while read rpmfile; do rpm -q \`basename $rpmfile .rpm\`; done | egrep 'not installed' | awk '{print $2}' | xargs`

How to replay syslog events using the performance testing feature of ArcSight SmartConnectors

Aside

[Updated 2016/08/22]

For testing ArcSight SmartConnector settings or Logger and Enterprise Security Manager (ESM) content, it is quite useful to be able to replay previously captured syslog events. The built in PerfTestSyslog class in ArcSight SmartConnectors make this easy.

There are several ways to capture syslog traffic into a text file for use in replay scenarios. Below are some methods that I have used – may not be the most elegant, but gets the job done.

Run a packet capture of syslog traffic

On the node that has inbound syslog traffic, run a packet capture using tcpdump:

tcpdump -nn -i eth0 -s0 -w syslog-traffic.pcap port 514

where eth0 is the network interface receiving the syslog traffic, syslog-traffic.pcap is the resulting pcap format output file of captured events and 514 is the port that syslog traffic is expected to be received.

After capturing a suitable size of events, import the pcap file into Wireshark, click on one of the syslog packets, right click and select Follow UDP stream. A decoded content window will appear where you can select Save As .. and dump it to a sample events file. Ensure to select ASCII versus Raw format. This will be your event input file to feed the PerfTestSyslog function of the ArcSight SmartConnector.

Replaying the syslog events using an ArcSight SmartConnector is controlled via the GUI that is displayed when the PerfTestSyslog class is launched. In my example, I have a Test Connector installed on my current host (RedHat Enterprise Linux, however Windows, Solaris or AIX would work just as well) in the /opt/agents/syslog-udp-1514 directory. This connector is up and running listening on UDP 1514 for syslog messages, however we are also going to use it to feed the syslog event to the same connector. Just think of it in two separate unrelated processes, since you could just as easily use this to feed the syslog events to another host somewhere on the network.

cd /opt/agents/syslog-udp-1514/current/bin ./arcsight agent runjava com.arcsight.agent.loadable._PerfTestSyslog -H 127.0.0.1 -P 1514 -f ~arcsight/udp.txt -x 50

In this example, we are launching the connector framework (./arcsight) and telling the PerfTestSyslog class to read the ~arcsight/udp.txt file (our previously saved syslog events captured with tcpdump) and send them to Host 127.0.0.1 on Port 1514. The last argument is interesting – it configures a slider allowing the user to dynamically increase the Event Per Second (EPS) rate up to a maximum of (in our case) 50 EPS.

A sample capture file has events that look like:

<190>Jun 27 2012 12:16:53: %PIX-6-106015: Deny TCP (no connection) from 10.50.215.102/15603 to 2.3.4.5/80 flags FIN ACK on interface outside

You can also eliminate the original timestamp if you chose:

%PIX-6-106015: Deny TCP (no connection) from 10.50.215.102/15605 to 204.110.227.10/80 flags FIN ACK on interface outside

The PerfTestSyslog class has a number of pretty useful options, including -m to randomize the Device Address. This is really good for faking events from multiple firewalls.

Various configuration options exist on both the receiving SmartConnector (that is listening on UDP 1514) and the transmitting program, including the ability to keep the original timestamp intact or replace it with current time. This is especially useful for testing new content or performing historical analysis on previously saved event data where the original timestamp is needed.

Update:

For situations where you would like to run this without a GUI, you can add the -n option to start with No GUI. In that case, although the rate is no longer dynamic, you do need to specify a starting event rate otherwise it appears the default is 0 .. eg. no events will be sent. Instead of only specifying -x for max rate, also specify the starting rate with -r

./arcsight agent runjava com.arcsight.agent.loadable._PerfTestSyslog -H 127.0.0.1 -P 1514 -f ~arcsight/udp.txt -n -r 50 -x 50

How to survive your first few hours on Stand Up Paddleboard

Posted on August 5, 2013 by apomeroy

I used to do yoga on a regular basis. Hot yoga in fact. There are a few skills and strengths that hot yoga taught me, including how not to panic when starting a completely new and foreign sport that requires at least a little core strength (unless you don’t mind swimming a whole lot). Yoga gives you core strength for balance and endurance, breath control to calm the body and lower your heart rate, and mind control to allow a peaceful, tranquil zen like meditation. Hmm. Sound like it should apply to this thing called Stand Up Paddleboard? Here’s my experience.

Day One

So on a nice sunny Austin weekend, we hit the Texas Rowing Center to rent our very first Stand Up Paddleboard. With waivers appropriately signed and some green changing hands, now we’re on the dock to get sized for a paddle. Paddles need sizing? Oh.. to do this right, you have to actually stand up on this board thing, so since it’s a ways from your shoulders down to the water you need a long paddle. Got it. Now they zap strap this bungee cord onto your ankle so you can’t steal the board .. or just so when you go dive into the drink, you don’t lose the board .. and your main flotation device. Got it. Climbing onto the board from the dock was interesting, and I managed to do it without careening over the board into the drink. Kneeling on the board, it was a little awkward to maneuver away from the dock, but didn’t take long to get the hang of it. A ways out from the dock, I started the rather harrowing process of standing up. From the kneeling position that is pretty comfortable and stable (and also good for praying), lean forward and plant your palms on the board. Gradually get your feet underneath you so it looks like you’re crouching on the board. Raise your butt up to the sky kind of like you are doing a downward dog, but with poor form. Now realize the trick is to raise your butt up to the sky first, then just stand up. I’ll tell you the last step was the hardest, since by now the board is rocking side to side like a dingy in a perfect storm on the ocean, with your legs uncontrollably shaking like you’ve never used them before in your life. Of course having absolutely no room in my mind for any other thoughts than panic, I could only think of balancing myself by holding the paddle out in front of me parallel to the water – one big balancing act. Turns out I looked like one of those high wire acrobats that manages to cross Niagra Falls with nothing more than a rod for balance .. except I definitely wasn’t that good. I must have been a circus performer in a former life, rather than the scientific mind that I thought I had in this life. It doesn’t dawn on me until Day Two that a much better strategy for balancing on the board initially when standing up would be a three legged stool, versus the two legged wobbly tower of my body. Why not just brace yourself with the paddle in front of you to make a tripod?

by now the board is rocking side to side like a dingy in a perfect storm

The water on Town Lake in Austin in August is bathwater warm and it’s about 95F with bright sun. Last time I checked I wasn’t make of sugar (or salt), so I’m pretty sure I won’t melt if I do bite it into the drink. Except for the lake weeds where we are currently have been known to tangle people up so badly that once they’ve fallen in, they are never seen again. No trace whatsoever. Great incentive not to fall in.

As we make our way through the hour of allotted terror time, I’m able to stand up twice for a whopping total of 60 seconds. Back to the dock we go. How to get off the board back onto the dock? This should be entertaining.

Ok, Let’s catalog Day One accomplishments: Got on the board from the dock without crashing into the water? Check. Manuvered out from the dock greater than 10 yards? Check. Actually stood up on the Stand Up Paddleboard? Check. Well kind of .. twice for 30 seconds each. Manage to get off the SUP back onto the dock without dragging the attendant into the water with me? Check. Realize that Kayaking is much safer and probably the better sport for me? Check.

Day Two

Realizing that the uncontrollable shakes in my legs were probably from a lack of balance or core strength in the legs, I decide to do a week of balancing on one leg whenever I could. Looked pretty funny brushing my teeth teetering on one leg like a stork, but sure seemed to help on Day Two. Foolish enough to return to the dock of terror for another run at Stand Up Paddleboard, I grab a life jacket and paddle and again manage to navigate away from the dock. This time we paddled out and over to an open part of Town Lake – no weeds, so there’s nothing to hold me back except fear it self.

I manage to stand up, this time using the paddle against the board to stable myself, and manage to stay upright for over a minute without dying. My legs have tremors just like Day One, but the feeling of the board is more familiar. I realize that the board it actually pretty stable and you don’t need to perfectly balance on it – a little wobbling side to side is just fine! First breakthrough feeling. I begin very gently paddling up the lake towards Mopac bridge and although I’m not moving very quickly, I’m not falling off! Still lots of leg shaking going on. Small paddle strokes like a little girl, I work my way up the lake. One piece of advice: look over the lake to the horizon or some other point away from the board .. don’t focus on the water just in front of the board. Another break through.

About 20 minutes later I absolutely have to kneel back down on the board to give my legs and toes a break. No kidding, there were indentations where my toes had dug into the board. Paddling while kneeling seems to be pretty easy, but you just can’t get any speed going.

I stand back up again (including the mandatory stabilization with my paddle on the board), and ignore the tremors from my leg muscles. It was the strangest thing, I could apply power to each paddle stroke regardless of my shaking muscles. Break through!

Day Three

Heading out on Day Three was an amazing experience. Not event a hint of shakes from my leg muscles I was able to stand up right away and paddle pretty hard up past the Mopac bridge, standing up the whole way. What a great sport! Definitely a good workout, with strong paddling, each stroke works shoulders, arms, core, and legs (if you’re stoking hard enough). So now the freedom of movement on the lake is amazing! I’m starting to consider buying a board. Lots of good options for SUP boards and gear in Austin, including the awesome SUPATX store that provides the boards for Texas Rowing Center. By the way, TRC is the largest SUP rental operation in the world with over 120 boards. Even then, you probably want to go during the week days if it’s your first few times, since it’s popular and you will likely have a wait a bit for a board on the weekend. The venerable REI also has a great selection of SUPs and add on accessories. Being a gear hound, REI was like a candy store .. straps to make board carries easy and even suction cup based lights you can mount on the board for night paddles. One note, you do need to hit the Round Rock REI store since the downtown REI doesn’t carry SUP gear.

Day Four

By Day Four, I was feeling like a SUP rockstar, but wanted to know how much of a workout an hour long SUP session would be .. so I strapped on my heart rate monitor, put the iPhone into an otter box (so if i did dump, at least I wouldn’t have to buy a new phone).

It didn’t take very much heavy paddling to figure out how to lean into the paddle stroke so there is a huge amount of power there .. enough the board is leaving a consistent wake with only three to four strokes per side. We paddled out from the TRC dock up the lake past the Mopac bridge and past the UT Rowing Center dock. Took about 17 minutes to get down there, but with the wind blowing west over the lake it took quite a bit more effort to get back and took about 20 minutes (of pretty concerted paddling as you can see by the ramp up on the right side of the heart rate chart). Digifit claims the out and back paddle was 1.88 miles over 1 hour and 13 minutes (yes we took some time for a lay down on the boards out at the furthest point) and burnt 318 calories.

Coming back into the dock, I hopped off the board onto the dock without even getting my feet wet.

We are now planning entire workout routines around SUP, including working on having the balance and coordination to try yoga on the board. Meetup.com has some great groups for all sorts of outdoor fitness activities in Austin, including SUP based yoga. Kinda cool! Lots of good places to get SUP boards and gear in Austin including the venerable REI,

So that’s it. After four days of Stand Up Paddleboard, I’m hooked. Of course it helps that there is all sorts of gear you can buy to enhance your SUP experience and for a gear hound like me, that’s awesome!

IT Security Topics

Posted on July 27, 2013 by apomeroy

Austin Sights and Stuff Todo

Posted on July 27, 2013 by apomeroy

Foodie Posts

Posted on July 27, 2013 by apomeroy

Travel Posts

Posted on July 27, 2013 by apomeroy

Nepal Trek Spring 2022 Tsum Valley February 9, 2022 - My brother John and I have joined a group of fellow trekkers that are undertaking a charity trip to the Tsum Valley in Nepal. Help for the Tsum Valley Our friends at the Compassion Project have visited Nepal and the … Continue reading →

More fantastic diving with Mike Severns October 19, 2011 - Just finished a trip to Maui and had the good fortune of diving again with the crew at Mike Severns diving. The crew this time included dive masters AJ and Warren (as usual) but I also had a chance to … Continue reading →

Indianapolis Food Stops! September 2, 2011 - Ok, next time we’re in Indianapolis to see the Colts, we’re checking out these places: Zest 1134 E. 54th St. Indianapolis, IN 46220 (317) 466-1853 www.zestexcitingfood.com/

Merry Christmas and Happy Holidays .. fantastic diving December 24, 2010 - Well, here’s a shout out to Mike Severns Diving in Maui .. (808) 879-6596. As usual, the crew including Warren, AJ, Michelle and last but certainly never least, skilled and fearless (or at least never speechless) captain Andy, managed to … Continue reading →

Diving site wishlist January 19, 2010 - Here’s a list of the sites I’ve heard are worth checking out: Thanks Adam S: Kay Op in Tulum Dosojos (2 cenotes there Bat Cave and Barbie Cave)