A year or more ago on my old blog I wrote an article about how I think we could use logging information in hybrid integration scenarios to bring together this holistic view of whats happening across the stack. The old link was:
http://geekswithblogs.net/michaelstephenson/archive/2014/01/23/155259.aspx
More recently Ive spent some time playing around with Azure Service Bus Event Hubs and exploring the idea of instrumenting my applications and integration components with log4net where it is configured to send logging information to Azure Service Bus Event Hubs. The aim was to implement the scenario described on the article above and now Event Hubs is GA I felt that it was the perfect place to send my logging data because of the following reasons:
- It is capable of dealing with massive volume
- It has some good ways to process data from the stream such as Stream Analytics
- Strategically I believe it will fit really well as an on ramp to the Microsoft Business Intelligence Stack
- Its easy to send data to it
If your in the .net world then log4net is one of the most common libraries you will use for logging and again about a year ago I showed how you could create a log4net appender for Azure Service Bus Messaging, so it made sense to do a new post with a log4net appender for Event Hubs.
In the project I have 2 appenders
- SBEventsAppender = A simple appender for Event Hubs
- SBEvebtsBufferedAppender = An appender where you can buffer the events and process them with multiple threads and the batch methods in the service bus SDK
The appenders are pretty easy to use and the code download below gives you a sample console application demonstrating their use and some unit/performance tests in visual studio to let you try them out too.
Performance
To give you an example of the performance I have the following test which simulates something like a real user scenario calling a component that might log 3 events and do some stuff which takes about 100ms to complete.
[TestMethod]
public void BufferedLogging()
{
BufferedLog.Debug(“Buffered test”);
BufferedLog.Debug(“Buffered test”);
BufferedLog.Debug(“Buffered test”);
Thread.Sleep(100);
}
Running this on my development machine at home, I was running the Buffered appender with 4 publishing threads using a visual studio load test with 100 concurrent users. After running this test for 10 minutes I had been averaging 960 tests per second and completed a total of 580,731 tests. This means Ill have pushed somewhere in the region of 1.7 million events to the event hub. That’s pretty good performance for most logging scenarios.
Now obviously in your application you dont want to send every single logging item to a centralized log store but for those key events (perhaps info, warn, error in the log4net world) you have a viable option for most application scenarios where you can send your log events and do multiple interesting things with them out of process.
Video
Code Download
To get the code sample Click Here
Feedback
Ive been using this for a little while now and it seems to work really well but I would like to hear feedback from others to see if it suits their scenarios well too.
Nice article. I’ve been coming at this same problem with a different tech stack – Java/log4j plus some log4net and aggregating (with nxlog and logstash) into either Splunk or ElasticSearch. Looking to track events/activity across 10s of systems across time.
Also had a thought about using a fake geo value for each actor/system/service to be able to show an activity over time across multiple systems. Inferred from the log while pushing to the aggregator/index. Might make for a very eye-opening set of dashboards/stats screens…
Please make a log4j appender for Event Hubs!
Take a look at this for dev/test: http://log4stuff.com/
Thank you very much for sharing this code (works perfectly). All I can say is amazing 🙂
Great piece of code! Why don’t you put it on GitHub so that we can maintain it and create an NuGet package for it 🙂 Thanks for the article!