Do you need to match few 100k events per second? Siddhi Open source CEP Engine and WSO2 CEP Server 2.0 Released

I have written about Siddhi, a new open source java Complex Event Processing engine available under Apache License we were working on. At last, it has emerged, and we have out first production grade release of Siddhi.

Now Siddhi has become the runtime for WSO2 CEP server, and we have just released the WSO2 CEP 2.0 and Siddhi 2.0 comes bundled with it.

WSO2 CEP adds network support for Siddhi, and now you can use it with Thrift, HTTP, SOAP, and JMS.  Siddhi/WSO2 CEP 2.0 has several cool (new) features, but let me briefly list few things I believe are the key.

 1) Very fast (can do 1-2.5 Million events/ sec as a jar library)
 2) Fast across the network too (about 0.25 million event/sec over thrift calls)
 3) New Siddhi Query language (supports Filters, Windows, Pattern, Sequences, Joins). You can find the specification from http://docs.wso2.org/wiki/display/CEP200/Siddhi+Language+Specification
 4) Supports Hazzlecast based highly available deployments and large working memories
 5) Supports periodic snapshots of CEP state so it can recover the state in case of a failure
 6) Integrate closely with WSO2 BAM. Basically you can setup a system where you collect data from many event sources and send them to both BAM and CEP using the same thrift transport so that the former can do batch style hadoop/hive based processing and the latter can do real time processing.

 You can download the server from http://wso2.com/products/complex-event-processor.

 If you are looking for Siddhi jar to be used with your java code, it is in the .zip file. Alternatively you can get the jar from http://dist.wso2.org/maven2/org/wso2/siddhi/siddhi-distribution/1.0.0-wso2v2/.

 We have moved the code and mailing list to WSO2.

 You can find the code from https://svn.wso2.org/repos/wso2/carbon/platform/branches/4.0.0/dependencies/commons/siddhi/1.0.0-wso2v3/.

 You can find documents and samples from http://docs.wso2.org/wiki/display/CEP200/Complex+Event+Processor+Documentation.

 Any questions, you can ask at http://stackoverflow.com/questions/tagged/wso2/.

 I will soon write more about architecture and how some of the Siddhi/CEP features are implemented.

Offline Profiling with JProfiler

We use offline profiling when we want to run the profiling in the headless mode (without UI). Another option is to profile by connecting remotely using JProfiler GUI, which is great.  However, sometime even that option is not available if the network is slow or only allow SSH connection.

Following are quick instructions for offline profiling. For more details, refer to the "JProfiler Manual".

1. Create a JProfile session and configure the triggers. Make sure you add a "save snapshot trigger with a timer", else you will not get anything. Do this like every 5-10 minutes. You can also give the target snapshot file location when you configure. 
2. Copy local .jprofiler7/config.xml and JProfiler binaries or agent code to the remote machine 
3. Add following before the java command.
   

   -agentpath:JPROF_LOCATION/bin/macos/libjprofilerti.jnilib=offline,id=SESSION_ID,config=/Users/srinath/.jprofiler7/config.xml 

   Here replace the configuration file and JProfiler location with your machine's values. SESSION_ID is the session ID of the session you created with JProfiler UI.  Here ~./jprofiler7/config.xml file has settings for all sessions creates by jprofiler, and it will pick up the right value.  If you are profiling in a remote machine copy your local configs to the remote host. 
4. Start and run the program, and it will print  the following
JProfiler> Using JVMTI
JProfiler> JVMTI version 1.1 detected.
JProfiler> 64-bit library
JProfiler> Offline profiling mode.
JProfiler> Using config file /Users/srinath/.jprofiler7/config.xml (id: 194)
JProfiler> Listening on port: 8849.
JProfiler> Instrumenting native methods.
JProfiler> Can retransform classes.
JProfiler> Can retransform any class.
JProfiler> Native library initialized
JProfiler> VM initialized
JProfiler> Using dynamic instrumentation
JProfiler> Time measurement: elapsed time
JProfiler> CPU profiling enabled
JProfiler> Hotspot compiler enabled


5.  Open the snapshot file and analyze using JProfiler UI.