Anomaly Detection at Large Scale
The other day I attended another Data Science London meetup looking into Anomaly Detection at Large Scale. The event was held at the amazing Level39 venue which was nice watching presentations as the sun set.
One thing which was surprising was the size of the community now! It's really nice networking with smart data science people doing all kinds of interesting things in London and the number of new faces in the group was refreshing. Carlos has really done a brilliant job of bringing together data scientist's to share ideas and learn from each other.
The Presentations
Finding the needle in the haystack, or looking for unusual patterns in big data
The first talk was by Adi Andrei with his presentation on identifying and learning unusual patterns in mountains of aircraft sensor data from NASA.
Anomaly Detection in IT Systems at Scale
The second talk was by Tom Veasey describing some of the data characteristics which make anomaly detection for real world problems challenging and some of the techniques he uses at Prelert for anomaly detection.
A Gentle Introduction to Apache Spark and Clustering for Anomaly Detection
The final talk was by Sean Owen giving a walkthrough of an example of anomaly detection using Apache Spark and introducing the concept of k-means clustering and how a data scientist would iteratively improve clustering in a session with Spark. Super interesting!