Anomaly Detection at Large Scale

The other day I attended another Data Science London meetup looking into Anomaly Detection at Large Scale. The event was held at the amazing Level39 venue which was nice watching presentations as the sun set.

One thing which was surprising was the size of the community now! It's really nice networking with smart data science people doing all kinds of interesting things in London and the number of new faces in the group was refreshing. Carlos has really done a brilliant job of bringing together data scientist's to share ideas and learn from each other.

The Presentations

Finding the needle in the haystack, or looking for unusual patterns in big data

The first talk was by Adi Andrei with his presentation on identifying and learning unusual patterns in mountains of aircraft sensor data from NASA.

View slides

Anomaly Detection in IT Systems at Scale

The second talk was by Tom Veasey describing some of the data characteristics which make anomaly detection for real world problems challenging and some of the techniques he uses at Prelert for anomaly detection.

View slides

A Gentle Introduction to Apache Spark and Clustering for Anomaly Detection

The final talk was by Sean Owen giving a walkthrough of an example of anomaly detection using Apache Spark and introducing the concept of k-means clustering and how a data scientist would iteratively improve clustering in a session with Spark. Super interesting!

View slides