5G, big data, analytics and AI
It’s less than a week until data teams come together at this year’s Spark AI Summit 2020. Over the week of June 22-26, thousands of engineers, scientists, developers and business and tech leaders, who are shaping the future of big data, analytics and AI will meet – virtually of course.
From well-known brands like Databricks, Microsoft and Apple to global data science, security and AI specialists, this promises to be an extraordinary virtual event and perhaps one of the largest ever staged.
For those not familiar with Spark, it is a general-purpose distributed data processing engine that’s used in a variety of ways to process and analyze big data. It’s often incorporated into apps by developers and data scientists in order to rapidly query, analyse and transform data on a huge scale.
Here at Illuminate Technologies, our research using Spark, goes back many years, when we started looking into threat hunting in 5G network architectures and the notion that it would not always be done by using rules and queries. Instead, additional data sources such as log files would need to be fused in real-time with our sensor data and then analysed to find subtle changes in behaviour that indicate a threat to carrier scale networks. The challenge was to integrate our highly system engineered processing technology for mobile network protocols into the Spark analytics platform so that it would smoothly scale to the extremely high ingest rates expected in 5G networks at peak efficiency.
In our session at the Summit we will cover some of the techniques that can be used to integrate existing processing code such as JNI wrappers, autoloading shared libraries and maven build integration. The session will cover both file sources and structured streaming ones that can be used to construct ETL pipelines that analyse sensor data in real-time.
Register for free and join the Illuminate Technologies session, with Developer and Solutions Architect, Doug Carson: ‘Consolidate Your Technical Debt With Spark Data Sources -Tools and Techniques to Integrate Native Code’