Whenever you start poking around a dataset to see what you’ve got to play with, you probably immediately write ‘import pandas as pd’ – why? Because Pandas is the gold standard in data analysis libraries; it’s so simple & yet so powerful. The problem is, Pandas just doesn’t scale, so if you’re going to be […]
Read more
I mentioned in a previous article that for performance reasons, you should avoid the use of UDF’s wherever possible. And while that statement still stands, if you absolutely must use a UDF, you should consider a Pandas UDF rather than those that come out of the box with Spark. The standard UDF’s in Spark operate […]
Read more
Data serialisation plays a critical role in the performance of our data analytics scripts. In this article, we’ll discuss serialisation as a whole & then dive into Kryo serialisation in PySpark. What is serialisation? Often, we store our data in arrays, tables, trees, classes and other data structures, which are inefficient to transport. So, we […]
Read more
When we ingest data into Spark it’s automatically partitioned and those partitions are distributed across different nodes. So, imagine we have bought some data into Spark, it might be split across partitions like this: Partition 1 Partition 2 Partition 3 1, 7, 5, 9, 12 15, 21, 12, 17, 19 45, 32, 25, 99 When […]
Read more
In the last article on performance tuning Apache Spark, we spoke about caching, UDF’s and limiting your input data, which are all simple ways to see some potentially drastic performance improvements. In this post, we’re going to talk about the environmental factors that you can control to overcome some of the more common Spark issues. […]
Read more
Apache Spark provides us with a framework to crunch a huge amount of data efficiently by leveraging parallelism which is great! Tweaking it can be a bit of an art which is not so great! In this article, I will talk about some of the ways that I have successfully tuned the performance of my […]
Read more