Spark

Koalas = Pandas simplicity + Spark’s scalability

Whenever you start poking around a dataset to see what you’ve got to play with, you probably immediately write ‘import pandas as pd’ – why? Because Pandas is the gold standard in data analysis libraries; it’s so simple & yet so powerful. The problem is, Pandas just doesn’t scale, so if you’re going to be […]

Read more
Spark

Solving the UDF performance problem with Pandas

I mentioned in a previous article that for performance reasons, you should avoid the use of UDF’s wherever possible. And while that statement still stands, if you absolutely must use a UDF, you should consider a Pandas UDF rather than those that come out of the box with Spark. The standard UDF’s in Spark operate […]

Read more
Spark

Using Kryo Serialization to boost Spark performance by 20%

Data serialisation plays a critical role in the performance of our data analytics scripts. In this article, we’ll discuss serialisation as a whole & then dive into Kryo serialisation in PySpark. What is serialisation? Often, we store our data in arrays, tables, trees, classes and other data structures, which are inefficient to transport. So, we […]

Read more
Spark

Repartition vs Coalesce

When we ingest data into Spark it’s automatically partitioned and those partitions are distributed across different nodes. So, imagine we have bought some data into Spark, it might be split across partitions like this: Partition 1 Partition 2 Partition 3 1, 7, 5, 9, 12 15, 21, 12, 17, 19 45, 32, 25, 99 When […]

Read more
Spark

Overcoming Spark Errors

In the last article on performance tuning Apache Spark, we spoke about caching, UDF’s and limiting your input data, which are all simple ways to see some potentially drastic performance improvements. In this post, we’re going to talk about the environmental factors that you can control to overcome some of the more common Spark issues. […]

Read more
Spark

Performance Tuning Apache Spark

Apache Spark provides us with a framework to crunch a huge amount of data efficiently by leveraging parallelism which is great! Tweaking it can be a bit of an art which is not so great! In this article, I will talk about some of the ways that I have successfully tuned the performance of my […]

Read more