Okay, so the first thing I should note is that you should avoid UDF’s (User Defined Functions) like the plague unless you absolutely have to; I have spoken about why that is here. Now that […]
Category: Spark
Fault Tolerant Friday: Creating robust pipelines in Python
When you’re dealing with tonnes of different datasources, issues crop up all the time: the datasource hasn’t been populated; the schema has changed; the schedule changes or the file format changes. The key with our […]
Starter Sundays: PySpark Basics, Selecting & Filtering Data
Welcome to this weeks Starter Sunday – which is where I cover off some basic concepts of Spark, Python or Hive. Today, we’re looking at selecting and filtering data from our dataframes in Spark, specifically, […]
Spark Saturdays: Spark Structured Streaming, Data Sources and Sinks
In Spark Streaming, we have the concept of sources and sinks. A source is very much what you would expect it to be – it’s the source of the data we will be ingesting into […]
Fault Tolerant Friday: Spark Structured Streaming, Introduction to check-pointing
Streaming applications need to be fault tolerant. They need to be able to cope with system failures, JVM crashes and all that sort of horrible stuff. To cope with that, Spark Streaming uses the concept […]
Spark Streaming: Introduction to DStreams
In my last article, I showed an example of a Spark Streaming application for ingesting application log files. Today, I am going to cover off an overview of DStreams in Spark Streaming. In subsequent articles, […]
Deploying a log monitor with Spark Structured Streaming
In this article I am going to cover how to create a simple streaming log monitor using Spark Streaming. In subsequent posts, I’ll go deeper on some of these aspects. The first thing we need […]
Handling corrupted input files during ETL
Often, we develop ETL pipelines and they just work. They do exactly what we expect them to do and produce the correct outputs. One day though, your script stops working and the error points to […]
Melting our Spark dataframes with Koalas
Melting dataframes is the process of taking a short, wide table and making it into a long thin one, using column headings as categorical data within the resulting dataframe. In the below example we have […]
Testing our UDF’s with UnitTest
Notebooks make it easy to test our code on the fly and check the output dataframe looks correct. However, it is good practice to run some unit tests with some edge cases – things you […]
Testing in PySpark (Part 1)
Testing is one of the most painful parts of data engineering, especially when you have a really huge data set. But it is an absolutely necessary part of every project, as without it, we can’t […]
Data Cleaning in PySpark
Most of our time as data engineers is spent cleaning up data. We have to deal with null values; duplicated rows; weird date formats; inconsistently capitalised words; accents (like the Spanish tilde in words); outliers […]
