Content about Databricks, a unified analytics platform for big data and machine learning.
Have you ever stumbled upon a Spark ETL and you were left wondering how a simple loading of a dataset can take hours, even though the filtered dataset you are specifying is relatively small?
In this blogpost we will continue our journey of testing our Data Pipelines. If you haven't checked out the first post, make sure you do.
Unit testing is often regarded as a main pillar of testing your software applications, and it usually involves testing a single/unit component and ensuring that it covers all the edge cases the software developer can think of.
