Explore articles about PySpark, big data processing, data pipelines, testing strategies, and data engineering best practices.
Data quality is a rather critical part of any production data pipeline. In order to provide accurate SLA metrics and to ensure that the data is correct, it is important to have a way to validate the data and report the metrics for further analysis.
Have you ever stumbled upon a Spark ETL and you were left wondering how a simple loading of a dataset can take hours, even though the filtered dataset you are specifying is relatively small?
In this blogpost we will continue our journey of testing our Data Pipelines. If you haven't checked out the first post, make sure you do.
Unit testing is often regarded as a main pillar of testing your software applications, and it usually involves testing a single/unit component and ensuring that it covers all the edge cases the software developer can think of.