Develop Spark ETL pipelines with no risk against production data

Delivering high-quality data products requires strict testing of pipelines before deploying those into production. Today, to test using quality data, one either needs to use a subset of the data, or is forced to create multiple copies of the entire data. Testing against sample data is not good enough. The alternative, however, is costly and time consuming. We will demonstrate how to get the entire production data set with zero-copy.

You will learn:

• Set up your environment in under 5 minutes

• Create multiple isolated testing environments without copying data

• Easily run multiple test on your environment using git-like operations (such as commit, branch, revert, etc.)

Speakers:

• Iddo Avneri
• Amit Kesarwani