NOV 2022 - Modernize your Data wrangling approach for Datalakes by Dan King

Abstract

The evolution of technology and sheer amount of data increases the use of modern-day architectures. Carefully using the term modern day architecture because the data lake architecture isn’t just for data warehousing and prescriptive analytics. It spans multiple modern-day use cases and architectural designs. Data wrangling and choosing your tools effectively continues to change as well. Think about Azure data factory v1 and how it evolved into ADF V2. Look at some of the additional data movement components and options. Databricks and notebooks modernized the movement of data and ACID transaction with delta lake. Now we have witnessed the next shift with Power Query and data wrangling within your ADF pipelines. In this session we will look at some of these services and open mic discuss the pros and cons of each. I will demo different data movement techniques and provide a forum to talk through experiences.

Speaker Bio

Dan King is currently a Solutions Architect Competency lead at 3Cloud Solutions. His daily activities are not only working with customers but with a primary focus on repeatable processes and standardizing go to market analytics architectural design to fit the needs of different industries. From a Microsoft mindset you can relate it to common data models with common ETL/ELT processes.

His experience spans over a decade that started with networking and telecommunication to being a SQL database administrator with a final landing zone in data analytics. Which today is his core focus and goal. With expertise in analyzing data, data movement and ETL design, data modelling and visualization, in his own words he has been honored to see how organizations take data from various sources through ingestion as it flows to storage and eventually lands in the presentation layer.

Dan holds multiple Microsoft certifications and continues to progress with the changes in technology. He loves to workout, outdoor sports, and finds tremendous value in training his peers in areas that he holds knowledge in. Family is a core value, and he loves family. He views the SQL community as his extended family.