Cloud Data Warehousing

[At the request of the hotel staff or Adept Events organisation, you will have to use the CoronaCheck app to present a valid coronavirus entry pass.  You can generate a coronavirus entry pass using the CoronaCheck app on your mobile phone.

If corona restrictions prevent us from running in-person at the venue in Utrecht, we will revert to virtual delivery. In that case we will not run one full days but two morning sessions on March 22 – 23.]

As we stand, engineers have been building cloud tech based data warehouses in The Netherlands since 2015. But how do we actually design a cloud based data warehouse that really leverages the new technological capabilities the cloud brings? How does it differ from the classical, on premise data warehousing approach? And how can we go about implementing such a cloud based data warehouse technically?

Organizations are increasingly looking into the pros and cons of cloud data warehousing as the technology is evolving rapidly and more and more organisations are ‘taking the leap’. One of the main advantages of cloud analytical databases is their virtually ‘endless’, independently scalable (cheap) storage and compute. For most companies, however, migrating an existing data warehouse to the cloud is not that trivial. An important reason for this is that to take advantage of the independently scalable storage and compute, a lift-and-shift approach often does not work. The common cause for this is that the data architectural design principles that were used to design the classical data warehouse are outdated too!

In addition, the ‘modern data warehouse’ often demands a different consumption pattern than that we are used to in the classical data warehouse setup. As they are, these architectures were often focused on tactical / strategical information provisioning. However, in this day and age of data science and advanced analytics, it is only logical for the data scientist to be one of the primary users of the data warehouse.  After all, people have started to realize that data scientists currently perform a lot of work that could and should be covered under the label of ‘data warehousing’. A well designed modern data warehouse can therefore offer a competitive advantage for any modern data driven / information supported company.

Seminar

In this seminar, Rogier Werschkull will explain how one should deal with the solution and data architectural changes, offering lessons learned and best practices. Should you indeed move your data warehouse to the cloud? If so, what could be the new design principles forging your ‘modern data warehouse’? How do the different cloud data warehouse vendors compare? What cloud data warehouse solution and surrounding toolset should we choose and why? And how can we start building such a new data architecture, learning from how others have done it so far?

The cloud analytical databases that will be looked at will be restricted to the major players: Amazon Redshift, Google BigQuery, Snowflake and Azure SQL Data Warehouse /  Synapse. The supporting tools we will look at, fall in the category of EL tools (for onboarding data) like Fivetran and Stitch and modern ELT tools like Data Build Tools (DBT) and Matillion.

It is important to realize that while we will look at the technical aspects of these new cloud based solutions, the focus of this seminar will be to get a head start with implementing an analytics data architecture that you can really call a ‘modern data warehouse’. The essence of this is to grasp what implementing a cloud based ELT (not ETL!) analytical data architecture means in real life.

Learning objectives

Attendees will learn:

• What a data warehouse really is and what it is not. On architectural versus technology concerns
• What is ‘modern data warehousing’ about? How does it differ from ‘classical’ data warehousing, looking at consumer requirements / access patterns and the required data architecture?
• What is the role of the ‘Data Scientist’ in the context of the modern data warehouse?
• What does the cloud agnostic data warehouse data architecture look like that Rogier has been using since 2016
• What are the core data architectural principles that forge the modern data warehouse
• What is ‘separation of concerns’ , the primary architectural principle that drives modern data warehouse design?
• Why data warehouse automation should play a vital role in building the modern data warehouse
• Why can a modern data warehouse be built better using a cloud (native) solution?
• What are the major differences / pro’s / cons of the most commonly used cloud analytical databases: Snowflake, Google BigQuery, Amazon Redshift and Azure Synapse
• What additional tooling can be used to build a cloud based data warehouse with and why?
• What is the physical data modelling approached used per data warehouse layer and why?
• Most important lessons learned from technical and implementation perspective.

Who is It For?

Data management and analytics program and project managers, Data management and analytics architects. Data warehouse architects, Technology strategists and planners, Data engineers, Data warehouse / ETL developers and data integration specialists. Anyone with a role in migrating legacy data warehouses to the cloud.