Your journey into Data Engineering: Prepping Ingredients for Cooking
Prep your ingredients for cooking and get your organization up and running for the workshop Data Engineering.
This workshop will give you insights into the relevant services that are available on the Azure platform and the tools to determine the tasks of a Data Engineer. You will gain an understanding of the appropriate storage service to choose to implement a solution based on a given set of business and technical requirements. Also, you will become familiar with several transformation techniques using Azure Data Factory, Data flow, and Azure Databricks. And in conclusion, you will gain an understanding of how to perform data loads into different data storage options whilst transforming data.
What are the benefits of using Data Engineering?
1. Informed Business Decisions
Clean data helps make businesses making informed decisions. Using untransformed data will cost a lot of time, money and prevent uninformed decisions and quality problems. If you cook a whole carrot without cutting it up first, it takes longer to get cooked compared to when you cut it up in small pieces.
2. Speed and Efficiency
Clean and correctly stored data will improve the data gathering time as you don’t need to search where what data is landed. Your data is organized and you know where to find what. Just like in the kitchen where hopefully you didn’t store your milk in the dry store but in the fridge where it belongs.
3. Insights
Once your data is clean and transformed it’s easier to visualize and analyze. This way you can gain insights as well as perform predictions on future relations and prepare your business for it. Because of this, your marketing department will for example be able to use the data for targeted campaigns.
Who is this workshop for?
For those of you familiar with Data Engineering... this is the workshop for you! We’re going to prepare our ingredients for cooking. Please read article 1 Cooking with Data and AI to get an understanding of the analogy of cooking we perform at Intercept to help you with your Data and AI projects and gain an understanding of this evolving world.
Attendees of this workshop should have some expertise in integrating, transforming, and consolidating data from structured and unstructured data systems into a structure that is suitable for building analytical solutions.
Also, you should be able to interpret data through exploration and be able to build and maintain secure and compliant data processing pipelines. The attendees can ensure that data pipelines and data stores are high-performing, efficient, organized, and reliable.
You also need to know data processing languages such as SQL, Python, or Scala.
What can you expect during the workshop?
Data Engineering services on Azure
Data storage options
When to use which service
Data Transformation techniques
You will learn about:
Data Engineering Services on Azure
Data storage options
Data transformation and cleansing techniques techniques
All whilst cooking a dish and live demoing the Azure services and techniques for Data Engineering. Get ready for some interactions😉
Tags
Data
Data and AI
There are no planned events at this moment.
Possibly interesting as well:
The basic principles of Data Science on Azure
Do you want to start a data science project on Azure? Starting from scratch? Don’t miss this e-book!
When it comes to choosing between data processing tools on Azure, there are options aplenty! Think of Azure Data Factory, Azure Databricks, or Azure Synapse Analytics. But… how do we know what tool is right for us? It’s not that difficult, really! That is, if you know what these tools can do! Our Data Engineer, Lisa Hoving, will teach you how to make the right decision during this deep dive.
Breaking the bank on Azure: what Apache Spark tool is the most cost-effective?
Figuring out which tool has the lowest execution costs is often a complicated task. In this article, we break down the costs. With this information, you can decide which option is best for your company.
Data Factory vs. Databricks: When your love for data is missing some (Apache) Spark
How do you choose the data partner that suits you best? Azure Data Factory has great benefits when you start out, but Apache Spark allows you to explore deeper layers. So is Azure Databricks your data partner that brings the "Spark" back into your data project? One thing is for sure, in Azure you don't necessarily have to choose a monogamous data solution.
Today, companies save all kinds of data, even when they don’t yet have a use case for it. But where should all these different types of datasets be stored? This is where the Data Lake comes in!
What your data journey looks like depends on your use case. Nevertheless, within any data project, there are five generic steps that any company will take to get insights from their data. Have a look!
The Cloud is even with a lot of data traffic the solution for ISVs
Many organizations, mainly ISVs, think that moving to the Public Cloud comes with high costs. This does not have to be the case at all. In fact, moving to the Public Cloud is often more cost beneficial.
