DP-203 Azure Data Engineer

3 Minutes

Azure Synapse Analytics Event Delivery Guarantees Kappa Architecture Microsoft DP-203 Microsoft Exams Process Time Series Data

Configure Checkpoints/Watermarking During Processing – Design and Implement a Data Stream Processing Solution-3

Katie Cox 2024-08-07 0 Comment

An important metric to call out is Watermark Delay. The values represented in the metric indicate the realized processing delay of the event message. Under normal operation the Watermark...

4 Minutes

Azure Synapse Analytics Microsoft DP-203

Configure Checkpoints/Watermarking During Processing – Design and Implement a Data Stream Processing Solution-2

Katie Cox 2024-07-15 0 Comment

The data points provided in Table 7.7 are visually represented in Figure 7.41. Notice that event messages that arrive within a 5‐second window get the same watermark. For example,...

3 Minutes

Event Delivery Guarantees Microsoft DP-203

Configure Checkpoints/Watermarking During Processing – Design and Implement a Data Stream Processing Solution-1

Katie Cox 2024-06-23 0 Comment

A checkpoint is a concept that identifies a point in time when your data is in a verifiable, production‐ready state. For example, a checkpoint could be considered a backup...

4 Minutes

Microsoft DP-203 Microsoft Exams

Azure Databricks – Design and Implement a Data Stream Processing Solution

Katie Cox 2024-05-12 0 Comment

In Exercise 7.6 you created a stream processing solution using Azure Databricks. That stream processing solution used the Append output mode and displayed the data stream in the console,...

4 Minutes

Azure Synapse Analytics Event Delivery Guarantees Microsoft DP-203

Azure Synapse Analytics – Design and Implement a Data Stream Processing Solution

Katie Cox 2024-04-30 0 Comment

There are numerous methods for performing upserts from an Azure Stream Analytics data stream in Azure Synapse Analytics. One approach is to replicate the pattern shown in Figure 7.25....

2 Minutes

Azure Synapse Analytics Kappa Architecture Microsoft DP-203

Upsert Data – Design and Implement a Data Stream Processing Solution

Katie Cox 2024-03-09 0 Comment

An upsert in the stream processing context is the same as in other data analytics scenarios, such as batch or pipeline transformation processing. The action performed when executing an...

5 Minutes

Microsoft DP-203 Process Time Series Data

Upsert Stream Processed Data in Azure Cosmos DB – Design and Implement a Data Stream Processing Solution

Katie Cox 2024-02-21 0 Comment

FIGURE 7.34 Upserting streamed data on Azure Cosmos DB—configuring output FIGURE 7.35 Streaming data into Azure Cosmos DB using the command console FIGURE 7.36 Inserting streamed data on Azure...

5 Minutes

Kappa Architecture Microsoft DP-203

Handle Schema Drift in Azure Stream Analytics – Design and Implement a Data Stream Processing Solution

Katie Cox 2024-01-13 0 Comment

FIGURE 7.38 Handling schema drift in a stream processing solution in ADLS FIGURE 7.39 Handling schema drift in a stream processing solution in Azure Cosmos DB The most critical...

4 Minutes

Azure Synapse Analytics Microsoft DP-203 Process Time Series Data

Process Data Within One Partition – Design and Implement a Data Stream Processing Solution

Katie Cox 2023-12-30 0 Comment

Partitioning has been covered in many different scenarios and contexts. For example, in Exercise 2.2 you provisioned an Azure Cosmos DB and changed the partition key to /pk, and...

4 Minutes

Event Delivery Guarantees Kappa Architecture Microsoft DP-203

Develop and Create Windowed Aggregates – Design and Implement a Data Stream Processing Solution

Katie Cox 2023-11-05 0 Comment

FROM [brainwaves].[BrainwaveWindowMedians]WHERE WindowId = ’10s Tumble’ORDER BY IngestionTime The first action in Exercise 7.7 was to create a new table to store the stream of brain wave data readings....