Flight Info Alerts - Connecting to your Event Hub

Once you have received your connection string and set up an alert, you can start receiving flight changes and updates via the Event Hub.

Accessing your Alerts Data

Now that you have configured an Alert, OAG will begin streaming individual Alert events to your Microsoft Azure Event Hub. These are accessible using the connection strings that you were provided with when you completed your profile (at the same time as receiving your Account ID).

To find out more about Event Hubs we encourage you to read more here on the Microsoft knowledge base. This covers the basics of working with Event Hubs as well as links to articles with guidance on working within different code bases.

Each Alert is retained for 7 days before being permanently deleted. These 7 days of history allow customers to replay the recent history of messages which can be extremely useful if experiencing system issues or for disaster recovery. If you wish to keep a longer history then please ensure you store each event for future use.

Partitions


Event Hubs use a partitioned consumer model. This helps with having multiple concurrent streams of data from the Event Hub. At OAG the number of partitions in use will change over time and so you should ensure that when interacting with the Event Hub that you continuously validate the partitions in use. They are numbered sequentially with 0 being the first. OAG will always stream data for the same service number i.e. carrier code and flight number, to the same partition so you know you will always process the data in the correct sequence (sequence numbers are unique to each partition rather than to each Event Hub).

Checkpointing


Customers who choose to collect the data in batches will need to employ a checkpointing system to ensure they do not process duplicate data.

To keep track of your position in the event stream and know when the data stream is complete, you will need to store the position of the event sequence. Checkpointing is the responsibility of the consumer and occurs on a per-partition basis. If your reader disconnects and reconnects, you will be able to begin reading at the checkpoint stored from the last time you read from the event hub.

When your reader connects, it can pass the offset to the event hub to specify a point in the event stream to start reading the events. The same mechanism can be used to provide failover resilience to return to older data by setting a lower offset using the checkpointing process, thereby replaying the event stream from a point before the most recent checkpoint.

Other Helpful Resources


Microsoft Azure Event Hubs developer documentation

https://docs.microsoft.com/en-us/azure/event-hubs

Understanding the consumer side of Azure Event Hubs

https://www.serverless360.com/blog/understanding-consumer-side-of-azure-event-hubs-checkpoint-initialoffset-eventprocessorhost

https://devblogs.microsoft.com/azure-sdk/eventhubs-clients/

Java specific examples (includes EventPosition) –available in all supported languages

https://github.com/Azure/azure-event-hubs/tree/master/samples/Java/Basic

To access events from a position in time, offset or sequence number -available in all supported languages
https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.eventhubs.eventposition?view=azure-dotnet

https://www.codota.com/code/java/classes/com.microsoft.azure.eventhubs.EventPosition

Using AWS Lambda with self-managed Apache Kafka –use this to connect through to Azure Event Hub


https://docs.aws.amazon.com/lambda/latest/dg/kafka-smaa.html

Azure Event Hubs client library for Java -Version 5.7.0


https://docs.microsoft.com/en-us/java/api/overview/azure/messaging-eventhubs-readme?view=azure-java-stable