Once you have received your connection string and set up an alert, you can start receiving flight changes and updates via the Event Hub.
Accessing your Alerts Data
Now that you have configured an Alert, OAG will begin streaming individual Alert events to your Microsoft Azure Event Hub. These are accessible using the connection strings that you were provided with when you completed your profile (at the same time as receiving your Account ID).
The event hub name is part of your connection string as the value of EntityPath. For example, in this connection string:
To find out more about Event Hubs we encourage you to read more here on the Microsoft knowledge base. This covers the basics of working with Event Hubs as well as links to articles with guidance on working within different code bases.
Each Alert is retained for 7 days before being permanently deleted. These 7 days of history allow customers to replay the recent history of messages which can be extremely useful if experiencing system issues or for disaster recovery. If you wish to keep a longer history then please ensure you store each event for future use.
Partitions
Event Hubs use a partitioned consumer model. This helps with having multiple concurrent streams of data from the Event Hub. At OAG the number of partitions in use will change over time and so you should ensure that when interacting with the Event Hub that you continuously validate the partitions in use. They are numbered sequentially with 0 being the first. OAG will always stream data for the same service number i.e. carrier code and flight number, to the same partition so you know you will always process the data in the correct sequence (sequence numbers are unique to each partition rather than to each Event Hub).
Checkpointing
Customers who choose to collect the data in batches will need to employ a checkpointing system to ensure they do not process duplicate data.
To keep track of your position in the event stream and know when the data stream is complete, you will need to store the position of the event sequence. Checkpointing is the responsibility of the consumer and occurs on a per-partition basis. If your reader disconnects and reconnects, you will be able to begin reading at the checkpoint stored from the last time you read from the event hub.
When your reader connects, it can pass the offset to the event hub to specify a point in the event stream to start reading the events. The same mechanism can be used to provide failover resilience to return to older data by setting a lower offset using the checkpointing process, thereby replaying the event stream from a point before the most recent checkpoint.
For more information on checkpointing, please refer to this article that explores the concepts in greater detail.
Other Helpful Resources
Microsoft Azure Event Hubs developer documentation
https://docs.microsoft.com/en-us/azure/event-hubs
Understanding the consumer side of Azure Event Hubs
https://devblogs.microsoft.com/azure-sdk/eventhubs-clients/
Java specific examples (includes EventPosition) –available in all supported languages
https://github.com/Azure/azure-event-hubs/tree/master/samples/Java/Basic
To access events from a position in time, offset or sequence number -available in all supported languages
https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.eventhubs.eventposition?view=azure-dotnet
https://www.codota.com/code/java/classes/com.microsoft.azure.eventhubs.EventPosition
Using AWS Lambda with self-managed Apache Kafka –use this to connect through to Azure Event Hub
https://docs.aws.amazon.com/lambda/latest/dg/kafka-smaa.html