Message Resubmission challenges with ‘RequiresDuplicateDetection’ enabled Service Bus Entities

|  Posted: May 21, 2018  |  Categories: Azure Service Bus ServiceBus360

We have come across many business scenarios involving message resubmission to an Azure Service Bus entity with ‘Requires Duplicate Detection’ property enabled, and the requirement is to prevent message loss during resubmission. ServiceBus360 features are designed after a careful investigation into real word integration solution challenges and feedback from our existing customers. With a lot of attention to detail, we have built this capability to avoid message loss when you try to resubmit messages to Azure Service Bus entities with ‘Requires Duplicate Detection’ property enabled. To understand this article, it is a pre-requisite to have knowledge of the ‘Requires Duplicate Detection’ property and ‘Duplicate Detection Time History Window’ property on an Azure Service Bus entity.

Requires Duplicate Detection

When set to true, the ‘Requires Duplicate Detection’ property will ensure that all enqueued messages with a duplicate value of the Message-Id property within the ‘Duplicate Detection History Time Window’ will be ignored.

Duplicate Detection Time History Window

If duplicate detection is enabled for a queue the ‘Duplicate Detection History Time Window’ value will specify how long the Message-Id values for the received messages will be retained to check for duplicate messages. The property is a Timespan with a default value of 10 minutes and a maximum allowed value of 7 days.

Now, consider a business scenario, where an integration solution orchestration has an application upstream, that pushes messages into a queue with the following properties:

  • Requires Duplicate Detection: True
  • Duplicate Detection History Time Window: 10 minutes
  • Time To Live: 1min
    NOTE: Once the entity is created with the ‘Requires Duplicate Detection’ property enabled, it can never be modified.

The application downstream is expected to process the active messages from the queue defined above.

ServiceBus360 can facilitate Queue property definition at the time of creation itself which is not possible in Azure Portal.

Create queue

 

In the image below you can see that ServiceBus360 is displaying the properties of the created queue, with ‘Enable Dead-Lettering on Message Expiration’ turned on and ‘Default message Time to Live’ set to 1 minute. With this configuration, as the application upstream enqueue messages to the created queue, after 1 minute the messages will automatically be dead lettered due to ‘TTL Expired’ exception.

edit-queues

ServiceBus360 has the capability to send messages to queue for testing purpose, to mock the application upstream. Send 10 messages to the queue and after 1 minute, the active messages will be dead-lettered.

dead letter messages

The challenge with this scenario

In the above-explained scenario, the message in the queue get dead -lettered before the downstream application could process them. Even if the ‘Default message Time to Live’ is redefined to a higher value, the requirement is to resubmit the already dead lettered messages to make them available for the downstream application to process. As the ‘Duplicate Detection History Time Window‘ is 10 mins, while resubmitting the dead lettered messages, the message will be lost being detected as the duplicate message.

Solution with ServiceBus360

Servicebus360 provides a solution to overcome the above challenge and achieve the above requirement. In ServiceBus360 while resubmitting a message, if the selected target queue has ‘Requires Duplicate Detection’ property enabled, the system would provide the following options to define the message id

  • Regenerate the selected single or multiple messages id (new GUID will be created and assigned to every message id)
  • Edit the message id to meet your business requirement, like invoice/202#/####.
    duplicate detection message resubmit

As the new message ids are regenerated, these messages will not be lost, when resubmitted to the queue with ‘Requires Duplicate Detection’ property enabled. Also, ServiceBus360 helps you to automate these tasks by creating Activities. For example, while creating a ‘Dead- Letter Activity’ on a ‘Requires Duplicate Detection’ property enabled ServiceBus Messaging Entity, enabling ‘Regenerate Message Ids’ option will take care of resubmitting messages with the newly generated GUID as message Id.

Configure activities with duplicate detection messages

Read more documentation on ServiceBus360 capabilities – Documentation link

Try servicebus360

Author: Reshma Sulthan

Junior Software Test Engineer.

One Platform Operations, Monitoring and Analytics Software
BizTalk360

microsoft biztalk

Learn more

Over 500 customers across 30+ countries depend on BizTalk360

ServiceBus360

Azure Composite Applications

Learn more

Start managing your Azure services in minutes

One Platform - Operations, Monitoring and Analytics Software
BizTalk360

microsoft biztalk

Learn more

Over 500 customers across 30+ countries depend on BizTalk360

One Platform - Operations, Monitoring and Analytics Software
ServiceBus360

Azure Composite Applications

Learn more

Start managing your Azure services in minutes