Survive Failures and Increase Responsiveness When Using Storage Queue
Improve the customer’s UI experience with these solutions
Latest posts by Ranvijay Kumar
When designing applications for scale, many use Microsoft Azure Storage Queues for cloud messaging between components. Some complications of using the software involve end-user satisfaction, however. Any transaction on database tables requires updating Azure Search Document asynchronously. When an updated record is the master table record, then it is required to retrieve all related Azure search index documents and update them. This can be very time consuming, as there are sometimes as many as 5,000 documents.
Asynchronous calls seem to be the best choice with batch updates, but Azure index refresh can cause additional delays, making the end user wait for their operation to complete. Also, a couple of application services might not be available during their wait time.
Another problem that may arise while using Azure Storage Queue involves handling failures during the aforementioned operations.
There are two types of failure of application:
- Transient, self-healing failures such as intermittent network connectivity issues, causing additional delays
- Enduring failures that require intervention, need to be monitored and brought to the technical team, which can provide root cause analysis and process in the backend.
- Transfer heavy workload task to Queue (Messaging System), which acts as a buffer between task and services that can handle it in the backend (without which it might result in failure or timeout).
- Enable service to handle transient failures by retry (maximum three retries), and notify the service team of enduring failure.
- Queue messages are processed in less time. Service can be scheduled as continuous web jobs, so queue messages are processed by the task in less wait time.
- Configure for scale instances during heavy workloads. At any special event, handling each message is consumed by only one instance at any point in time.
- Use PeekLock mode when it retrieves a message from the queue, so that the message is not deleted and will instead be hidden. This allows the next message in the queue to be picked by other instances and ensures the same message is not processed multiple times.
- Ensure that business logic is not blocked while requests are being processed.
- Request from approach is posted to queue in the form of messages. The processing service receives the message from the queue and processes them by enabling a pool of instances, handling multiple requests.
- Add an additional table to handle message status. This gets updated only when the message is processed successfully by the processing service.
- Able to handle wide variations in heavy workload
- Helps impact of availability and responsiveness for both application and service instances.
- Multiple long-running requests can be processed by different instances of processing service concurrently
- The high probability of getting lost requests due to processing service failure are now handled and processed by working instances
- Scalable solution instances can be added or removed depending on workload
- If a message fails even after retry, the technical team is notified with the error details so that they can provide analysis and a quicker solution
- Retry of three attempts are made per request to avoid transient failures
These solutions can be used when:
- The workload for an application is divided into tasks that can run asynchronously
- Tasks are independent and can run in parallel
- The volume of work is highly variable, requiring a scalable solution
- The solution must provide high availability, and must be resilient if the processing for a task fails
Cannot be used when:
- It is not easy to separate the application workload into discrete tasks, or if there is a high degree of dependence between tasks
- Tasks must be performed synchronously, and the application logic must wait for a task to complete before continuing
- Tasks must be performed in a specific sequence
- Maximum 32 messages can be processed by processing service at a time
- Message size should not exceed 64KB
Azure SDK, Azure Storage to create Queue, API programming, C# Programming language
These solutions can be used to improve customer UI experience and distribute load to the backend services/task using Azure PaaS features.