4

I am taking an AWS course and trying to understand how the AWS SQS (Simple Queue Service) works. I am a little bit confused by the visibility timeout. I know that visibility timeout is the duration for a message to be processed within.

This is the model

enter image description here

As you can see in the diagram, the consumer will pull the message or the messages and process the message or the messages within the visibility timeout. If the message or the messages are not processed within the visibility timeout, the consumer will not receive or process the message or the messages and the messages will remain in the queue.

The followings are my confusions:

  1. Is the visibility timeout valid for all the messages together in one poll? Or the timeout is applied to each message separately? For example, the consumer polls 5 messages at a time. If 2 of them are not processed within the timeout, 2 of them are sent back to the queue and the other three are deleted. Or all the 5 messages are processed within a timeout and if any of them failed, all of them get sent back to the queue.

  2. This is my second confusion. How is the visibility timeout valid? Is that for the time the consumer starts polling messages and receive the messages? Or is that valid for the time the consumer starts polling messages and finish processing messages on the consumer end (for example, saving messages into the database)?

Improve this question

1 Answer 1

Reset to default
8

The visibility timeout begins when you receive the message. If you receive multiple messages in a batch, then the visibility timeout for all of them begins at that time.

enter image description here

To tell SQS that you have completely processed an SQS message, you delete the message from SQS using DeleteMessage. If you finish processing just 3 of 5 messages, then you delete just those 3 messages from SQS.

In-flight messages become visible again to other consumers if you do not delete them from SQS before the visibility timeout expires, or you explicitly make them visible again by setting their visibility timeout to zero (which is effectively you telling SQS "I am not going to process this message, let someone else"). You can also increase the visibility timeout dynamically, once you are processing an in-flight message.

Improve this answer
7
  • 2
    You can change the visibility timeout on a message by message basis. It's a pretty common pattern to have the timeout set to a value that is relatively short, and then have the processor of the item determine what to set the timeout to, based on what it needs to do. That way if the process errors out before the message updates the timeout another process can pick it up quicker, but once the timeout is updated the time to do the work is, hopefully, more in line with what the timeout is set to.
    – Jason Wadsworth
    Commented Jan 28, 2020 at 17:33
  • 3
    @JasonWadsworth Right, and you can return an in-flight message to the visible queue by changing its visibility timeout to zero. I didn't want to over-complicate my answer. But you're right, that's an important and valuable strategy.
    – jarmod
    Commented Jan 28, 2020 at 17:42
  • Hi, but I still get a question though. You said the visibility timeout for all the messages begins when the consumer receives the messages. So the visibility timeout will start at the same time for all the messages. For example, if visibility timeout is 10 seconds, and the consumer fetched 5 messages., the consumer will need to process all the 5 messages within 10 seconds. Is that correct?
    – Wai Yan Hein
    Commented Jan 29, 2020 at 11:02
  • 1
    Yes, to avoid any of them being made visible in the queue again. But you can extend the visibility timeout per message, if needed. Note that if you don’t explicitly delete messages as you process them and your Lambda fails then the entire batch will be made visible again, even if you actually completed processing some of the messages. So ideally you should explicitly delete messages as you complete their processing.
    – jarmod
    Commented Jan 29, 2020 at 12:58
  • 1
    @hhprogram per doc, the new timeout period applies only to the particular receipt of the message. ChangeMessageVisibility doesn't affect the timeout of later receipts of the message. If you return the message to the queue by setting the message's visibility timeout to zero, you're essentially saying "I am not going to process this message". That makes it available to another consumer, who will receive it with the queue's default visibility timeout.
    – jarmod
    Commented Aug 14, 2021 at 17:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.