Amazon SQS - An API to delay queue messages
Hi, I’ve always found scheduling queue messages to be a pain with SQS. SQS has support only up to 15 minutes. Anything beyond that requires you to build your own service to handle the delays.
I've always wished there was an API that just handled the scheduling for me. As such I’m developing an API to make scheduling messages to SQS and SNS dead simple. Call the endpoint with your payload and delay time, and it automatically publishes to your queue or topic when you need it. I have a prototype here:
I’d love to hear some feedback on this idea and whether it solves a real painpoint. I’d also love to hear about the last time anyone had to do job scheduling - how well did your solution work? Would anyone hacking on their own projects find this useful right now?
Personally, I wouldn't use SQS to schedule or delay jobs. I would look to a purpose built tool that is made to handle job scheduling. I use and highly recommend Sidekiq. You'll see "scheduled jobs" listed as the very first feature. If you look at SQS, scheduling is nowhere to be found on the list of "main benefits."
Hey jlar188, I think we mean different things by "job scheduling". I think you're talking about scheduling overnight batch jobs, like a background process. I'm talking about scheduling messages in a message queue as part of a microservices architecture.
This article explains the difference better than I can: https://www.mikeperham.com/2011/05/04/background-processing-vs-message-queueing/
Edit: deleted my comment below because I was unclear
No, I am not talking about overnight batch jobs. I am referring to the fact that you can schedule a job with Sidekiq to run whenever you want - 5 seconds from now, 3 hours from now, or at 8:00am on Tuesday. That sounds like scheduling to me.
Regardless, my point was less about Sidekiq as the specific solution and more about using a tool that is designed to be a job scheduler and not just a queuing system. Just my opinion.
Ok thanks for the feedback
This comment was deleted 6 years ago
We use lambdas to get around scheduling.
Hey whipdancer, could you explain a little more what you mean?
You can schedule lambdas on an interval. We use scheduled lambdas to do sqs message processing.