aws-sqs producer does not support new FIFO queues

[ecorengia]

Hi all, I have being using aws-sqs component without problems on the past
using standard SQS queues.

But now, I'm not able to send messages to the new FIFO queue mechanism
available on some regions.

I have tried the consumer piece, and that seems to be okay. The problems is
with the producer. When I try to send a new message I get an error saying
"The request must contain the parameter MessageGroupId".

I'm using Camel version 2.18.3. This is my test route:

from("direct:" + ROUTE_ID + "_send_sync_notification")
.routeId(ROUTE_ID + "_send_sync_notification")
.process(exchange -> exchange.getIn().getHeaders().clear())

.to("aws-sqs://{{SQS_fifo_queue}}?amazonSQSClient=#myClient&region=us-east-2");

Stacktrace is copied below. Do you know if it is a known issue or if I'm
missing something? I couldn't find this problem on your issue tracker.

Thanks! Emilio.

[2017-05-02 13:18:43.040] boot - 18028 WARN [Camel (TitlesIntegration)
thread #38 - Multicast] --- DeadLetterChannel: Failed delivery for
(MessageId: ID-ecorengia-laptop-37936-1493741878671-0-23 on ExchangeId:
ID-ecorengia-laptop-37936-1493741878671-0-24). On delivery attempt: 0
caught: com.amazonaws.AmazonServiceException: The request must contain the
parameter MessageGroupId. (Service: AmazonSQS; Status Code: 400; Error Code:
MissingParameter; Request ID: 2b1a16b0-cf4c-5115-9b90-47327cee3ec9)
com.amazonaws.AmazonServiceException: The request must contain the parameter
MessageGroupId. (Service: AmazonSQS; Status Code: 400; Error Code:
MissingParameter; Request ID: 2b1a16b0-cf4c-5115-9b90-47327cee3ec9)
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1378)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:924)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454)
at
com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365)
at
com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1741)
at
com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1711)
at
com.amazonaws.services.sqs.AmazonSQSClient.sendMessage(AmazonSQSClient.java:1487)
at
org.apache.camel.component.aws.sqs.SqsProducer.process(SqsProducer.java:62)
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681)
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:609)
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:238)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681)
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:609)
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:238)
at org.apache.camel.processor.Splitter.process(Splitter.java:108)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:128)
at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:96)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:827)
at
org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:85)
at
org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:320)
at
org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)





--
View this message in context: http://camel.465427.n5.nabble.com/aws-sqs-producer-does-not-support-new-FIFO-queues-tp5798568.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[Claus Ibsen]

Hi

Can you set the header yourself, eg setHeader("MessageGroupId",
constant("something"))

I am not sure if the camel-aws component is supposed to set some group
id by itself. Isn't that intended to be set by the end user to
something relevant about the message, such as customId or appId or
something.

Post by ecorengia
Hi all, I have being using aws-sqs component without problems on the past
using standard SQS queues.
But now, I'm not able to send messages to the new FIFO queue mechanism
available on some regions.
I have tried the consumer piece, and that seems to be okay. The problems is
with the producer. When I try to send a new message I get an error saying
"The request must contain the parameter MessageGroupId".
from("direct:" + ROUTE_ID + "_send_sync_notification")
.routeId(ROUTE_ID + "_send_sync_notification")
.process(exchange -> exchange.getIn().getHeaders().clear())
.to("aws-sqs://{{SQS_fifo_queue}}?amazonSQSClient=#myClient&region=us-east-2");
Stacktrace is copied below. Do you know if it is a known issue or if I'm
missing something? I couldn't find this problem on your issue tracker.
Thanks! Emilio.
[2017-05-02 13:18:43.040] boot - 18028 WARN [Camel (TitlesIntegration)
thread #38 - Multicast] --- DeadLetterChannel: Failed delivery for
ID-ecorengia-laptop-37936-1493741878671-0-24). On delivery attempt: 0
caught: com.amazonaws.AmazonServiceException: The request must contain the
MissingParameter; Request ID: 2b1a16b0-cf4c-5115-9b90-47327cee3ec9)
com.amazonaws.AmazonServiceException: The request must contain the parameter
MissingParameter; Request ID: 2b1a16b0-cf4c-5115-9b90-47327cee3ec9)
at
com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1378)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:924)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454)
at
com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365)
at
com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:1741)
at
com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:1711)
at
com.amazonaws.services.sqs.AmazonSQSClient.sendMessage(AmazonSQSClient.java:1487)
at
org.apache.camel.component.aws.sqs.SqsProducer.process(SqsProducer.java:62)
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681)
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:609)
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:238)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:681)
at
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:609)
at
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:238)
at org.apache.camel.processor.Splitter.process(Splitter.java:108)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:120)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:128)
at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:96)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:542)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:197)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at
org.apache.camel.processor.MulticastProcessor.doProcessParallel(MulticastProcessor.java:827)
at
org.apache.camel.processor.MulticastProcessor.access$200(MulticastProcessor.java:85)
at
org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:320)
at
org.apache.camel.processor.MulticastProcessor$1.call(MulticastProcessor.java:305)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
View this message in context: http://camel.465427.n5.nabble.com/aws-sqs-producer-does-not-support-new-FIFO-queues-tp5798568.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[ecorengia]

Thanks for your response Claus!

I have tried setting up manually MessageGroupId, but not luck error still
the same. I did all the same I do for standard SQS queues, but for FIFO ones
I face this problem on the producer side. I use ARN format to specify the
queue name, I never had to set anything else aside of amazonSQSClient.



--
View this message in context: http://camel.465427.n5.nabble.com/aws-sqs-producer-does-not-support-new-FIFO-queues-tp5798568p5798575.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[Claus Ibsen]

Hi

I wonder if somehow that header gets dropped, or does aws-sdk have
some special API you must call to set that header?

Maybe you can find some existing code / docs / tutorial on using those
FIFO queues and we can take a look, and see what needs to be improved
in camel-aws.

And you are welcome to log a JIRA about this as well.

And we love contributions btw
http://camel.apache.org/contributing

Post by ecorengia
Thanks for your response Claus!
I have tried setting up manually MessageGroupId, but not luck error still
the same. I did all the same I do for standard SQS queues, but for FIFO ones
I face this problem on the producer side. I use ARN format to specify the
queue name, I never had to set anything else aside of amazonSQSClient.
--
View this message in context: http://camel.465427.n5.nabble.com/aws-sqs-producer-does-not-support-new-FIFO-queues-tp5798568p5798575.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[ecorengia]

Yep. It seems like MessageGroupId is required by FIFO queues only. I will
investigate a little bit more and try to add an improvement on camel-aws.

On Tue, May 2, 2017 at 4:15 PM, Claus Ibsen-2 [via Camel] <

Post by Claus Ibsen
Hi
I wonder if somehow that header gets dropped, or does aws-sdk have
some special API you must call to set that header?
Maybe you can find some existing code / docs / tutorial on using those
FIFO queues and we can take a look, and see what needs to be improved
in camel-aws.
And you are welcome to log a JIRA about this as well.
And we love contributions btw
http://camel.apache.org/contributing
On Tue, May 2, 2017 at 3:06 PM, ecorengia <[hidden email]

Post by ecorengia
Thanks for your response Claus!
I have tried setting up manually MessageGroupId, but not luck error

still

Post by ecorengia
the same. I did all the same I do for standard SQS queues, but for FIFO

ones

Post by ecorengia
I face this problem on the producer side. I use ARN format to specify

the

Post by ecorengia
queue name, I never had to set anything else aside of amazonSQSClient.
--
View this message in context: http://camel.465427.n5.nabble.

com/aws-sqs-producer-does-not-support-new-FIFO-queues-
tp5798568p5798575.html

Post by ecorengia
Sent from the Camel - Users mailing list archive at Nabble.com.

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2
------------------------------
If you reply to this email, your message will be added to the discussion
http://camel.465427.n5.nabble.com/aws-sqs-producer-does-not-
support-new-FIFO-queues-tp5798568p5798577.html
To unsubscribe from aws-sqs producer does not support new FIFO queues, click
here
<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=5798568&code=ZW1pbGlvY29yZW5naWFAZ21haWwuY29tfDU3OTg1Njh8MzUxMjcwNjM0>
.
NAML
<http://camel.465427.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>

--
*Eng. Emilio Corengia <https://about.me/emilio.corengia>** | *
***@gmail.com
*Skype* emilio.corengia *|* *M* [phone]




--
View this message in context: http://camel.465427.n5.nabble.com/aws-sqs-producer-does-not-support-new-FIFO-queues-tp5798568p5798578.html
Sent from the Camel - Users mailing list archive at Nabble.com.

[ecorengia]

I have opened https://issues.apache.org/jira/browse/CAMEL-11224 in order to
follow up this improvement.



--
View this message in context: http://camel.465427.n5.nabble.com/aws-sqs-producer-does-not-support-new-FIFO-queues-tp5798568p5798582.html
Sent from the Camel - Users mailing list archive at Nabble.com.