REST endpoint to rabbitmq queue connection timeout

[Valdis Andersons]

Hi All,

Would anyone know how to configure the timeout values for the below connection pool or the connection pool itself? Been having some issues lately in a test environment with this and the timeout check seems to happen every few milliseconds producing huge amounts of log files.
We're suspecting it's a network issue that's causing this as there is nothing much between the REST endpoint on the buffer queue and only if there is some decent load on the endpoint (10req/sec or so):

rest(restEndpoint).post(postEndpoint)
.type(FeedbackRequest.class)
.route()
.process(this::process)
.startupOrder(Integer.MAX_VALUE - 2)
.marshal().json(JsonLibrary.Jackson, String.class)
.to(feedbackEndpoint)
.endRest();

where process() just sets a persistence header for RabbitMQ to flush the message to disk:


exchange.getIn().setHeader("rabbitmq.DELIVERY_MODE", 2);


2018-10-04 17:01:43,275 [https-jsse-nio-8443-exec-65] ERROR o.a.c.p.DefaultErrorHandler ID-VM1-1538665977944-0-101281 - Failed delivery for (MessageId: ID-VM1-1538665977944-0-101280 on ExchangeId: ID-VM1-1538665977944-0-101281). Exhausted after delivery attempt: 1 caught: java.util.NoSuchElementException: Timeout waiting for idle object

Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route24 ] [route24 ] [restlet:///cch/feedback?restletMethods=POST ] [ 3798]
[route24 ] [process22 ] [***@0x37c71429 ] [ 0]
[route24 ] [marshal1 ] [marshal[***@c940d2] ] [ 0]
[route24 ] [route24 ] [rabbitmq://vm1:5671/feedbackBufferExchange?connectionFactory=rabbitConnec] [ 3782]

Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
at org.apache.camel.component.rabbitmq.RabbitMQProducer.execute(RabbitMQProducer.java:73)
at org.apache.camel.component.rabbitmq.RabbitMQProducer.basicPublish(RabbitMQProducer.java:285)
at org.apache.camel.component.rabbitmq.RabbitMQProducer.processInOnly(RabbitMQProducer.java:272)
at org.apache.camel.component.rabbitmq.RabbitMQProducer.process(RabbitMQProducer.java:195)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
at org.apache.camel.component.restlet.RestletConsumer$1.handle(RestletConsumer.java:68)
at org.apache.camel.component.restlet.MethodBasedRouter.handle(MethodBasedRouter.java:58)


Thanks,
Valdis

[Zoran Regvart]

Hi Valdis,
I think you're looking for `channelPoolMaxSize` and
`channelPoolMaxWait` parameters[1]. From the snippet, I guess, you
would set them on the `feedbackEndpoint`, and from the message history
I see that you already set the `connectionFactory`, so you would
configure these much in the same way.

zoran

[1] https://github.com/apache/camel/blob/master/components/camel-rabbitmq/src/main/docs/rabbitmq-component.adoc

On Thu, Oct 4, 2018 at 6:15 PM, Valdis Andersons

Post by Valdis Andersons
Hi All,
Would anyone know how to configure the timeout values for the below
connection pool or the connection pool itself? Been having some issues
lately in a test environment with this and the timeout check seems to happen
every few milliseconds producing huge amounts of log files.
We’re suspecting it’s a network issue that’s causing this as there is
nothing much between the REST endpoint on the buffer queue and only if there
rest(restEndpoint).post(postEndpoint)
.type(FeedbackRequest.class)
.route()
.process(this::process)
.startupOrder(Integer.MAX_VALUE - 2)
.marshal().json(JsonLibrary.Jackson, String.class)
.to(feedbackEndpoint)
.endRest();
exchange.getIn().setHeader("rabbitmq.DELIVERY_MODE", 2);
2018-10-04 17:01:43,275 [https-jsse-nio-8443-exec-65] ERROR
o.a.c.p.DefaultErrorHandler ID-VM1-1538665977944-0-101281 - Failed delivery
java.util.NoSuchElementException: Timeout waiting for idle object
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor
Elapsed (ms)
[route24 ] [route24 ]
[restlet:///cch/feedback?restletMethods=POST
] [ 3798]
] [ 0]
[route24 ] [marshal1 ]
] [ 0]
[route24 ] [route24 ]
[rabbitmq://vm1:5671/feedbackBufferExchange?connectionFactory=rabbitConnec]
[ 3782]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
java.util.NoSuchElementException: Timeout waiting for idle object
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
at
org.apache.camel.component.rabbitmq.RabbitMQProducer.execute(RabbitMQProducer.java:73)
at
org.apache.camel.component.rabbitmq.RabbitMQProducer.basicPublish(RabbitMQProducer.java:285)
at
org.apache.camel.component.rabbitmq.RabbitMQProducer.processInOnly(RabbitMQProducer.java:272)
at
org.apache.camel.component.rabbitmq.RabbitMQProducer.process(RabbitMQProducer.java:195)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
at
org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
at
org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
at
org.apache.camel.component.restlet.RestletConsumer$1.handle(RestletConsumer.java:68)
at
org.apache.camel.component.restlet.MethodBasedRouter.handle(MethodBasedRouter.java:58)
Thanks,
Valdis
Vhi Group DAC (Vhi) is a holding company for insurance and healthcare
services, which include Vhi Healthcare DAC, Vhi Insurance DAC, Vhi Health
Services DAC and Vhi Investments DAC. Vhi Healthcare DAC trading as Vhi
Healthcare and Vhi Insurance DAC trading as Vhi Insurance are regulated by
the Central Bank of Ireland. Vhi Healthcare is tied to Vhi Insurance DAC for
health insurance in Ireland which is underwritten by Vhi Insurance DAC. Vhi
Healthcare is tied to Zurich Life Assurance plc for Vhi Life Term Insurance
and Vhi Mortgage Protection which is underwritten by Zurich Life Assurance
plc. Vhi Healthcare is tied to Collinson Insurance Services Limited for
MultiTrip Travel Insurance, Backpacker Travel Insurance and Vhi Dental
Insurance which are underwritten by Great Lakes Insurance SE, UK branch and
for Vhi Canada Cover and Vhi International Health Insurance which are
underwritten by Astrenska Insurance Limited. For more information about the
Vhi Group please go to: https://www.vhi.ie/about-vhi.
Tá Vhi Group DAC (Vhi) ina chuideachta sealbhaíochta le haghaidh seirbhísí
árachais agus seirbhísí cúram sláinte, lena n-áirítear Vhi Healthcare DAC,
Vhi Insurance DAC, Vhi Health Services DAC agus Vhi Investments DAC. Déanann
Banc Ceannais na hÉireann rialáil ar Vhi Healthcare DAC, ag trádáil dó mar
Vhi Healthcare, agus ar Vhi Insurance DAC, ag trádáil dó mar Vhi Insurance.
Tá Vhi Healthcare ceangailte le Vhi Insurance DAC le haghaidh árachas
sláinte in Éirinn, rud atá frithgheallta ag Vhi Insurance DAC. Tá Vhi
Healthcare ceangailte le Zurich Life Assurance plc le haghaidh Árachais
Saoil de chuid Vhi agus Árachas Cosanta Morgáiste de chuid Vhi atá
frithgheallta ag Zurich Life Assurance plc. Tá Vhi Healthcare ceangailte le
Collinson Insurance Services Limited le haghaidh Árachas Taistil Ilturais
agus Turasóirí Mála Droma agus Árachas Fiaclóireachta de chuid Vhi atá
frithgheallta ag Great Lakes Insurance SE, UK branch agus le haghaidh
Clúdach Cheanada de chuid Vhi agus Árachas Sláinte Idirnáisiúnta de chuid
Vhi atá frithgheallta ag Astrenska Insurance Limited. Chun tuilleadh
https://www.vhi.ie/about-vhi.
This e-mail and any files transmitted with it contain information which may
be confidential and which may also be privileged and is intended solely for
the use of the individual or entity to whom it is addressed. Unless you are
the intended recipient you may not copy or use it, or disclose it to anyone
else. Any opinions expressed are that of the individual and not necessarily
that of the Vhi Group. If you have received this e-mail in error please
notify the sender by return.

[Valdis Andersons]

Thank you very much Zoran, I'll try it out with those parameters and see how it goes.


Valdis

-----Original Message-----
From: Zoran Regvart [mailto:***@regvart.com]
Sent: 05 October 2018 10:45
To: ***@camel.apache.org
Subject: Re: REST endpoint to rabbitmq queue connection timeout

Hi Valdis,
I think you're looking for `channelPoolMaxSize` and `channelPoolMaxWait` parameters[1]. From the snippet, I guess, you would set them on the `feedbackEndpoint`, and from the message history I see that you already set the `connectionFactory`, so you would configure these much in the same way.

zoran

[1] https://scanmail.trustwave.com/?c=6600&d=prK32zpd-TqvMeNzuqm1P9sl2_MVSbEt5EF_ffZc_A&s=33&u=https%3a%2f%2fgithub%2ecom%2fapache%2fcamel%2fblob%2fmaster%2fcomponents%2fcamel-rabbitmq%2fsrc%2fmain%2fdocs%2frabbitmq-component%2eadoc

Post by Valdis Andersons
Hi All,
Would anyone know how to configure the timeout values for the below
connection pool or the connection pool itself? Been having some issues
lately in a test environment with this and the timeout check seems to
happen every few milliseconds producing huge amounts of log files.
We’re suspecting it’s a network issue that’s causing this as there is
nothing much between the REST endpoint on the buffer queue and only if
rest(restEndpoint).post(postEndpoint)
.type(FeedbackRequest.class)
.route()
.process(this::process)
.startupOrder(Integer.MAX_VALUE - 2)
.marshal().json(JsonLibrary.Jackson, String.class)
.to(feedbackEndpoint)
.endRest();
exchange.getIn().setHeader("rabbitmq.DELIVERY_MODE", 2);
2018-10-04 17:01:43,275 [https-jsse-nio-8443-exec-65] ERROR
o.a.c.p.DefaultErrorHandler ID-VM1-1538665977944-0-101281 - Failed
java.util.NoSuchElementException: Timeout waiting for idle object
Message History
----------------------------------------------------------------------
-----------------------------------------------------------------
RouteId ProcessorId Processor
Elapsed (ms)
[route24 ] [route24 ]
[restlet:///cch/feedback?restletMethods=POST
] [ 3798]
] [ 0]
[route24 ] [marshal1 ]
] [ 0]
[route24 ] [route24 ]
[rabbitmq://vm1:5671/feedbackBufferExchange?connectionFactory=rabbitConnec]
[ 3782]
Stacktrace
----------------------------------------------------------------------
-----------------------------------------------------------------
java.util.NoSuchElementException: Timeout waiting for idle object
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
ectPool.java:1174)
at
org.apache.camel.component.rabbitmq.RabbitMQProducer.execute(RabbitMQP
roducer.java:73)
at
org.apache.camel.component.rabbitmq.RabbitMQProducer.basicPublish(Rabb
itMQProducer.java:285)
at
org.apache.camel.component.rabbitmq.RabbitMQProducer.processInOnly(Rab
bitMQProducer.java:272)
at
org.apache.camel.component.rabbitmq.RabbitMQProducer.process(RabbitMQP
roducer.java:195)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:14
8)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryEr
rorHandler.java:548)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInterna
lProcessor.java:201)
at
org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
at
org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInterna
lProcessor.java:201)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyn
cProcessor.java:97)
at
org.apache.camel.component.restlet.RestletConsumer$1.handle(RestletCon
sumer.java:68)
at
org.apache.camel.component.restlet.MethodBasedRouter.handle(MethodBase
dRouter.java:58)
Thanks,
Valdis
Vhi Group DAC (Vhi) is a holding company for insurance and healthcare
services, which include Vhi Healthcare DAC, Vhi Insurance DAC, Vhi
Health Services DAC and Vhi Investments DAC. Vhi Healthcare DAC
trading as Vhi Healthcare and Vhi Insurance DAC trading as Vhi
Insurance are regulated by the Central Bank of Ireland. Vhi Healthcare
is tied to Vhi Insurance DAC for health insurance in Ireland which is
underwritten by Vhi Insurance DAC. Vhi Healthcare is tied to Zurich
Life Assurance plc for Vhi Life Term Insurance and Vhi Mortgage
Protection which is underwritten by Zurich Life Assurance plc. Vhi
Healthcare is tied to Collinson Insurance Services Limited for
MultiTrip Travel Insurance, Backpacker Travel Insurance and Vhi Dental
Insurance which are underwritten by Great Lakes Insurance SE, UK
branch and for Vhi Canada Cover and Vhi International Health Insurance
which are underwritten by Astrenska Insurance Limited. For more
https://scanmail.trustwave.com/?c=6600&d=prK32zpd-TqvMeNzuqm1P9sl2_MVS
bEt5BB8L6UPrw&s=33&u=https%3a%2f%2fwww%2evhi%2eie%2fabout-vhi
Tá Vhi Group DAC (Vhi) ina chuideachta sealbhaíochta le haghaidh
seirbhísí árachais agus seirbhísí cúram sláinte, lena n-áirítear Vhi
Healthcare DAC, Vhi Insurance DAC, Vhi Health Services DAC agus Vhi
Investments DAC. Déanann Banc Ceannais na hÉireann rialáil ar Vhi
Healthcare DAC, ag trádáil dó mar Vhi Healthcare, agus ar Vhi Insurance DAC, ag trádáil dó mar Vhi Insurance.
Tá Vhi Healthcare ceangailte le Vhi Insurance DAC le haghaidh árachas
sláinte in Éirinn, rud atá frithgheallta ag Vhi Insurance DAC. Tá Vhi
Healthcare ceangailte le Zurich Life Assurance plc le haghaidh
Árachais Saoil de chuid Vhi agus Árachas Cosanta Morgáiste de chuid
Vhi atá frithgheallta ag Zurich Life Assurance plc. Tá Vhi Healthcare
ceangailte le Collinson Insurance Services Limited le haghaidh Árachas
Taistil Ilturais agus Turasóirí Mála Droma agus Árachas Fiaclóireachta
de chuid Vhi atá frithgheallta ag Great Lakes Insurance SE, UK branch
agus le haghaidh Clúdach Cheanada de chuid Vhi agus Árachas Sláinte
Idirnáisiúnta de chuid Vhi atá frithgheallta ag Astrenska Insurance
https://scanmail.trustwave.com/?c=6600&d=prK32zpd-TqvMeNzuqm1P9sl2_MVS
bEt5BB8L6UPrw&s=33&u=https%3a%2f%2fwww%2evhi%2eie%2fabout-vhi
This e-mail and any files transmitted with it contain information
which may be confidential and which may also be privileged and is
intended solely for the use of the individual or entity to whom it is
addressed. Unless you are the intended recipient you may not copy or
use it, or disclose it to anyone else. Any opinions expressed are that
of the individual and not necessarily that of the Vhi Group. If you
have received this e-mail in error please notify the sender by return.