Camel routes and threads

[vkarkhanis]

Hello,
As I learnt, each route in camel runs in its own thread unless its a direct
endpoint. We have a configuration, where we have 14 different routes inside
the camel context:

We are seeking clarification on following points:

1. the defaultThreadPolicy has poolSize="10" maxPoolSize="20". Does that
mean that maximum of 10 threads will be executing the 10 routes at a time
out of the 14 configured. If yes, can we override this configuration using
<threadPoolProfile> such that poolSize="5" maxPoolSize="5" do that there are
only 5 threads which execute 5 different routes at a given point in time ?
2. Is there any way by which we can limit the number of thread spawned to
executes routes ?

We are seeking the above information, because we are encountering out of
memory exception. The reason for this, as we see is, there is a customized
Publish-Subscribe mechanism and we have converted that to a customized
endpoint. This endpoint is used in 14 different routes and we get messages
on all of these endpoints which are processed. Since everything is in the
same camel context, we run out of memory during this parallel processing.

This is where we need to limit number of threads which process the routes in
parallel.

Appreciate any help.

Thank you,
Regards

vkarkhanis

[Claus Ibsen]

Hi

You can read about thread stuff here
http://camel.apache.org/threading-model.html

You can also share thread pools among the routes. Or reduce the sizes
of the thread pool profiles etc.

Also some Camel components have consumers that support parallel
processing such as seda / jms / jetty / cxf etc.
You can also configure their thread pool settings

Post by vkarkhanis
Hello,
As I learnt, each route in camel runs in its own thread unless its a direct
endpoint. We have a configuration, where we have 14 different routes inside
1. the defaultThreadPolicy has poolSize="10" maxPoolSize="20". Does that
mean that maximum of 10 threads will be executing the 10 routes at a time
out of the 14 configured. If yes, can we override this configuration using
<threadPoolProfile> such that poolSize="5" maxPoolSize="5" do that there are
only 5 threads which execute 5 different routes at a given point in time ?
2. Is there any way by which we can limit the number of thread spawned to
executes routes ?
We are seeking the above information, because we are encountering out of
memory exception. The reason for this, as we see is, there is a customized
Publish-Subscribe mechanism and we have converted that to a customized
endpoint. This endpoint is used in 14 different routes and we get messages
on all of these endpoints which are processed. Since everything is in the
same camel context, we run out of memory during this parallel processing.
This is where we need to limit number of threads which process the routes in
parallel.
Appreciate any help.
Thank you,
Regards
vkarkhanis
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-routes-and-threads-tp5734326.html
Sent from the Camel - Users mailing list archive at Nabble.com.

--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen-H+wXaHxf7aLQT0dZR+***@public.gmane.org
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

[vkarkhanis]

Hello Claus,
Thanks for the reply.

I have following configuration:

<route>
<from uri="quartz://"/>
<process ref="processor1">
<split executorServiceRef="customThreadPoolProfile">
<process ref="processor1">
<process ref="processor2">
</split>

</route>

In camel context:

<threadPoolProfile id="customThreadPoolProfile" poolSize="5" maxPoolSize="5"
maxQueueSize="1000" rejectedPolicy="CallerRuns"/>


In the above setup, when put parallelProcessing="true" in <split> tag, the
logs show that the customThreadPoolProfile is used, but the number of
threads are not restricted to 5. Instead around 20 threads are spawned. As I
can read from the documents, the parallelProcessing="true" is implicitly set
if "executorServiceRef" is set. But if I dont mention
parallelProcessing="true", the processing happens synchronously.

As per my uinderstading, the number of threads to execute this route, should
be limited to 5 with the customThreadPoolProfile. Is my understanding
incorrect?

Thank you,
Regards

Vaibhav A. Karkhanis

[Claus Ibsen]

Hi

What version of Camel do you use?

Post by vkarkhanis
Hello Claus,
Thanks for the reply.
<route>
<from uri="quartz://"/>
<process ref="processor1">
<split executorServiceRef="customThreadPoolProfile">
<process ref="processor1">
<process ref="processor2">
</split>
</route>
<threadPoolProfile id="customThreadPoolProfile" poolSize="5" maxPoolSize="5"
maxQueueSize="1000" rejectedPolicy="CallerRuns"/>
In the above setup, when put parallelProcessing="true" in <split> tag, the
logs show that the customThreadPoolProfile is used, but the number of
threads are not restricted to 5. Instead around 20 threads are spawned. As I
can read from the documents, the parallelProcessing="true" is implicitly set
if "executorServiceRef" is set. But if I dont mention
parallelProcessing="true", the processing happens synchronously.
As per my uinderstading, the number of threads to execute this route, should
be limited to 5 with the customThreadPoolProfile. Is my understanding
incorrect?
Thank you,
Regards
Vaibhav A. Karkhanis
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-routes-and-threads-tp5734326p5734731.html
Sent from the Camel - Users mailing list archive at Nabble.com.

--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen-H+wXaHxf7aLQT0dZR+***@public.gmane.org
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

[vkarkhanis]

Hello Claus,
We use camel version 2.9.0 ..

Thank you,
vkarkhanis

[Claus Ibsen]

I think we had a bug in one of these older releases with the thread
pool profiles not being picked up in the XML DSLs. You may try a newer
Camel version.

An alternative is to use a <threadPool> instead of a
<threadPoolProfile>, as that may still possible work in the 2.9.0
release in the XML DSL.

Post by vkarkhanis
Hello Claus,
We use camel version 2.9.0 ..
Thank you,
vkarkhanis
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-routes-and-threads-tp5734326p5734783.html
Sent from the Camel - Users mailing list archive at Nabble.com.

--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: cibsen-H+wXaHxf7aLQT0dZR+***@public.gmane.org
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

[vkarkhanis]

Hello Claus,
Thanks for the reply.
Unfortunately one other jar in the project conflicts with camel version
higher than 2.9.0, due to which we cannot upgrade. But let me try the
<threadPool> suggestion and get back with the results.

Thank you,
Regards

vkarkhanis

[vkarkhanis]

Hello Claus,
We tried using <threadPool>, but it did not work either..

Any suggestions?

Thank you
Regards

Vaibhav A. Karkhanis

[vkarkhanis]

Hello,
We have switched to camel 2.10.0 to resolve the issue. We realized that
camel version 2.9.0 has problems with the <split> functionality if
"parallelProcessing="true" is used. It does not take the customThreadPool
settings. Also the parallel processing tend to be operative only for one
instance of route. Subsequent route triggering are processed serially.
The higher versions of camel resolved the problem for us. Also by using
Custom Thread Pools we get much more control over the number of threads used
for processing.

Thank you,
Regards

vkarkhanis