Camel lock gets acquired on files but not picked sometime

[Hemant Pekhale]

Hi All,

I have a route which picks files from a folder and do some processing on it
and store it in other folder. I have used parameter "delete=true" for
deleting file from source folder.

Sometime it happens that files are stuck in the source folder and camel
lock files are also created for all the files but files are neither process
nor gets deleted.

There is a VB script which copied files in the source folder. When I
manually copied the files in the source folder then it never gives issue
but when the files are copied from VB script then only issue occurs. Also
the issue does not occur for the first file but it occurs randomly after
processing number of files.

This happens on server. I tried to reproduce this issue locally but no luck.

Does anybody have idea about it?

Thanks!

[Claus Ibsen]

Hi

Are you able to modify the VB script so it copies the file using a
temporary name and then rename the file when the copy is done.

It sounds a bit like the problem is that the VB script is currently
still copying the file, while Camel gets a lock on the file, and
attempt to process it.
File locks is not as easy and it depends a bit what file system, OS,
and how big the files are and other things.

A good strategy is to copy the files using temporary names or in
another folder, and then do a file rename when the copy is over, that
latter then is an atomic file operation which should not cause file
locking issues.

You can also try to use some of the other Camel file lock strategies
instead of marker files.
You can see more in the file documentation about these.

Post by Hemant Pekhale
Hi All,
I have a route which picks files from a folder and do some processing on it
and store it in other folder. I have used parameter "delete=true" for
deleting file from source folder.
Sometime it happens that files are stuck in the source folder and camel
lock files are also created for all the files but files are neither process
nor gets deleted.
There is a VB script which copied files in the source folder. When I
manually copied the files in the source folder then it never gives issue
but when the files are copied from VB script then only issue occurs. Also
the issue does not occur for the first file but it occurs randomly after
processing number of files.
This happens on server. I tried to reproduce this issue locally but no luck.
Does anybody have idea about it?
Thanks!

[Hemant Pekhale]

Hi Claus,

Thanks for the reply.

Yes, I am able to modify the script. But if we do copy the file with
temporary name and then rename, I doubt that there may be lock acquired on
temporary file name already. because my route has no waiting parameter. So
I think it might not be helpful for our case. Please correct me if I am
wrong.

Regarding the file lock strategies, I think readLock think
readLockCheckInterval think can be tried. Do you have any other suggestion?

FYI : Whatever I can try I need to try on Production server so just wanted
to tackle this with minimum and required trials.

Thanks!

Post by Claus Ibsen
Hi
Are you able to modify the VB script so it copies the file using a
temporary name and then rename the file when the copy is done.
It sounds a bit like the problem is that the VB script is currently
still copying the file, while Camel gets a lock on the file, and
attempt to process it.
File locks is not as easy and it depends a bit what file system, OS,
and how big the files are and other things.
A good strategy is to copy the files using temporary names or in
another folder, and then do a file rename when the copy is over, that
latter then is an atomic file operation which should not cause file
locking issues.
You can also try to use some of the other Camel file lock strategies
instead of marker files.
You can see more in the file documentation about these.

Post by Hemant Pekhale
Hi All,
I have a route which picks files from a folder and do some processing on

it

Post by Hemant Pekhale
and store it in other folder. I have used parameter "delete=true" for
deleting file from source folder.
Sometime it happens that files are stuck in the source folder and camel
lock files are also created for all the files but files are neither

process

Post by Hemant Pekhale
nor gets deleted.
There is a VB script which copied files in the source folder. When I
manually copied the files in the source folder then it never gives issue
but when the files are copied from VB script then only issue occurs. Also
the issue does not occur for the first file but it occurs randomly after
processing number of files.
This happens on server. I tried to reproduce this issue locally but no

luck.

Post by Hemant Pekhale
Does anybody have idea about it?
Thanks!

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

[Claus Ibsen]

Hi

You can configure Camel to only pickup files with a certain file
pattern, or to exclude names with temporary names, etc.
There is a number of different filter options, see its doc.

Post by Hemant Pekhale
Hi Claus,
Thanks for the reply.
Yes, I am able to modify the script. But if we do copy the file with
temporary name and then rename, I doubt that there may be lock acquired on
temporary file name already. because my route has no waiting parameter. So
I think it might not be helpful for our case. Please correct me if I am
wrong.
Regarding the file lock strategies, I think readLock think
readLockCheckInterval think can be tried. Do you have any other suggestion?
FYI : Whatever I can try I need to try on Production server so just wanted
to tackle this with minimum and required trials.
Thanks!

Post by Claus Ibsen
Hi
Are you able to modify the VB script so it copies the file using a
temporary name and then rename the file when the copy is done.
It sounds a bit like the problem is that the VB script is currently
still copying the file, while Camel gets a lock on the file, and
attempt to process it.
File locks is not as easy and it depends a bit what file system, OS,
and how big the files are and other things.
A good strategy is to copy the files using temporary names or in
another folder, and then do a file rename when the copy is over, that
latter then is an atomic file operation which should not cause file
locking issues.
You can also try to use some of the other Camel file lock strategies
instead of marker files.
You can see more in the file documentation about these.

Post by Hemant Pekhale
Hi All,
I have a route which picks files from a folder and do some processing on

it

Post by Hemant Pekhale
and store it in other folder. I have used parameter "delete=true" for
deleting file from source folder.
Sometime it happens that files are stuck in the source folder and camel
lock files are also created for all the files but files are neither

process

Post by Hemant Pekhale
nor gets deleted.
There is a VB script which copied files in the source folder. When I
manually copied the files in the source folder then it never gives issue
but when the files are copied from VB script then only issue occurs. Also
the issue does not occur for the first file but it occurs randomly after
processing number of files.
This happens on server. I tried to reproduce this issue locally but no

luck.

Post by Hemant Pekhale
Does anybody have idea about it?
Thanks!

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

[Claus Ibsen]

Also you can use "done" files, eg in the VB script you write a 2nd
empty file with a special name, that tells Camel that the other file
is done being copied and ready.
There is some options on Camel file endpoint you can configure for that also.

Post by Claus Ibsen
Hi
You can configure Camel to only pickup files with a certain file
pattern, or to exclude names with temporary names, etc.
There is a number of different filter options, see its doc.

Post by Hemant Pekhale
Hi Claus,
Thanks for the reply.
Yes, I am able to modify the script. But if we do copy the file with
temporary name and then rename, I doubt that there may be lock acquired on
temporary file name already. because my route has no waiting parameter. So
I think it might not be helpful for our case. Please correct me if I am
wrong.
Regarding the file lock strategies, I think readLock think
readLockCheckInterval think can be tried. Do you have any other suggestion?
FYI : Whatever I can try I need to try on Production server so just wanted
to tackle this with minimum and required trials.
Thanks!

Post by Claus Ibsen
Hi
Are you able to modify the VB script so it copies the file using a
temporary name and then rename the file when the copy is done.
It sounds a bit like the problem is that the VB script is currently
still copying the file, while Camel gets a lock on the file, and
attempt to process it.
File locks is not as easy and it depends a bit what file system, OS,
and how big the files are and other things.
A good strategy is to copy the files using temporary names or in
another folder, and then do a file rename when the copy is over, that
latter then is an atomic file operation which should not cause file
locking issues.
You can also try to use some of the other Camel file lock strategies
instead of marker files.
You can see more in the file documentation about these.

Post by Hemant Pekhale
Hi All,
I have a route which picks files from a folder and do some processing on

it

Post by Hemant Pekhale
and store it in other folder. I have used parameter "delete=true" for
deleting file from source folder.
Sometime it happens that files are stuck in the source folder and camel
lock files are also created for all the files but files are neither

process

Post by Hemant Pekhale
nor gets deleted.
There is a VB script which copied files in the source folder. When I
manually copied the files in the source folder then it never gives issue
but when the files are copied from VB script then only issue occurs. Also
the issue does not occur for the first file but it occurs randomly after
processing number of files.
This happens on server. I tried to reproduce this issue locally but no

luck.

Post by Hemant Pekhale
Does anybody have idea about it?
Thanks!

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

[Trilok Agarwal]

You can make the processes mutually exclusive through setting up a
scheduler by giving a cron expression so that your FTP or File component
would not read files when they are being written, just in case
if processing of your file not real time and copying of file via VB script
is also scheduled.

Otherwise readLock=changed would help, try it out-
From camel documentation-

*changed* is using file length/modification timestamp to detect whether the
file is currently being copied or not. Will at least use 1 sec. to
determine this, so this option cannot consume files as fast as the others,
but can be more reliable as the JDK IO API cannot always determine whether
a file is currently being used by another process. The option
*readLockCheckInterval* can be used to set the check frequency. This option
is *only* avail for the FTP <http://camel.apache.org/ftp2.html> component
from *Camel 2.8* onward. Note: from *Camel 2.10.1* onward the FTP
<http://camel.apache.org/ftp2.html> option *fastExistsCheck* can be enabled
to speedup this *readLock* strategy, if the FTP server support the LIST
operation with a full file name (some servers may not)

Post by Claus Ibsen
Also you can use "done" files, eg in the VB script you write a 2nd
empty file with a special name, that tells Camel that the other file
is done being copied and ready.
There is some options on Camel file endpoint you can configure for that also.

Post by Claus Ibsen
Hi
You can configure Camel to only pickup files with a certain file
pattern, or to exclude names with temporary names, etc.
There is a number of different filter options, see its doc.

Post by Hemant Pekhale
Hi Claus,
Thanks for the reply.
Yes, I am able to modify the script. But if we do copy the file with
temporary name and then rename, I doubt that there may be lock acquired

on

Post by Claus Ibsen

Post by Hemant Pekhale
temporary file name already. because my route has no waiting parameter.

So

Post by Claus Ibsen

Post by Hemant Pekhale
I think it might not be helpful for our case. Please correct me if I am
wrong.
Regarding the file lock strategies, I think readLock think
readLockCheckInterval think can be tried. Do you have any other

suggestion?

Post by Claus Ibsen

Post by Hemant Pekhale
FYI : Whatever I can try I need to try on Production server so just

wanted

Post by Claus Ibsen

Post by Hemant Pekhale
to tackle this with minimum and required trials.
Thanks!

Post by Claus Ibsen
Hi
Are you able to modify the VB script so it copies the file using a
temporary name and then rename the file when the copy is done.
It sounds a bit like the problem is that the VB script is currently
still copying the file, while Camel gets a lock on the file, and
attempt to process it.
File locks is not as easy and it depends a bit what file system, OS,
and how big the files are and other things.
A good strategy is to copy the files using temporary names or in
another folder, and then do a file rename when the copy is over, that
latter then is an atomic file operation which should not cause file
locking issues.
You can also try to use some of the other Camel file lock strategies
instead of marker files.
You can see more in the file documentation about these.
On Mon, Aug 6, 2018 at 3:19 PM, Hemant Pekhale <

Post by Hemant Pekhale
Hi All,
I have a route which picks files from a folder and do some

processing on

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen
it

Post by Hemant Pekhale
and store it in other folder. I have used parameter "delete=true" for
deleting file from source folder.
Sometime it happens that files are stuck in the source folder and

camel

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
lock files are also created for all the files but files are neither

process

Post by Hemant Pekhale
nor gets deleted.
There is a VB script which copied files in the source folder. When I
manually copied the files in the source folder then it never gives

issue

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
but when the files are copied from VB script then only issue occurs.

Also

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
the issue does not occur for the first file but it occurs randomly

after

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
processing number of files.
This happens on server. I tried to reproduce this issue locally but

no

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen
luck.

Post by Hemant Pekhale
Does anybody have idea about it?
Thanks!

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

[Hemant Pekhale]

Thanks for the replies guys.

Actually processing of files is realtime and there is one more process
other than vb script which creates files in source folder so not feasible
with done filename.
Need to try with file consumer options only.

Post by Trilok Agarwal
You can make the processes mutually exclusive through setting up a
scheduler by giving a cron expression so that your FTP or File component
would not read files when they are being written, just in case
if processing of your file not real time and copying of file via VB script
is also scheduled.
Otherwise readLock=changed would help, try it out-
From camel documentation-
*changed* is using file length/modification timestamp to detect whether the
file is currently being copied or not. Will at least use 1 sec. to
determine this, so this option cannot consume files as fast as the others,
but can be more reliable as the JDK IO API cannot always determine whether
a file is currently being used by another process. The option
*readLockCheckInterval* can be used to set the check frequency. This option
is *only* avail for the FTP <http://camel.apache.org/ftp2.html> component
from *Camel 2.8* onward. Note: from *Camel 2.10.1* onward the FTP
<http://camel.apache.org/ftp2.html> option *fastExistsCheck* can be enabled
to speedup this *readLock* strategy, if the FTP server support the LIST
operation with a full file name (some servers may not)

Post by Claus Ibsen
Also you can use "done" files, eg in the VB script you write a 2nd
empty file with a special name, that tells Camel that the other file
is done being copied and ready.
There is some options on Camel file endpoint you can configure for that also.

Post by Claus Ibsen
Hi
You can configure Camel to only pickup files with a certain file
pattern, or to exclude names with temporary names, etc.
There is a number of different filter options, see its doc.
On Mon, Aug 6, 2018 at 3:52 PM, Hemant Pekhale <

Post by Hemant Pekhale
Hi Claus,
Thanks for the reply.
Yes, I am able to modify the script. But if we do copy the file with
temporary name and then rename, I doubt that there may be lock

acquired

Post by Claus Ibsen
on

Post by Claus Ibsen

Post by Hemant Pekhale
temporary file name already. because my route has no waiting

parameter.

Post by Claus Ibsen
So

Post by Claus Ibsen

Post by Hemant Pekhale
I think it might not be helpful for our case. Please correct me if I

am

Post by Claus Ibsen

Post by Claus Ibsen

Post by Hemant Pekhale
wrong.
Regarding the file lock strategies, I think readLock think
readLockCheckInterval think can be tried. Do you have any other

suggestion?

Post by Claus Ibsen

Post by Hemant Pekhale
FYI : Whatever I can try I need to try on Production server so just

wanted

Post by Claus Ibsen

Post by Hemant Pekhale
to tackle this with minimum and required trials.
Thanks!

Post by Claus Ibsen
Hi
Are you able to modify the VB script so it copies the file using a
temporary name and then rename the file when the copy is done.
It sounds a bit like the problem is that the VB script is currently
still copying the file, while Camel gets a lock on the file, and
attempt to process it.
File locks is not as easy and it depends a bit what file system, OS,
and how big the files are and other things.
A good strategy is to copy the files using temporary names or in
another folder, and then do a file rename when the copy is over, that
latter then is an atomic file operation which should not cause file
locking issues.
You can also try to use some of the other Camel file lock strategies
instead of marker files.
You can see more in the file documentation about these.
On Mon, Aug 6, 2018 at 3:19 PM, Hemant Pekhale <

Post by Hemant Pekhale
Hi All,
I have a route which picks files from a folder and do some

processing on

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen
it

Post by Hemant Pekhale
and store it in other folder. I have used parameter "delete=true"

for

Post by Claus Ibsen

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
deleting file from source folder.
Sometime it happens that files are stuck in the source folder and

camel

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
lock files are also created for all the files but files are neither

process

Post by Hemant Pekhale
nor gets deleted.
There is a VB script which copied files in the source folder. When

I

Post by Claus Ibsen

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
manually copied the files in the source folder then it never gives

issue

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
but when the files are copied from VB script then only issue

occurs.

Post by Claus Ibsen
Also

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
the issue does not occur for the first file but it occurs randomly

after

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen

Post by Hemant Pekhale
processing number of files.
This happens on server. I tried to reproduce this issue locally but

no

Post by Claus Ibsen

Post by Hemant Pekhale

Post by Claus Ibsen
luck.

Post by Hemant Pekhale
Does anybody have idea about it?
Thanks!

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
-----------------
Camel in Action 2: https://www.manning.com/ibsen2