Kafka component Spring boot auto configuration

Discussion:

Yoshimo

2018-12-04 07:59:42 UTC

Hello Camel users,

I am trying to auto configure the Kafka component using spring boot, but the configuration is not working. More specifically I am following this page:

https://github.com/apache/camel/blob/master/components/camel-kafka/src/main/docs/kafka-component.adoc

and I am trying to set this configuration in my yaml file:
camel.component.kafka.configuration.value-deserializer

However the default deserializer is used when I consume from the Kafka topic. If I specify the deserializer in the Kafka route then it works as expected. Also I have defined the configuration camel.springboot.name in my yaml file and I can see that it is picked up normally. Are there any configurations that enable this functionality and I am missing?

Thank you,
Yiannis

Claus Ibsen

2018-12-05 10:15:05 UTC

Permalink

Hi

What version of Camel, Kafka and SB are you using?

Post by Yoshimo
Hello Camel users,
https://github.com/apache/camel/blob/master/components/camel-kafka/src/main/docs/kafka-component.adoc
camel.component.kafka.configuration.value-deserializer
However the default deserializer is used when I consume from the Kafka topic. If I specify the deserializer in the Kafka route then it works as expected. Also I have defined the configuration camel.springboot.name in my yaml file and I can see that it is picked up normally. Are there any configurations that enable this functionality and I am missing?
Thank you,
Yiannis

--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Yoshimo

2018-12-05 11:24:22 UTC

Permalink

Hello Claus,

thank you for looking into this.
Camel: 2.23.0
Kafka: 2.0.0
SB: 2.1.0.RELEASE

I am also attaching my pom if you want to have a look.

Thank you,
Yiannis

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Claus Ibsen
Hi
What version of Camel, Kafka and SB are you using?

--
Claus Ibsen
------------
Camel in Action 2: https://www.manning.com/ibsen2

Yoshimo

2018-12-05 11:56:20 UTC

Permalink

Here is what I add to my yaml file:

camel.component.kafka.configuration.value-deserializer: com.openbet.platform.pricepublisher.serde.SportsEventDeserializer

and here is my route:

from("kafka:prices?brokers={{kafka.servers}}")
.convertBodyTo(DBModel.class)
......

With this setup when I start the app I can see in the logs the Kafka configuration in the consumer:

value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

and I get exceptions while deserializing. However if I remove the config from the yaml file and update my route to be:

kafka:prices?brokers={{kafka.servers}}&valueDeserializer=com.openbet.platform.pricepublisher.serde.SportsEventDeserializer

then I see the following in the Kafka consumer config:

value.deserializer = class com.openbet.platform.pricepublisher.serde.SportsEventDeserializer

and the app works as expected. Just for reference the rest of my yaml file is:

kafka:
sportsTopic: sports
pricesTopic: prices
servers: localhost:25445
streams:
"[application.id]": price-publisher
"[bootstrap.servers]": ${kafka.servers}
"[default.key.serde]": com.openbet.platform.pricepublisher.serde.ActivityKeySerDe
"[default.value.serde]": com.openbet.platform.pricepublisher.serde.ActivityValueSerDe

Thank you,
Yiannis

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Hi
Can you post your yaml configuration of how you try to configure this
kafka value deserializer

Post by Yoshimo
Hello Claus,
thank you for looking into this.
Camel: 2.23.0
Kafka: 2.0.0
SB: 2.1.0.RELEASE
I am also attaching my pom if you want to have a look.
Thank you,
Yiannis
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Claus Ibsen
Hi
What version of Camel, Kafka and SB are you using?

--
Claus Ibsen
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
------------
Camel in Action 2: https://www.manning.com/ibsen2

Claus Ibsen

2018-12-05 12:17:02 UTC

Permalink

Post by Yoshimo
camel.component.kafka.configuration.value-deserializer: com.openbet.platform.pricepublisher.serde.SportsEventDeserializer
from("kafka:prices?brokers={{kafka.servers}}")
.convertBodyTo(DBModel.class)
......
value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

Try with

value.deserializer = org.apache.kafka.common.serialization.StringDeserializer

Post by Yoshimo
kafka:prices?brokers={{kafka.servers}}&valueDeserializer=com.openbet.platform.pricepublisher.serde.SportsEventDeserializer
value.deserializer = class com.openbet.platform.pricepublisher.serde.SportsEventDeserializer
sportsTopic: sports
pricesTopic: prices
servers: localhost:25445
"[application.id]": price-publisher
"[bootstrap.servers]": ${kafka.servers}
"[default.key.serde]": com.openbet.platform.pricepublisher.serde.ActivityKeySerDe
"[default.value.serde]": com.openbet.platform.pricepublisher.serde.ActivityValueSerDe
Thank you,
Yiannis
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Hi
Can you post your yaml configuration of how you try to configure this
kafka value deserializer

Post by Claus Ibsen
Hi
What version of Camel, Kafka and SB are you using?

--
Claus Ibsen
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
------------
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Yoshimo

2018-12-05 12:33:05 UTC

Permalink

Hey Claus,

I think that you misread my previous email. The value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer line is the logging output of the Kafka consumer being created by Camel when I try adding the Spring Boot configuration.

What I actually add to my yaml file is the following:
camel.component.kafka.configuration.value-deserializer: com.openbet.platform.pricepublisher.serde.SportsEventDeserializer

The above is being ignored and the Kafka consumer that gets created by Camel has the default deserialer.

Thank you

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Claus Ibsen

Try with
value.deserializer = org.apache.kafka.common.serialization.StringDeserializer

Hi
Can you post your yaml configuration of how you try to configure this
kafka value deserializer

Post by Claus Ibsen
Hi
What version of Camel, Kafka and SB are you using?

--
Claus Ibsen
Camel in Action 2: https://www.manning.com/ibsen2

--
Claus Ibsen
------------
Camel in Action 2: https://www.manning.com/ibsen2

Yoshimo

2018-12-06 08:53:46 UTC

Permalink

I have created a demo project which demonstrates the issue. The issue is in application.properties file and in TestRoute class.

https://drive.google.com/file/d/1PDZj5tNXqqr0EjB07wluLiwy4ACK1eiP/view?usp=sharing

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

I have attached a demo project which demonstrates the issue. The issue is in application.properties file and in TestRoute class.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Yoshimo
Hey Claus,
I think that you misread my previous email. The value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer line is the logging output of the Kafka consumer being created by Camel when I try adding the Spring Boot configuration.
camel.component.kafka.configuration.value-deserializer: com.openbet.platform.pricepublisher.serde.SportsEventDeserializer
The above is being ignored and the Kafka consumer that gets created by Camel has the default deserialer.
Thank you
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Claus Ibsen

Try with
value.deserializer = org.apache.kafka.common.serialization.StringDeserializer

Hi
Can you post your yaml configuration of how you try to configure this
kafka value deserializer

Post by Claus Ibsen
Hi
What version of Camel, Kafka and SB are you using?

--
Claus Ibsen
Camel in Action 2: https://www.manning.com/ibsen2

Luca Burgazzoli

2018-12-06 09:03:34 UTC

Permalink

I think you only need to add camel-kafka-starter as a dependency to
make it working.
Every component has a related starter that does the spring-boot magic

---
Luca Burgazzoli

Post by Yoshimo
I have created a demo project which demonstrates the issue. The issue is in application.properties file and in TestRoute class.
https://drive.google.com/file/d/1PDZj5tNXqqr0EjB07wluLiwy4ACK1eiP/view?usp=sharing
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

I have attached a demo project which demonstrates the issue. The issue is in application.properties file and in TestRoute class.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Claus Ibsen

Try with
value.deserializer = org.apache.kafka.common.serialization.StringDeserializer

Hi
Can you post your yaml configuration of how you try to configure this
kafka value deserializer

Post by Claus Ibsen
Hi
What version of Camel, Kafka and SB are you using?

--
Claus Ibsen
Camel in Action 2: https://www.manning.com/ibsen2

Yoshimo

2018-12-06 09:07:50 UTC

Permalink

Indeed, I added it and it works! Thank you very much!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Luca Burgazzoli
I think you only need to add camel-kafka-starter as a dependency to
make it working.
Every component has a related starter that does the spring-boot magic
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Luca Burgazzoli

I have attached a demo project which demonstrates the issue. The issue is in application.properties file and in TestRoute class.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Claus Ibsen

Try with
value.deserializer = org.apache.kafka.common.serialization.StringDeserializer

Hi
Can you post your yaml configuration of how you try to configure this
kafka value deserializer

Post by Claus Ibsen
Hi
What version of Camel, Kafka and SB are you using?

--
Claus Ibsen
Camel in Action 2: https://www.manning.com/ibsen2