Using Vector to retrieve Kafka messages

We have already mentioned installing Vector and migrating from Kafka to ClickHouse using Vector, but we can also get the Kafka messages into a file without using the Kafka binary. Of course, we assume that we are sure that the Kafka messages are created as JSON.

With this process, without installing the Kafka binary, we can make your servers behave as if kafka-console-consumer.sh existed.

Here is our Vector configuration:

[sources.kafka_messages]
type = "kafka"
auto_offset_reset = "earliest"
bootstrap_servers = "kafka:9092"
group_id = "vector"
topics = [ "test_topic" ]
decoding.codec = "json"


[sinks.clickhouse]
type = "file"
inputs = ["kafka_messages"]
path = "/tmp/output_file.txt"
encoding.charset = "UTF-8"
encoding.codec  = "json"

What do we run?

./vector --config /root/.vector/config/kafka_fetch.toml

Here is our output to /tmp/output_file.txt

{"column_1":3.0,"column_2":"abc.com.","headers":{},"message_key":null,"offset":19595045,"partition":5,"column_3":"pbd","sampleInterval":21600,"sampleRate":1,"source_type":"kafka","timestamp":"2024-03-19T19:58:48.767Z","topic":"test_topic"}

Today we have a great example of using Vector. You can quickly retrieve your data without installing the Kafka binary, and follow other uses of Vector on this blog.