Kafka Exporter
Created：2024-11-05 Last Modified：2024-11-05
This document was translated by ChatGPT
#1. Function
Through Kafka, DeepFlow can export generated metrics, flow logs, call logs, and IO events to external platforms.
#2. Introduction to Metrics
In DeepFlow, metrics can be divided into two types:
- Application performance metrics: See details here
- Corresponds to the
flow_metrics.application*tables in ClickHouse
- Corresponds to the
- Network performance metrics: See details here
- Corresponds to the
flow_metrics.network*tables in ClickHouse
- Corresponds to the
#3. Kafka Export
The protocol format uses JSON.
#4. DeepFlow Server Configuration Guide
Add the following configuration under the Server configuration to enable metrics export:
ingester:
exporters:
- protocol: kafka
enabled: true
endpoints: [broker1.example.com:9092, broker2.example.com:9092]
data-sources:
- flow_log.l7_flow_log
# - flow_log.l4_flow_log
# - flow_metrics.application_map.1s
# - flow_metrics.application_map.1m
# - flow_metrics.application.1s
# - flow_metrics.application.1m
# - flow_metrics.network_map.1s
# - flow_metrics.network_map.1m
# - flow_metrics.network.1s
# - flow_metrics.network.1m
# - event.perf_event
queue-count: 4
queue-size: 100000
batch-size: 1024
flush-timeout: 10
tag-filters:
export-fields:
- $tag
- $metrics
export-empty-tag: false
export-empty-metrics-disabled: false
enum-translate-to-name-disabled: false
universal-tag-translate-to-name-disabled: false
sasl:
enabled: false
security-protocol: SASL_SSL # currently only supports: SASL_SSL
sasl-mechanism: PLAIN # currently only supports: PLAIN
username: aaa
password: bbb
topic:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#5. Detailed Parameter Description
| Field | Type | Required | Description |
|---|---|---|---|
| protocol | string | Yes | Fixed value kafka |
| data-sources | strings | Yes | Values from ClickHouse flow_metrics.*/flow_log.*/event.perf_event data, also used as Kafka topic names |
| endpoints | strings | Yes | Remote receiving addresses, Kafka broker addresses; randomly selects one that can send successfully |
| batch-size | int | No | Batch size; when this value is reached, data is sent in batches. Default: 1024 |
| export-fields | strings | Yes | Recommended configuration: [$tag, $metrics] |
| sasl | struct | No | Kafka connection authentication method; currently only supports 'SASL_SSL' with 'PLAIN' |
| topic | string | No | Kafka topic name; if empty, defaults to deepflow.$data-source, e.g., deepflow.flow_log.l7_flow_log |