OpenTelemetry Exporter

Created:2023-10-04 Last Modified:2024-06-28

This document was translated by ChatGPT

#1. Features

By converting the standard OTLP protocol, DeepFlow's Span data can be delivered to external platforms, allowing external teams to supplement and enhance their own observability platforms.

#2. Span Overview

In DeepFlow, Spans can be categorized as:

  • Application Span: Application-level Span data generated using process-level Trace frameworks (Agent/SDK), including custom application Spans, middleware client embedded Spans, communication frameworks, etc. The Trace frameworks here include but are not limited to: Apache SkyWalking Agent, OpenTelemetry Java Agent, and others.
  • System Span: Spans collected by DeepFlow through eBPF with zero intrusion, covering system calls, application functions (such as HTTPS), API Gateway, and service mesh Sidecar.
  • Network Span: Spans collected by DeepFlow from network traffic using BPF, covering container network components like iptables/ipvs/OvS/LinuxBridge.

#3. OTel Related

You can find OTLP Proto (opens new window) here, and the Trace Semantic Conventions (opens new window) here. The internal Resource Semantic Conventions (opens new window) can be found here.

#4. Configuration

ingester:
  exporters:
    - protocol: opentelemetry
      enabled: true
      endpoints: [127.0.0.1:4317] # only support protocol grpc
      data-sources:
        - flow_log.l7_flow_log # only support 'flow_log.l7_flow_log'
      queue-count: 4
      queue-size: 100000
      batch-size: 32
      flush-timeout: 10
      tag-filters:
      export-fields:
        - $tag
        - $metrics
        - $k8s.label
      extra-headers:
        key1: value1
        key2: value2
      export-empty-tag: false
      export-empty-metrics_disabled: false
      enum-translate-to-name-disabled: false
      universal-tag-translate-to-name-disabled: false
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#5. Detailed Parameter Description

Field Type Required Description
protocol string Yes Fixed value opentelemetry
data-sources strings Yes Only supports flow_log.l7_flow_log
endpoints strings Yes Remote receiving address, only supports gRPC protocol, randomly selects one that can send successfully
batch-size int No Batch size, sends in batches when this value is reached. Default: 32
extra-headers map No Header fields for remote gRPC requests, such as tokens for authentication
export-fields strings Yes Recommended configuration: [$tag, $metrics, $k8s.label]

Detailed Configuration Reference

#6. General Field Mapping

In Flow_log, there is an internal logic that categorizes all data hierarchically. The mapping here refers to converting the hierarchical Flow_Log general fields to standard OTel format data.

#6.1.1 Tracing Info

Data related to OTel Spans remains unchanged, while other fields are included in span.attributes.

Original Field Name Mapped Location Mapped Name Remarks
x_request_id span.attributes df.span.x_request_id
syscall_trace_id_request span.attributes df.span.syscall_trace_id_request
syscall_trace_id_response span.attributes df.span.syscall_thread_0
syscall_thread_0 span.attributes df.span.syscall_thread_0
syscall_thread_1 span.attributes df.span.syscall_thread_1
syscall_cap_seq_0 span.attributes df.span.syscall_cap_seq_0
syscall_cap_seq_1 span.attributes df.span.syscall_cap_seq_1

#6.1.2 Service Info

Service application-level information, all included in resource.attributes. This includes application-related, process, and thread-related information. For special requirements regarding process and thread-related information, please use OTel Processor for conversion.

Original Field Name Mapped Location Mapped Name Remarks
auto_service resource.attributes service.name Standard field
auto_instance resource.attributes service.instance.id Standard
process_id resource.attributes process.pid
process_kname resource.attributes thread.name

#6.1.3 Flow Info

Includes fields: _id, time, flow_id, start_time, end_time, close_type, status, is_new_flow.

Original Field Name Mapped Location Mapped Name Remarks
_id resource.attributes df.flow_info.id
time resource.attributes df.flow_info.time
flow_id resource.attributes df.flow_info.flow_id
start_time span.start_time_unix_nano span.start_time_unix_nano Note time format conversion to OTel format
end_time span.end_time_unix_nano span.end_time_unix_nano Note time format conversion to OTel format

#6.1.4 Capture Info

Includes fields: signal_source, agent, nat_source, capture_nic, capture_nic_name, capture_nic_type, observation_point, l2_end, l3_end, has_pcap, nat_real_ip, nat_real_port

Original Field Name Mapped Location Mapped Name Remarks
signal_source resource.attributes df.capture_info.signal_source
agent resource.attributes df.capture_info.agent
nat_source resource.attributes df.capture_info.nat_source
capture_nic resource.attributes df.capture_info.capture_nic
capture_nic_name resource.attributes df.capture_info.capture_nic_name
capture_nic_type resource.attributes df.capture_info.capture_nic_type
observation_point resource.attributes df.capture_info.observation_point

#6.1.5 Universal Tag

Includes fields: region, az, host, chost, vpc, l2_vpc, subnet, router, dhcpgw, lb, lb_listener, natgw, pod_cluster, pod_ns, pod_node, pod_ingress, pod_service, pod_group, pod, service, auto_service, auto_service_type, auto_instance, auto_instance_type

For necessary conversions, please use OTel Processor.

Original Field Name Mapped Location Mapped Name Remarks
region resource.attributes df.universal_tag.region
az resource.attributes df.universal_tag.az
host resource.attributes df.universal_tag.host
chost resource.attributes df.universal_tag.chost
vpc resource.attributes df.universal_tag.vpc
l2_vpc resource.attributes df.universal_tag.l2_vpc
subnet resource.attributes df.universal_tag.subnet
router resource.attributes df.universal_tag.router
dhcpgw resource.attributes df.universal_tag.dhcpgw
lb resource.attributes df.universal_tag.lb
lb_listener resource.attributes df.universal_tag.lb_listener
natgw resource.attributes df.universal_tag.natgw
pod_cluster resource.attributes df.universal_tag.pod_cluster According to official documentation, this should be k8s.pod_cluster, convert if necessary
pod_ns resource.attributes df.universal_tag.pod_ns According to official documentation, this should be k8s.pod_ns, convert if necessary
pod_node resource.attributes df.universal_tag.pod_node According to official documentation, this should be k8s.pod_node, convert if necessary
pod_ingress resource.attributes df.universal_tag.pod_ingress According to official documentation, this should be k8s.pod_ingress, convert if necessary
pod_service resource.attributes df.universal_tag.pod_service According to official documentation, this should be k8s.pod_service, convert if necessary
pod_group resource.attributes df.universal_tag.pod_group According to official documentation, this should be k8s.pod_group, convert if necessary
pod resource.attributes df.universal_tag.pod According to official documentation, this should be k8s.pod.xxx, convert if necessary
pod_cluster resource.attributes df.universal_tag.pod_cluster According to official documentation, this should be k8s.pod_cluster, convert if necessary
service resource.attributes df.universal_tag.service
auto_service resource.attributes df.universal_tag.auto_service
auto_service_type resource.attributes df.universal_tag.auto_service_type
auto_instance resource.attributes df.universal_tag.auto_instance
auto_instance_type resource.attributes df.universal_tag.auto_instance_type

#6.1.6 Custom Tag

Original Field Name Mapped Location Mapped Name Remarks
k8s.labels.xxx resource.attributes df.custom_tag.k8s.labels.xxx

#6.1.7 Network Layer

Original Field Name Mapped Location Mapped Name Remarks
ip resource.attributes df.network.ip
is_ipv4 resource.attributes df.network.is_ipv4
is_internet resource.attributes df.network.is_internet
protocol resource.attributes net.transport = ip_(lowercase ${protocol}) Standard field

#6.1.8 Transport Layer

Original Field Name Mapped Location Mapped Name Remarks
client_port resource.attributes df.transport.client_port
server_port resource.attributes df.transport.server_port
tcp_flags_bit resource.attributes df.transport.tcp_flags_bit
syn_seq resource.attributes df.transport.syn_seq
syn_ack_seq resource.attributes df.transport.syn_ack_seq
last_keepalive_seq resource.attributes df.transport.last_keepalive_seq
last_keepalive_ack resource.attributes df.transport.last_keepalive_ack
req_tcp_seq resource.attributes df.transport.req_tcp_seq
resp_tcp_seq resource.attributes df.transport.resp_tcp_seq

#6.1.9 Application Layer

Original Field Name Mapped Location Mapped Name Remarks
l7_protocol resource.attributes df.application.l7_protocol Field mapping details

#7. Protocol Field Mapping

Here, special mappings of protocol-specific fields to OTLP standard fields are supplemented (for general fields, refer to the above):

#7.1 Application Protocol Additional Fields

The following fields apply to all application layer protocols:

Original Field Name Mapped Location Mapped Name Remarks
None resource.attributes telemetry.sdk.name=deepflow Custom
None resource.attributes telemetry.sdk.version=${current version} Custom
chost_0/pod_node_0 span.attributes net.host.name Standard, first get chost_x, if not present, try pod_node_x
chost_1/pod_node_1 span.attributes net.peer.name Standard, first get chost_x, if not present, try pod_node_x
client_port span.attributes net.host.port Standard
server_port span.attributes net.peer.port Standard
ip_0 span.attributes net.sock.host.addr Standard
ip_1 span.attributes net.sock.peer.addr Standard
response_status span.status span.status 0:OK -> Ok; 1:Server error, Client error -> Error; Not present -> Unset

#7.2 HTTP Protocol Suite

#7.2.1 HTTP

Original Field Name Mapped Location Mapped Name Remarks
version span.attributes http.flavor Standard field
request_type span.attributes http.method Standard field
request_domain span.attributes net.peer.name Standard field
request_resource span.attributes df.http.path Custom
request_id span.attributes df.global.request_id Custom
response_code span.attributes http.status_code Standard field
response_exception span.event event.name Standard field
http_proxy_client span.attributes df.http.proxy_client Custom
None span.name span.name= ${request_type} + ${request_source} Standard field, space in between

#7.2.2 HTTP2

TODO

#7.3 RPC Protocol Suite

#7.3.1 Dubbo

Original Field Name Mapped Location Mapped Name Remarks
None span.attributes rpc.system=apache_dubbo Standard field
None span.attributes rpc.service=${request_resource} Standard field
None span.attributes rpc.method=${request_type} Standard field
None span.attributes span.name= ${request_source} + "/" + ${request_type} == ${endpoint} Standard field, prioritize concatenation
response_exception span.event event.name Standard field
request_domain span.attributes df.dubbo.request_domain If not available, use net.peer.name as an additional field
version span.attributes df.dubbo.version Custom
request_id span.attributes df.global.request_id Custom
response_code span.attributes df.response_code Custom

#7.3.2 gRPC

Original Field Name Mapped Location Mapped Name Remarks
None span.attributes rpc.system=grpc Standard field
None span.attributes rpc.system=${request_resource} Standard field
None span.attributes rpc.system=${request_type} Standard field
None span.attributes span.name= ${request_source} + "/" + ${request_type} == ${endpoint} Standard field
response_exception span.event event.name Standard field
version span.attributes http.flavor Standard field
request_domain span.attributes df.grpc.request_domain If not available, use net.peer.name as an additional field
request_id span.attributes df.global.request_id Custom

#7.3.3 SOFARPC

TODO

#7.3.4 FastCGI

TODO

#7.4 SQL Protocol Suite

#7.4.1 MySQL

Original Field Name Mapped Location Mapped Name Remarks
None span.attributes db.system==mysql Standard
None span.attributes db.operation=${C/R/U/D} Standard field
None span.attributes db.statement=${request_resource} Standard field
request_type span.attributes df.mysql.request_type Custom: db.operation defined as SQL keyword
response_exception span.event event.name Standard field
None span.name span.name=${C/R/U/D} + ${db} + ${table} Standard field

#7.4.2 PostgreSQL

Original Field Name Mapped Location Mapped Name Remarks
None span.attributes db.system==postgresql Standard
None span.attributes db.operation=${C/R/U/D} Standard field
None span.attributes db.statement=${request_resource} Standard field
request_type span.attributes df.postgresql.request_type Custom: db.operation defined as SQL keyword
response_exception span.event event.name Standard field
None span.name span.name=${C/R/U/D} + ${db} + ${table} Standard field

#7.5 NoSQL Protocol Suite

#7.5.1 Redis

Original Field Name Mapped Location Mapped Name Remarks
None span.attributes db.system==redis Custom
None span.attributes db.operation=${request_type} Standard
None span.attributes db.statement=${request_resource} Standard
response_exception span.event event.name Standard
None span.name span.name=${request_type} Standard

#7.5.2 MongoDB

Original Field Name Mapped Location Mapped Name Remarks
None span.attributes db.system==mongodb Custom
None span.attributes db.operation=${request_type} Standard
None span.attributes db.statement=${request_resource} Standard
response_exception span.event event.name Standard
None span.name span.name=${request_type} Standard

#7.6 Messaging Protocols

#7.6.1 Kafka

Original Field Name Mapped Location Mapped Name Remarks
None span.attributes messaging.system=kafka Standard
None span.name span.name=${request_type} Non-standard
request_type span.attributes df.kafka.request_type Custom
request_id span.attributes df.global.request_id Custom
request_resource span.attributes df.global.request_resource Custom
request_domain span.attributes df.kafka.request_domain Custom
response_code span.attributes df.kafka.response_code Custom
response_exception span.event event.name Standard

#7.6.2 MQTT

Original Field Name Mapped Location Mapped Name Remarks
None span.attributes messaging.system=mqtt Standard
None span.attributes messaging.operation=${request_type} Standard: PUBLISH -> publish, SUBSCRIBE -> process, others discarded
None span.name span.name=${request_resource} + " " + ${messaging.operation} Standard
request_type span.attributes df.mqtt.request_type Custom
request_resource span.attributes df.mqtt.request_resource Custom
request_domain span.attributes df.mqtt.request_domain Custom
response_code span.attributes df.mqtt.response_code Custom
response_exception span.event event.name Standard

#7.7 Network Protocols

#7.7.1 DNS

Original Field Name Mapped Location Mapped Name Remarks
request_type span.attributes df.dns.request_type Custom
request_resource span.attributes df.dns.request_resource Custom
request_id span.attributes df.global.request_id Custom
response_code span.attributes df.dns.response_code Custom
response_exception span.event event.name Standard
response_result span.attributes df.dns.response_result Custom