This document was translated by ChatGPT

1. v6.4.9 [2024/01/18]

1.1 New Features

• API
  • SQL API adds the Derivative pre-operator, which can calculate the difference for Prometheus Counter type metrics to compute the rate.
  • SQL API adds TopK and Any operators to obtain high-frequency or arbitrary values of specified tags.
• CLI
  • Supports debugging eBPF Socket Data using deepflow-ctl.
• Agent
  • The eBPF functionality of the Agent is adapted to the Linux 3.10 kernel (detailed documentation).

1.2 Optimizations

• Agent
  • Optimized HTTP2 parsing performance, reducing CPU usage by 60% (thanks to mickeyzzc, jiekun, someview: Issue (opens new window), PR (opens new window), PR (opens new window)).
• Integration
  • Offloaded PromQL operators to ClickHouse, improving PromQL query performance.

2. v6.4.8 [2024/01/11]

N/A

3. v6.4.7 [2024/01/04]

3.1 New Features

• AutoTagging
  • Enhanced the ability to parse compressed HTTP2 headers, supporting decompression and restoration of cBPF and eBPF kprobe data.

4. v6.4.6 [2023/12/28]

4.1 Optimizations

• AutoProfiling
  • When enabling the Profiling feature, there is no need to generate symbol table files in the Pod where the Java process resides.
• Server
  • Added trace_id_index integer column in ClickHouse as an index column for the trace_id field, supporting the extraction of Timestamps to accelerate TraceID searches.
• Wasm
  • Wasm Plugin supports dynamic loading in the Agent.
• CLI
  • Added list command: deepflow-ctl domain additional-resource list --type <resource_type> --name <resource_name>.

5. v6.4.5 [2023/12/21]

5.1 New Features

• AutoMetrics
  • MySQL, PostgreSQL, and Redis protocol data support obfuscation, which can be enabled through the Agent's obfuscate-enabled-protocols (thanks to eddietanshuo: FR (opens new window)).
• AutoTagging
  • Supports injecting custom auto-grouping tags into all observability data, allowing the combination of multiple tag fields to automatically take the first non-empty tag and ignore subsequent tag columns.
• Integration
  • Server supports exporting metrics via the Prometheus RemoteWrite protocol (thanks to chenjiandongx: PR (opens new window)).

5.2 Optimizations

• Server
  • Optimized K8s Label synchronization mechanism, supporting regular expression filtering for interested tags and limiting the maximum length of tag values.
  • Upgraded ClickHouse to v23.8 (LTS).

6. v6.4.4 [2023/12/07]

6.1 New Features

• AutoMetrics
  • Supports parsing Geneve tunnel encapsulation (adapted to Kube-OVN).
• AutoTagging
  • Supports precise setting of Pod tags for all eBPF observability data of HostNetwork Pods.
  • Added gprocess process information tags to continuous profiling data.
  • Supports synchronization of RDS and Redis resources from Huawei Cloud.
  • Supports synchronization of Redis resources from Baidu Cloud.
• AutoProfiling
  • Supports displaying a universal CPU flame graph of all processes on a server, down to the thread level.
  • Interleaves the generation of Java process symbol tables in a staggered manner to avoid high load caused by clustering, supporting the modification of the Agent's java-symbol-file-refresh-defer-interval configuration item to adjust the base interval of the delay.
• Server
  • Supports balancing the data sending Server based on the amount of data sent by the Agent, improving the data balance of ClickHouse (thanks to jiekun: FR (opens new window)).

6.2 Optimizations

• Server
  • Modified the logic of the Avg operator, where Avg represents the use of a weighted average algorithm, and AAvg represents the use of an arithmetic average algorithm.
• Agent
  • Reduced memory consumption by 60% in scenarios with a large number of new TCP flows (PR (opens new window), PR (opens new window), PR (opens new window), PR (opens new window)).
• Integration
  • Supports an extensible Exporter interface (thanks to jiekun: FR (opens new window), FR (opens new window)).

7. v6.4.3 [2023/11/23]

7.1 New Features

• AutoMetrics
  • Supports parsing all encrypted application protocols over TLS, not limited to HTTP (thanks to dirtyren: FR (opens new window)).
• AutoTagging
  • Supports synchronizing tag information of cloud servers from Tencent Cloud and Huawei Cloud, and supports synchronizing resource set information of cloud servers from Alibaba Cloud.

7.2 Optimizations

• AutoMetrics
  • Streamlined connection metrics in flow logs, removing redundant rtt_client_avg and rtt_server_avg.
  • Optimized the default parsing port for DNS traffic, adding 5353, see the Agent's l7-protocol-ports configuration item for details.
• AutoTagging
  • Deprecated the Others type in l7_protocol (application protocol), merging HTTP_TLS and HTTP2_TLS into HTTP and HTTP2, adding is_tls in call logs to indicate whether it is encrypted traffic.

8. v6.4.2 [2023/11/09]

8.1 New Features

• AutoTracing
  • Supports extracting trace_id from SofaRPC Payload (Hessian encoding, TreeMap structure).
  • Supports extracting TraceID fields injected by SkyWalking and OpenTelemetry in Kafka messages.
  • Supports parsing SkyWalking sw3 Header (thanks to Wadud-Ma: FR (opens new window), PR (opens new window)).
• AutoTagging
  • Supports extracting the topic_name field from Kafka messages and assigning it to the request_resource in call logs (thanks to mickeyzzc: FR (opens new window)).
  • Supports extracting endpoints from HTTP URLs and assigning them to call logs and application performance metrics data, supporting the configuration of http-endpoint-extraction extraction rules for the Agent (thanks to DomineCore: FR (opens new window)).

8.2 Optimizations

• AutoMetrics
  • Moved the implementation of the ProtobufRPC protocol from Rust code to the Wasm Plugin.
  • Used eBPF to obtain the Accept/Connect types of Sockets, correcting the direction of call logs and application performance metrics.
• Agent
  • Used the TCP protocol to transmit the Agent's own logs.
• Server
  • Merged self-monitoring metrics of Agent and Server into a single table deepflow_system.deepflow_system in ClickHouse.
• API
  • Supports deleting cloud platforms by name.

8.3 Community PRs

• Thanks to paman6415: Update README.md (opens new window).

9. v6.4.1 [2023/10/26]

9.1 New Features

• AutoMetrics
  • Supports parsing the MongoDB protocol (thanks to mickeyzzc: FR (opens new window), PR (opens new window)).
  • In flow logs and network performance metrics, system latency (srt, srt_max) supports recording the latency of ICMP traffic and correcting the recording direction of ICMP traffic using ICMP Echo messages.
• AutoTagging
  • Added pod_group_type (K8s workload type) tag field to all observability data, extending the values of auto_service_type to represent the type of K8s workload.
  • Added coroutine ID fields syscall_coroutine_0 and syscall_coroutine_1 to call logs.

9.2 Community PRs

• Thanks to zhangzujian: Server Dockerfile: upgrade alpine packages (opens new window).
• Thanks to zhangzujian: Server: bump go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin (opens new window).
• Thanks to zhangzujian: Agent: upgrade packages to avoid fixed CVEs (opens new window).
• Thanks to zhangzujian: Server: bump github.com/emicklei/go-restful and github.com/gin-gonic/gin (opens new window).
• Thanks to yunwei37: Agent: fix typo for socket (opens new window).

10. v6.4.0 [2023/10/12]

10.1 Optimizations

• Server
  • ClickHouse uses Array(LowCardinality(String)) instead of Array(String) to optimize the read and write performance of low cardinality fields, such as tag_names, metrics_names, etc.
  • Profiling data supports plaintext (non-compressed) storage in ClickHouse (thanks to tailerong: FR (opens new window)).
• CLI
  • Provides deepflow-ctl for MacOS (thanks to zjj1002: FR (opens new window)).

10.2 Community PRs

• Thanks to xiaoziv: Agent: remove redundant clone operation (opens new window).
• Thanks to xiaoziv: Agent: optimize match expression (opens new window).