v6.1 CE Release Notes

Created:2024-02-27 Last Modified:2024-07-05

This document was translated by ChatGPT

#1. v6.1.8.5 [2023/01/12] LTS

  • Bugfix

#2. v6.1.8.4 [2023/01/05]

#2.1 New Features

  • Management
    • Support specifying database name when using managed MySQL

#3. v6.1.8.3 [2022/12/29]

  • Bugfix

#4. v6.1.8.2 [2022/12/22]

#4.1 New Features

  • AutoMetrics, AutoTracing
    • Added support for SOFARPC protocol
  • Grafana
    • Support as Grafana Tempo data source, displaying Tracing data on Tempo page

#5. v6.1.8.1 [2022/12/15]

#5.1 New Features

#6. v6.1.8 [2022/12/09]

#6.1 New Features

  • AutoMetrics, AutoTracing
  • Management
    • Added signal_source tag field to data tables to indicate signal source, currently supporting three types: Packet (cBPF), eBPF, OTel
    • deepflow-agent supports running in K8s macvlan/ipvlan CNI environments, automatically calculating Pod access relationships and performance metrics. Detailed configuration for macvlan environment reference document
    • Support deepflow-agent running in K8s Pods without SYS_ADMIN permissions, detailed permission requirements reference document (opens new window)
    • Support configuring the maximum length (opens new window) of eBPF Socket Data collected by deepflow-agent to improve collection performance
    • Support configuring the resource information (opens new window) synchronized to the ingester module in deepflow-server to reduce resource consumption of the ingester module's AutoTagging function in multi-K8s cluster environments

#6.2 Optimization

  • Management
    • Reduced memory overhead of deepflow-agent when watching K8s apiserver through message compression and trimming
    • Optimized memory consumption of deepflow-server when calculating service tags through Pod aggregation
    • Optimized write pressure on flow_tag database in deepflow-server through memory pre-aggregation
    • Managed MySQL service supports connecting with non-root users

#7. v6.1.7 [2022/11/23]

#7.1 New Features

#7.2 Optimization

  • SQL API
    • When the auto-grouped resource type resource_glX_type is an IP address, reuse resource_glX_id to represent subnet ID and resource_glX to represent IP address
  • Management
    • Reduced database permission requirements when using managed RDS and ClickHouse
    • Automatically balance the number of deepflow-agents for each deepflow-server service
    • Use OpenTelemetry to monitor internal call chains of deepflow-server

#8. v6.1.6 [2022/11/09]

#8.1 New Features

  • AutoMetrics
    • Added attribute.http_user_agent and attribute.http_referer fields to HTTP protocol
    • Added attribute.rpc_service field, value is Dubbo/gRPC's ServiceName
    • Added endpoint field, value is Dubbo/gRPC's ServiceName/MethodName
    • Support marking HTTP2 data conforming to gRPC protocol specification as gRPC (instead of HTTP2) protocol
  • AutoTagging
    • Support synchronizing AWS public cloud resource information and automatically injecting it as tags into observability data
    • Support injecting both cloud resource and container tags into observability data for public cloud managed K8s clusters
  • Management
    • Support configuring whether deepflow-agent enables parsing of various application protocols
    • Support configuring deepflow-agent to collect data from Golang/openssl process names using eBPF uprobe regular expressions
    • Support deepflow-agent standalone mode, where Flow Log and Request Log are written to local log files
    • Support i18n, default display is English

#8.2 Optimization

  • AutoMetrics
    • Match SQL keywords to reduce false positives in MySQL and PostgreSQL protocol recognition
  • Grafana
    • Display SQL query statements in Panel's Query Editor to help developers understand how to call the API
    • Optimize the display of empty field information in Distributed Tracing flame graph
  • Management
    • Optimize traffic between deepflow-server and clickhouse, prioritizing writing to clickhouse Pod on the same node
    • Support compressing OTel Span data received by deepflow-agent, reducing bandwidth consumption by about 7 times when sending to deepflow-server

#9. v6.1.5 [2022/10/26]

#9.1 New Features

  • AutoMetrics, AutoTracing
    • Support collecting PostgreSQL performance metrics and access logs, and associating them with distributed tracing
    • Support collecting HTTPS performance metrics and access logs using openssl library, and associating them with distributed tracing
  • Integration
    • deepflow-server supports providing RemoteRead interface for Prometheus
    • deepflow-agent supports directly receiving OpenTelemetry data, bypassing otel-collector
    • Support using custom variables and built-in variables in Grafana Variable query statements, see documentation, usage scenarios include:
      • Use the current selection value of variable pod_cluster to filter the value range of variable pod
      • Use the input content of variable ingress_wildcard to change the value range of variable ingress
      • Use the current values of built-in variables $__from and $__to to speed up the query of variable value range
    • Added two zero-intrusion observability Dashboards in Grafana: K8s Ingress, SQL Monitoring
  • SQL API
    • string_enum and int_enum type tags support using Enum() function to translate Value to Name for query filtering and result return
    • Support SELECT tags/attributes/labels to query all tag.X/attribute.X/label.X fields for each row of data without specifying specific field names
  • Management
    • Support using disk for ClickHouse cold data (as an alternative to object storage)
    • Support using deepflow-ctl agent rebalance to balance deepflow-agent to newly added and restored deepflow-servers

#9.2 Optimization

  • AutoMetrics
  • AutoTagging
    • Added deepflow-ctl cloud info command to debug resource information synchronized from cloud platform API
  • SmartEncoding
    • Reuse tag encoding values of deleted resources in a loop to improve compression rate and query speed
  • SQL API
    • Optimized display_name of server_port tag field enumeration values to include corresponding int value to avoid ambiguity
  • Management
    • deepflow-server changed to be deployed using Deployment Controller, please update helm chart during upgrade
    • deepflow-agent supports running in non-privileged mode, specific permission requirements reference document (opens new window)
    • Optimized mapping relationship between Prometheus metrics and DeepFlow Table, each Metrics corresponds to one Table

#10. v6.1.4 [2022/10/12]

#10.1 New Features

  • AutoTagging
    • Support synchronizing Tencent public cloud resource information and automatically injecting it as tags into observability data
  • AutoTracing
    • Support associating application Span and network Span in environments where eBPF cannot run, eliminating tracing blind spots
  • SQL API
    • show tags added return of map type fields, such as labels, attributes, tags
    • show tag values added limit, offset, like parameters
  • Production Environment Deployment
    • Support using managed ClickHouse and MySQL
    • deepflow-agent supports configuration delivery by deepflow-server when registration is incomplete
  • Grafana
    • Added DeepFlow self-monitoring Dashboard

#10.2 Optimization

  • SQL API
    • Metric data in OTel Span supports returning through show metrics API
  • System Capabilities
    • Support completing deepflow-server master election without relying on sidecar
    • deepflow-server supports backward compatibility with deepflow-agent
    • Default synchronization with NTP server of the container node where deepflow-server is located
    • Standardized -v output of processes

#11. v6.1.3 [2022/09/23]

#11.1 Application

  • AutoMetrics
    • Added metrics: client wait latency, SYN packet count, SYN-ACK packet count, SYN retransmission packet count, SYN-ACK retransmission packet count
    • Support collecting Golang HTTP2, HTTP2_TLS calls using eBPF uprobe
    • Support collecting uprobe data of Golang processes with trimmed standard symbol tables (Golang >= 1.13 and < 1.18)
  • AutoTracing
    • Support associating eBPF uprobe Span with cBPF Span, OTel Span, displaying in tracing flame graph
  • AutoTagging
    • Automatically generate cloud server tags for K8s nodes not associated with cloud servers
    • Support synchronizing Huawei public cloud resource information
  • Querier SQL API
    • Automatically return fields after GROUP BY, no need to explicitly declare after SELECT
  • Grafana
    • Added thumbnail display for topology graph (DeepFlow Topo) and tracing flame graph (DeepFlow AppTracing)
    • Optimized Span Tip in tracing flame graph, displaying the proportion of time consumed by Span itself

#11.2 System

  • Reduced the time to wait for agent to come online during initial deployment from 7 minutes to 4 minutes
  • Access to deepflow-server and clickhouse within the same K8s cluster no longer uses NodeIP
  • deepflow-server defaults to using externalTrafficPolicy=Cluster to avoid kube-proxy's externalTrafficPolicy=Local being unavailable in some environments and compatibility issues with some CNIs. It can be manually modified to Local to optimize cross-cluster traffic
  • deepflow-server added ext-metrics-ttl, flow-metrics-ttl, flow-log-ttl configuration parameters for initializing data retention duration
  • deepflow-agent supports writing l4_flow_log and l7_flow_log to local files
  • deepflow-agent removed dependency on libbpf

#12. v6.1.2 [2022/09/08]

V6.1.2 is the second version after open-sourcing, officially starting a bi-weekly minor version release cycle.

#12.1 Application

  • AutoMetrics
    • Added parsing capabilities for MySQL COM_STMT_PREPARE, COM_STMT_EXECUTE, COM_STMT_FETCH, COM_STMT_CLOSE commands (opens new window)
    • Support collecting MQTT 3.1 call logs and performance metrics
  • SQL API
    • Support SELECT labels to get all custom label columns

#12.2 System

  • deepflow-agent
    • Added inactive IP metric data configuration item, after disabling this configuration, IPs without reply traffic will be aggregated
    • Use BPF pre-filtering collection interface to improve collection performance
    • Provide deb installation package
  • deepflow-ctl
    • agent-group-config update no longer requires specifying agent-group-id parameter

#13. v6.1.1 [2022/08/25]

V6.1.1 took three months to develop, marking a major iteration of the V6 series and the first community edition of DeepFlow. From now on, the enterprise and community editions will iterate simultaneously, moving to a bi-weekly minor version release cycle, with the final minor version of V6.1 (LTS version) released after approximately four minor versions.

To create a highly automated open-source observability platform, we made significant adjustments to the software architecture in this version, including merging excessive microservices, eliminating some components' dependencies on DaemonSet, HostNet deployment modes, and refactoring some Python modules with Golang.

Most importantly, we greatly enhanced application performance monitoring capabilities in this version:

  • Support integrating and associating OpenTelemetry, SkyWalking tracing data, unlocking full-stack distributed tracing capabilities, and improving communication efficiency between different development teams
  • Support integrating and associating Prometheus, Telegraf metrics data, consolidating system, application, and business full-stack metrics on one platform, enabling collaboration between operations, development, and operations teams
  • Enhanced AutoTagging and SmartEncoding capabilities, supporting the injection of more K8s custom labels
  • Enhanced eBPF AutoTracing capabilities, supporting the collection of Golang HTTPS calls

Note: The following Release Notes do not include enterprise edition features.

#13.1 Application Features

  • Enhanced Metrics Data
    • Support integrating Prometheus metrics data (implementing remote_write interface)
    • Support integrating Telegraf metrics data
    • Support automatically injecting cloud resources, container resources, K8s custom label tags into integrated metrics data (AutoTagging)
  • Enhanced Distributed Tracing
    • Support integrating OpenTelemetry tracing data
    • Support integrating SkyWalking tracing data
    • Support associating application, system, network Spans, achieving full-stack distributed tracing capabilities
    • Support automatically injecting cloud resources, container resources, K8s custom label tags into integrated tracing data (AutoTagging)
  • Enhanced Application Call Collection Capabilities
    • Support eBPF collection of Golang HTTPS call logs and performance metrics
    • Support eBPF/BPF collection of MQTT 3.1 call logs and performance metrics (Beta)
    • Removed port number restrictions for application protocol parsing
  • Enhanced Grafana Support
    • Added interactive DeepFlow Query Editor, a new zero-threshold query condition editor
    • Added DeepFlow AppTracing Panel for drawing distributed tracing flame graphs
    • Added DeepFlow Topo Panel for drawing call topology graphs
  • Improved Search Capabilities
    • Support querying historical data of deleted resources
    • Support searching all data using K8s custom labels
    • Support grouping all data based on K8s custom labels
    • Support creating template variables based on all searchable fields
  • Adjusted Flow Log Aggregation Logic
    • Flow logs only record TCP Flags that appeared in the current cycle
    • Delay timeout after collecting FIN-FIN-ACK sequence, waiting for the last ACK
    • Store VLAN fields in traffic
  • Added Math Operator to Metrics

#13.2 System Features

  • Provided deepflow-ctl command line tool for CLI management
  • Agent (Collector)
    • Support parsing multiple specifications of TraceID, SpanID in HTTP, Dubbo protocols simultaneously
    • Support collecting network and application performance data of VIP on loopback interface
    • Default configuration of collection network interfaces supports Calico, Flannel, Qemu, Cilium, Kube-OVN, localhost, physical network cards
    • Moved all non-blocking configuration items to be delivered by the controller, providing declarative group configuration API
    • Support remotely limiting CPU usage of deepflow-agent (Rust)
    • Support self-healing after failing to allocate controller or data node for the collector
  • Server (Controller, Data Node)
    • Support synchronizing K8s custom labels
    • Support delayed deletion of information of deleted resources
    • Provide unified Schema API and SQL query API for all data tables
    • Merged main modules of controller and data node into deepflow-server
    • Optimized deployment method of controller and data node, reducing dependency on DaemonSet, HostNet
    • Optimized master controller election logic, eliminating dependency on Zookeeper
    • Eliminated plaintext keys stored in the database