This document was translated by ChatGPT

1. v6.1.8.5 [2023/01/12] LTS

• Bugfix

2. v6.1.8.4 [2023/01/05]

2.1 New Features

• Management
  • Support specifying database name when using managed MySQL

3. v6.1.8.3 [2022/12/29]

• Bugfix

4. v6.1.8.2 [2022/12/22]

4.1 New Features

• AutoMetrics, AutoTracing
  • Added support for SOFARPC protocol
• Grafana
  • Support as Grafana Tempo data source to display Tracing data on Tempo page

5. v6.1.8.1 [2022/12/15]

5.1 New Features

• AutoMetrics
  • Pre-aggregate eBPF Request Log to supplement application performance metrics data for tap_side = client process/server process (opens new window)

6. v6.1.8 [2022/12/09]

6.1 New Features

• AutoMetrics, AutoTracing
  • Added support for ProtobufRPC protocol suite, currently supporting KRPC (opens new window) protocol
• Management
  • Added signal_source tag field to data tables to indicate signal source, currently supporting three types: Packet (cBPF), eBPF, OTel
  • deepflow-agent supports running in K8s macvlan/ipvlan CNI environments and automatically calculates Pod access relationships and performance metrics. Detailed configuration for macvlan environment reference document
  • Support deepflow-agent running in K8s Pods without SYS_ADMIN permissions, detailed permission requirements reference document (opens new window)
  • Support configuring the maximum length (opens new window) of eBPF Socket Data collected by deepflow-agent to improve collection performance
  • Support configuring the resource information (opens new window) synchronized to the ingester module in deepflow-server to reduce resource consumption of the ingester module's AutoTagging function in multi-K8s clusters

6.2 Optimization

• Management
  • Reduce memory overhead of deepflow-agent when watching K8s apiserver through message compression and trimming
  • Optimize memory consumption of deepflow-server when calculating service Tag through Pod aggregation
  • Optimize write pressure on flow_tag database in deepflow-server through memory pre-aggregation
  • Managed MySQL service supports connecting with non-root users

7. v6.1.7 [2022/11/23]

7.1 New Features

• Metrics
  • Mark content_length and other fields in OTel as metrics, detailed field mapping reference document (opens new window)
  • Added session_length metric to Request Log
• Tracing
  • Support parsing sw8 field in Dubbo protocol to extract TraceID, SpanID
• Event
  • Automatically generate add, delete, and change events for cloud servers and K8s Pods, and add Grafana Dashboard (opens new window)
• Management
  • Support configuring port ranges for application protocol parsing to reduce false positives, configuration method reference l7-protocol-ports configuration item (opens new window)
  • deepflow-agent supports using fixed load balancer IP address to request deepflow-server, configuration method reference proxy_controller_ip and analyzer_ip fields (opens new window)
  • Support specifying tap-mode (opens new window) in agent-group-config to allow deepflow-agent to run in special CNI environments like macvlan
  • Support configuring kubernetes-cluster-name (opens new window) in deepflow-agent.yaml to specify K8s cluster name

7.2 Optimization

• SQL API
  • When the auto-grouped resource type resource_glX_type is an IP address, directly reuse resource_glX_id to represent subnet ID and resource_glX to represent IP address
• Management
  • Lower database permission requirements when using managed RDS and ClickHouse
  • Automatically balance the number of deepflow-agents for each deepflow-server service
  • Use OpenTelemetry to monitor internal call chains of deepflow-server

8. v6.1.6 [2022/11/09]

8.1 New Features

• AutoMetrics
  • Added attribute.http_user_agent and attribute.http_referer fields for HTTP protocol
  • Added attribute.rpc_service field, value is Dubbo/gRPC's ServiceName
  • Added endpoint field, value is Dubbo/gRPC's ServiceName/MethodName
  • Support marking HTTP2 data conforming to gRPC protocol specification as gRPC (instead of HTTP2) protocol
• AutoTagging
  • Support synchronizing AWS public cloud resource information and automatically injecting it as tags into observability data
  • Support injecting both cloud resource and container tags into observability data for public cloud managed K8s clusters
• Management
  • Support configuring whether deepflow-agent enables parsing of various application protocols
  • Support configuring deepflow-agent to collect data from Golang/openssl processes using eBPF uprobe with regular expressions
  • Support deepflow-agent standalone mode, where Flow Log and Request Log are written to local log files
  • Support i18n, default display is English

8.2 Optimization

• AutoMetrics
  • Match SQL keywords to reduce false positives in MySQL and PostgreSQL protocol recognition
• Grafana
  • Display SQL query statements in Panel's Query Editor to help developers understand how to call the API
  • Optimize the display of empty field information in Distributed Tracing flame graph
• Management
  • Optimize traffic between deepflow-server and clickhouse, prioritize writing to clickhouse Pod on the same node
  • Support compressing OTel Span data received by deepflow-agent, reducing bandwidth consumption by about 7 times when sending to deepflow-server

9. v6.1.5 [2022/10/26]

9.1 New Features

• AutoMetrics, AutoTracing
  • Support collecting PostgreSQL performance metrics and access logs, and associating them with distributed tracing
  • Support collecting HTTPS performance metrics and access logs using openssl library, and associating them with distributed tracing
• Integration
  • deepflow-server supports providing RemoteRead interface for Prometheus
  • deepflow-agent supports directly receiving OpenTelemetry data, bypassing otel-collector
  • Support using custom variables and built-in variables in Grafana Variable query statements, see documentation, usage scenarios include:
    • Use the current selection value of variable pod_cluster to filter the value range of variable pod
    • Use the input content of variable ingress_wildcard to change the value range of variable ingress
    • Use built-in variables $__from and $__to to improve query speed of variable value range
  • Added two zero-intrusion observability Dashboards in Grafana: K8s Ingress, SQL Monitoring
• SQL API
  • string_enum and int_enum type Tags support using Enum() function to translate Value to Name for query filtering and result return
  • Support SELECT tags/attributes/labels to query all tag.X/attribute.X/label.X fields of each row without specifying specific field names
• Management
  • Support using disk for ClickHouse cold data (as an alternative to object storage)
  • Support using deepflow-ctl agent rebalance to balance deepflow-agent to newly added and restored deepflow-servers

9.2 Optimization

• AutoMetrics
  • Streamline application protocol parsing process to lower the threshold for adding support for more application protocols (opens new window)
• AutoTagging
  • Added deepflow-ctl cloud info command to debug resource information synchronized from cloud platform API
• SmartEncoding
  • Reuse Tag encoding values of deleted resources in a loop to improve compression rate and query speed
• SQL API
  • Optimize display_name of server_port Tag field enumeration values to include corresponding int value to avoid ambiguity
• Management
  • deepflow-server changed to be deployed using Deployment Controller, please update helm chart during upgrade
  • deepflow-agent supports running in non-privileged mode, specific permission requirements reference document (opens new window)
  • Optimize mapping relationship between Prometheus metrics and DeepFlow Table, each Metrics corresponds to one Table

10. v6.1.4 [2022/10/12]

10.1 New Features

• AutoTagging
  • Support synchronizing Tencent public cloud resource information and automatically injecting it as tags into observability data
• AutoTracing
  • Support associating application Span and network Span in environments where eBPF cannot run, eliminating tracing blind spots
• SQL API
  • show tags added return of map type fields, such as labels, attributes, tags
  • show tag values added limit, offset, like parameters
• Production Environment Deployment
  • Support using managed ClickHouse and MySQL
  • Support deepflow-server to issue configurations when deepflow-agent has not completed registration
• Grafana
  • Added DeepFlow self-monitoring Dashboard

10.2 Optimization

• SQL API
  • Metric data in OTel Span supports returning through show metrics API
• System Capabilities
  • Support completing deepflow-server master election without relying on sidecar
  • deepflow-server supports backward compatibility with deepflow-agent
  • Default synchronization with NTP server of the container node where deepflow-server is located
  • Standardize -v output of processes

11. v6.1.3 [2022/09/23]

11.1 Application

• AutoMetrics
  • Added metrics: client wait latency, SYN packet count, SYN-ACK packet count, SYN retransmission packet count, SYN-ACK retransmission packet count
  • Support collecting Golang HTTP2, HTTP2_TLS calls using eBPF uprobe
  • Support collecting uprobe data of Golang processes with trimmed standard symbol table (Golang >= 1.13 and < 1.18)
• AutoTracing
  • Support associating eBPF uprobe Span with cBPF Span, OTel Span, and displaying in tracing flame graph
• AutoTagging
  • Automatically generate cloud server tags for K8s nodes not associated with cloud servers
  • Support synchronizing Huawei public cloud resource information
• Querier SQL API
  • Automatically return fields after GROUP BY, no need to explicitly declare after SELECT
• Grafana
  • Added thumbnail display for topology graph (DeepFlow Topo) and tracing flame graph (DeepFlow AppTracing)
  • Optimize Span Tip in tracing flame graph to show the proportion of time consumed by Span itself

11.2 System

• Optimize the time to wait for agent to come online during initial deployment from 7 minutes to 4 minutes
• Access to deepflow-server and clickhouse within the same K8s cluster no longer uses NodeIP
• deepflow-server defaults to using externalTrafficPolicy=Cluster to avoid kube-proxy's externalTrafficPolicy=Local functionality being unavailable and some CNI compatibility issues in some environments, can be manually modified to Local to optimize cross-cluster traffic
• deepflow-server added ext-metrics-ttl, flow-metrics-ttl, flow-log-ttl configuration parameters to initialize data retention duration
• deepflow-agent supports writing l4_flow_log and l7_flow_log to local files
• deepflow-agent removed dependency on libbpf

12. v6.1.2 [2022/09/08]

V6.1.2 is the second version after open-sourcing, and we officially start the rhythm of a small version every two weeks.

12.1 Application

• AutoMetrics
  • Added parsing capabilities for MySQL COM_STMT_PREPARE, COM_STMT_EXECUTE, COM_STMT_FETCH, COM_STMT_CLOSE commands (opens new window)
  • Support collecting MQTT 3.1 call logs and performance metrics
• SQL API
  • Support SELECT labels to get all custom label columns

12.2 System

• deepflow-agent
  • Added inactive IP metrics data configuration item, after closing this configuration, IPs without reply traffic will be aggregated
  • Use BPF pre-filtering collection interface to improve collection performance
  • Provide deb installation package
• deepflow-ctl
  • agent-group-config update no longer requires specifying agent-group-id parameter

13. v6.1.1 [2022/08/25]

V6.1.1 took three months to develop, it is a major iterative version of the V6 series, and also the first community version officially released by DeepFlow. From now on, the enterprise version and the community version will iterate simultaneously, and the development rhythm will be upgraded to a small version every two weeks, and the last small version of V6.1 (LTS version) will be released after about four small versions.

To create a highly automated open-source observability platform, we made significant adjustments to the software architecture in this version, including merging too many microservices, eliminating some components' dependency on DaemonSet, HostNet, etc., and refactoring some Python modules with Golang.

Most importantly, in this version, we greatly enhanced application performance monitoring capabilities:

• Support integrating and associating OpenTelemetry, SkyWalking tracing data, unlocking full-stack distributed tracing capabilities, and improving communication efficiency between different development teams
• Support integrating and associating Prometheus, Telegraf metrics data, precipitating system, application, and business full-stack metrics on one platform, enabling collaboration between operations, development, and operations
• Enhanced AutoTagging and SmartEncoding capabilities, supporting injecting more K8s custom labels
• Enhanced eBPF AutoTracing capabilities, supporting collecting Golang HTTPS calls

Note: The following Release Note does not include enterprise version features.

13.1 Application Features

• Enhanced Metrics Data
  • Support integrating Prometheus metrics data (implement remote_write interface)
  • Support integrating Telegraf metrics data
  • Support automatically injecting cloud resources, container resources, K8s custom label tags (AutoTagging) into integrated metrics data
• Enhanced Distributed Tracing
  • Support integrating OpenTelemetry tracing data
  • Support integrating SkyWalking tracing data
  • Support associating application, system, and network Spans to achieve full-stack distributed tracing capabilities
  • Support automatically injecting cloud resources, container resources, K8s custom label tags (AutoTagging) into integrated tracing data
• Enhanced Application Call Collection Capabilities
  • Support eBPF to collect Golang HTTPS call logs and performance metrics
  • Support eBPF/BPF to collect MQTT 3.1 call logs and performance metrics (Beta)
  • Remove port number restrictions for application protocol parsing
• Enhanced Grafana Support
  • Added interactive DeepFlow Query Editor, a brand new zero-threshold query condition editor
  • Added DeepFlow AppTracing Panel for drawing distributed tracing flame graphs
  • Added DeepFlow Topo Panel for drawing call topology graphs
• Enhanced Search Capabilities
  • Support querying historical data of deleted resources
  • Support searching all data using K8s custom labels
  • Support grouping all data based on K8s custom labels
  • Support creating template variables based on all searchable fields
• Adjusted Flow Log Aggregation Logic
  • Flow logs only record TCP Flags that appeared in the current cycle
  • Delay timeout after collecting FIN-FIN-ACK sequence, waiting for the last ACK
  • Store VLAN field in traffic
• Added Math Operator for Metrics

13.2 System Features

• Provide deepflow-ctl command line tool for CLI management
• Agent (Collector)
  • Support parsing multiple specifications of TraceID, SpanID in HTTP, Dubbo protocols simultaneously
  • Support collecting network and application performance data of VIP on loopback interface
  • Default configuration of collection network interface supports Calico, Flannel, Qemu, Cilium, Kube-OVN, localhost, physical network card
  • Move all configuration items that do not hinder the startup of the collector to be issued by the controller, providing declarative group configuration API
  • Support remotely limiting CPU usage of deepflow-agent (Rust)
  • Support self-healing after failing to allocate controller or data node
• Server (Controller, Data Node)
  • Support synchronizing K8s custom labels
  • Support delayed deletion of information of deleted resources
  • Provide unified Schema API and SQL query API for all data tables
  • Merge main modules of controller and data node into deepflow-server
  • Optimize deployment method of controller and data node, reduce dependency on DaemonSet, HostNet
  • Optimize master controller election logic, remove dependency on Zookeeper
  • Eliminate plaintext keys stored in the database