Network Flow Log

Created:2024-06-01 Last Modified:2024-06-24

This document was translated by ChatGPT

Without inserting any code into the application, DeepFlow automatically generates network flow logs for all services. Database table name: flow_log.l4_flow_log.

#1. Tags

List of automatically injected tags: IP, protocol, port, network header fields, collection location, cloud resources, K8s resources, K8s custom labels. Detailed field descriptions are as follows.

Name DisplayName Description
_id UID
time Time Round end_time to seconds.
region Region
az Availability Zone
host VM Hypervisor Host running virtual machine.
chost Cloud Host Including virtual machines
vpc VPC
l2_vpc Forwarding VPC VPC where the MAC address is located.
subnet Subnet
router Router
dhcpgw DHCP Gateway
lb Load Balancer
lb_listener Load Balancer Listener
natgw NAT Gateway
redis Redis
rds RDS
pod_cluster K8s Cluster
pod_ns K8s Namespace
pod_node K8s Node
pod_ingress K8s Ingress
pod_service K8s Service
pod_group_type K8s Workload Type
pod_group K8s Workload Such as Deployment
pod K8s POD
service Service
resource_gl0_type Auto Instance Type Deprecated,please use auto_instance_type.
resource_gl0 Auto Instance Tag Deprecated,please use auto_instance.
resource_gl1_type Type - K8s Workload First Deprecated,please use auto_service_type.
resource_gl1 Instance - K8s Workload First Deprecated,please use auto_service.
resource_gl2_type Auto Service Type Deprecated,please use auto_service_type.
resource_gl2 Auto Service Tag Deprecated,please use auto_service.
auto_instance_type Auto Instance Type The type of 'auto_instance'.
auto_instance Auto Instance Tag The instance of IP
auto_service_type Auto Service Type The type of 'auto_service'.
auto_service Auto Service Tag On the basis of 'auto_instance'
gprocess Process
tap_port_host Tap Port Host Deprecated,please use capture_nic_host.
tap_port_chost Tap Port Cloud Host Deprecated,please use capture_nic_chost.
tap_port_pod_node Tap Port K8s Node Deprecated,please use capture_nic_pod_node.
capture_nic_host Host of Capture NIC
capture_nic_chost Cloud Host of Capture NIC
capture_nic_pod_node K8s Node of Capture NIC
host_ip VM Hypervisor The management IP address of VM Hypervisor.
host_hostname VM Hypervisor The hostname of VM Hypervisor.
chost_ip Cloud Host The primary IP address of Cloud Host.
chost_hostname Cloud Host The hostname of Cloud Host.
pod_node_ip K8s Node The primary IP address of K8s Node.
pod_node_hostname K8s Node The hostname of K8s Node.
k8s.label K8s Label
k8s.annotation K8s Annotation
k8s.env K8s Env
cloud.tag Cloud Tag
os.app OS APP
eth_type Ether Type
vlan VLAN TAG
mac MAC Address
ip IP Address
is_ipv4 IPv4 Flag
is_internet Internet IP Flag Whether the IP address is an external Internet address.
province Province The province to which the Internet IP address belongs.
protocol Network Protocol
tunnel_tier Tunnel Tiers
tunnel_type Tunnel Type
tunnel_tx_id TX Tunnel ID
tunnel_rx_id RX Tunnel ID
tunnel_tx_ip TX Tunnel IP Address
tunnel_tx_ip_0 TX Tunnel src IP Address
tunnel_tx_ip_1 TX Tunnel dst IP Address
tunnel_rx_ip RX Tunnel IP Address
tunnel_rx_ip_0 RX Tunnel src IP Address
tunnel_rx_ip_1 RX Tunnel dst IP Address
tunnel_tx_mac TX Tunnel MAC Address
tunnel_tx_mac_0 TX Tunnel src MAC Address
tunnel_tx_mac_1 TX Tunnel dst MAC Address
tunnel_rx_mac RX Tunnel MAC Address
tunnel_rx_mac_0 RX Tunnel src MAC Address
tunnel_rx_mac_1 RX Tunnel dst MAC Address
client_port Client Port
server_port Server Port
tcp_flags_bit TCP Flag Set The set of TCP flags in all packets in the current natural minute.
syn_seq Seq no. of SYN Packet
syn_ack_seq Seq no. of SYN-ACK Packet
last_keepalive_seq Seq no. of Heartbeat Packet Seq number in the most recent heartbeat packet.
last_keepalive_ack Ack no. of Heartbeat Packet Ack number in the most recent heartbeat packet.
l7_protocol Application Protocol
request_domain Request Domain
flow_id Flow ID
start_time Start Time Unit: microseconds. Indicates the start time of the flow within the current natural minute
end_time End Time Unit: microseconds. Indicates the end time of the flow within the current natural minute. If the flow is closed within this minute
close_type Flow Close Type
status Status Determined by the close_type and protocol: Normal/ForceReport/Non-TCP timeout/Disconnected* = Normal
is_new_flow New Flow Flag
signal_source Signal Source
tap Traffic Access Point Deprecated,please use capture_network_type.
capture_network_type Network Location The network location for capturing traffic uses a fixed value (Cloud Network) to represent intra-cloud traffic
vtap DeepFlow Agent Deprecated,please use agent.
agent DeepFlow Agent
nat_source NAT Source
tap_port TAP Port Identifier Deprecated
tap_port_name TAP Port Name Deprecated
tap_port_type TAP Port Type Deprecated
capture_nic Capture NIC ID When the value of tap_port_type is 'Local NIC'
capture_nic_name Capture NIC Name When the value of tap_port_type is 'Local NIC'
capture_nic_type Capture NIC Type Indicates the type of traffic collection location
tap_side TAP Side Deprecated
observation_point Observation Point The logical location of the collection location in the traffic path
l2_end Boundary of L2 Network Indicates whether the traffic is collected on the client NIC or the server NIC.
l3_end Boundary of L3 Network Indicates whether the traffic is collected in the Layer 2 network where the client or server is located.
has_pcap PCAP File Whether the PCAP file is stored
nat_real_ip NAT IP Address The real IP address before (after) NAT
nat_real_port NAT Port The real port number before NAT works

generate from csv file: l4_flow_log.en

#2. Metrics

List of metrics: throughput, load, latency, TCP anomalies, retransmissions, zero window. Detailed field descriptions are as follows.

Field DisplayName Unit Description
byte Byte Byte
byte_tx Byte TX Byte
byte_rx Byte RX Byte
total_byte_tx Total Byte TX Byte
total_byte_rx Total Byte RX Byte
packet Packet Packet
packet_tx Packet TX Packet
packet_rx Packet RX Packet
total_packet_tx Total Packet TX Packet
total_packet_rx Total Packet RX Packet
l3_byte L3 Payload Byte
l3_byte_tx L3 Payload TX Byte
l3_byte_rx L3 Payload RX Byte
bpp Bytes per Packet Byte
bpp_tx Bytes per Packet TX Byte
bpp_rx Bytes per Packet RX Byte
new_flow New Flow Flow
closed_flow Closed Flow Flow
syn_count SYN Packet Packet
synack_count SYN-ACK Packet Packet
l4_byte L4 Payload Byte
l4_byte_tx L4 Payload TX Byte
l4_byte_rx L4 Payload RX Byte
direction_score Direction Score The higher the score
log_count Log Count
retrans_syn SYN Retransmission Packet
retrans_synack SYN-ACK Retransmission Packet
retrans TCP Retransmission Packet
retrans_tx TCP Client Retransmission Packet
retrans_rx TCP Server Retransmission Packet
zero_win TCP ZeroWindow Packet
zero_win_tx TCP Client ZeroWindow Packet
zero_win_rx TCP Server ZeroWindow Packet
retrans_syn_ratio SYN Retrans. % %
retrans_synack_ratio SYN-ACK Retrans. % %
retrans_ratio TCP Retrans. % %
retrans_tx_ratio TCP Client Retrans. % %
retrans_rx_ratio TCP Server Retrans. % %
zero_win_ratio TCP ZeroWindow % %
zero_win_tx_ratio TCP Client ZeroWindow % %
zero_win_rx_ratio TCP Server ZeroWindow % %
tcp_establish_fail Error Flow
client_establish_fail Client Error Flow
server_establish_fail Server Error Flow
tcp_establish_fail_ratio Error % %
client_establish_fail_ratio Client Error % %
server_establish_fail_ratio Client Error % %
tcp_transfer_fail Transfer Error Flow All transfer errors.
tcp_transfer_fail_ratio Transfer Error % %
tcp_rst_fail RST Flow All RST errors.
tcp_rst_fail_ratio RST % %
client_source_port_reuse Est. - Client Port Reuse Flow
server_syn_miss Est. - Server SYN Miss Flow
client_establish_other_rst Est. - Client Other RST Flow
client_ack_miss Est. - Client ACK Miss Flow
server_reset Est. - Server Direct RST Flow
server_establish_other_rst Est. - Server Other RST Flow
client_rst_flow Transfer - Client RST Flow
server_rst_flow Transfer - Server RST Flow
server_queue_lack Transfer - Server Queue Overflow Flow
tcp_timeout Transfer - TCP Timeout Flow
client_half_close_flow Close - Client Half Close Flow
server_half_close_flow Close - Server Half Close Flow
rtt Avg TCP Est. Delay us
tls_rtt Avg TLS Est. Delay us
rtt_client Avg TCP Est. Client Delay us
rtt_server Avg TCP Est. Server Delay us
srt Avg TCP/ICMP ACK Delay us
art Avg Data Delay us
cit Avg Client Idle Delay us
rtt_max Max TCP Est. Delay us
tls_rtt_max Max TLS Est. Delay us
rtt_client_max Max TCP Est. Client Delay us
rtt_server_max Max TCP Est. Server Delay us
srt_max Max TCP/ICMP ACK Delay us
art_max Max Data Delay us
cit_max Max Client Idle Delay us
srt_sum Total TCP/ICMP ACK Delay us
srt_count TCP TCP/ICMP Delay Count
art_sum Total Data Delay us
art_count Data Delay Count
cit_sum Total Client Idle Delay us
cit_count Client Idele Delay Count
duration Duration us The duration from start_time to the last packet (not end_time).
l7_request Request
l7_response Response
rrt Avg App. Delay us
rrt_sum Total App. Delay us
rrt_count App. Delay Count
rrt_max Max App. Delay us
l7_error App. Error
l7_client_error App. Client Error
l7_server_error App. Server Error
l7_server_timeout App. Server Timeout
l7_error_ratio App. Error % %
l7_client_error_ratio App. Client Error % %
l7_server_error_ratio App. Server Error % %
l7_parse_failed L7 Protocol Parse Failed Packet Cumulative number of application protocol parsing failures
row Row Count

generate from csv file: l4_flow_log.en

#3. Grafana Dashboard

Based on the above data, you can build rich dashboards using Grafana. We have pre-configured a Network - Flow Log dashboard in Grafana, as shown below:

Network Flow Log

Network Flow Log

You can also visit DeepFlow Online Demo (opens new window) to see the effect.