Monitoring Cloud Servers
This document was translated by ChatGPT
#1. Introduction
DeepFlow supports monitoring cloud servers and automatically injects cloud resource information into all observability data (AutoTagging) by calling cloud vendor APIs. Note that DeepFlow Server must run on K8s. If you do not have a K8s cluster, you can refer to the All-in-One Quick Deployment section to deploy DeepFlow Server first.
#2. Deployment Topology
#3. Create Public Cloud Domain
DeepFlow currently supports resource information synchronization for the following public clouds (marked as TBD
are under code organization):
Cloud Provider (English) | Cloud Provider (Chinese) | Type Identifier Used in DeepFlow |
---|---|---|
AWS | AWS | aws |
Aliyun | 阿里云 | aliyun |
Baidu Cloud | 百度云 | baidu_bce |
Huawei Cloud | 华为云 | huawei |
Microsoft Azure | 微软云 | TBD |
QingCloud | 青云 | qingcloud |
Tencent Cloud | 腾讯云 | tencent |
You can get the configuration file template for creating a public cloud Domain using the deepflow-ctl domain example <domain_type>
command.
For example, for Aliyun:
deepflow-ctl domain example aliyun > aliyun.yaml
Modify the configuration file aliyun.yaml
, filling in the AK/SK (requires read-only permissions for cloud resources) and the Region information where the resources are located:
name: aliyun
type: aliyun
config:
# AccessKey Id
secret_id: xxxxxxxx ## FIXME: your secret_id
# AccessKey Secret
secret_key: xxxxxxx ## FIXME: your secret_key
include_regions: 华北2(北京) ## The region where deepflow is docked, if it is empty, it means all regions, and the regions are separated by commas
2
3
4
5
6
7
8
Create the public cloud Domain using the modified configuration file:
deepflow-ctl domain create -f aliyun.yaml
#4. Deploy DeepFlow Agent
Download deepflow-agent
curl -O https://deepflow-ce.oss-cn-beijing.aliyuncs.com/rpm/agent/stable/linux/$(arch | sed 's|x86_64|amd64|' | sed 's|aarch64|arm64|')/deepflow-agent-rpm.zip
unzip deepflow-agent-rpm.zip
yum -y localinstall x86_64/deepflow-agent-1.0*.rpm
2
3
curl -O https://deepflow-ce.oss-cn-beijing.aliyuncs.com/deb/agent/stable/linux/$(arch | sed 's|x86_64|amd64|' | sed 's|aarch64|arm64|')/deepflow-agent-deb.zip
unzip deepflow-agent-deb.zip
dpkg -i x86_64/deepflow-agent-1.0*.systemd.deb
2
3
curl -O https://deepflow-ce.oss-cn-beijing.aliyuncs.com/bin/agent/stable/linux/$(arch | sed 's|x86_64|amd64|' | sed 's|aarch64|arm64|')/deepflow-agent.tar.gz
tar -zxvf deepflow-agent.tar.gz -C /usr/sbin/
cat << EOF > /etc/systemd/system/deepflow-agent.service
[Unit]
Description=deepflow-agent.service
After=syslog.target network-online.target
[Service]
Environment=GOTRACEBACK=single
LimitCORE=1G
ExecStart=/usr/sbin/deepflow-agent
Restart=always
RestartSec=10
LimitNOFILE=1024:4096
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
touch /etc/deepflow-agent.yaml
cat << EOF > deepflow-agent-docker-compose.yaml
version: '3.2'
services:
deepflow-agent:
image: registry.cn-hongkong.aliyuncs.com/deepflow-ce/deepflow-agent:stable
container_name: deepflow-agent
restart: always
#privileged: true ## Docker version below 20.10.10 requires the opening of the privileged mode, See https://github.com/moby/moby/pull/42836
cap_add:
- SYS_ADMIN
- SYS_RESOURCE
- SYS_PTRACE
- NET_ADMIN
- NET_RAW
- IPC_LOCK
- SYSLOG
volumes:
- /etc/deepflow-agent.yaml:/etc/deepflow-agent/deepflow-agent.yaml:ro
- /sys/kernel/debug:/sys/kernel/debug:ro
- /var/run/docker.sock:/var/run/docker.sock
network_mode: "host"
pid: "host"
EOF
docker compose -f deepflow-agent-docker-compose.yaml up -d
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
Modify the deepflow-agent configuration file /etc/deepflow-agent.yaml
:
controller-ips:
- 10.1.2.3 # FIXME: K8s Node IPs
2
Start deepflow-agent:
systemctl enable deepflow-agent
systemctl restart deepflow-agent
2
Note:
If deepflow-agent fails to start due to missing dependencies, you can download the statically linked compiled deepflow-agent. Note that the statically linked compiled deepflow-agent has severe performance issues under multithreading:
curl -O https://deepflow-ce.oss-cn-beijing.aliyuncs.com/rpm/agent/stable/linux/static-link/$(arch | sed 's|x86_64|amd64|' | sed 's|aarch64|arm64|')/deepflow-agent-rpm.zip
unzip deepflow-agent-rpm.zip
yum -y localinstall x86_64/deepflow-agent-1.0*.rpm
2
3
curl -O https://deepflow-ce.oss-cn-beijing.aliyuncs.com/deb/agent/stable/linux/static-link/$(arch | sed 's|x86_64|amd64|' | sed 's|aarch64|arm64|')/deepflow-agent-deb.zip
unzip deepflow-agent-deb.zip
dpkg -i x86_64/deepflow-agent-1.0*.systemd.deb
2
3
curl -O https://deepflow-ce.oss-cn-beijing.aliyuncs.com/bin/agent/stable/linux/static-link/$(arch | sed 's|x86_64|amd64|' | sed 's|aarch64|arm64|')/deepflow-agent.tar.gz
tar -zxvf deepflow-agent.tar.gz -C /usr/sbin/
cat << EOF > /etc/systemd/system/deepflow-agent.service
[Unit]
Description=deepflow-agent.service
After=syslog.target network-online.target
[Service]
Environment=GOTRACEBACK=single
LimitCORE=1G
ExecStart=/usr/sbin/deepflow-agent
Restart=always
RestartSec=10
LimitNOFILE=1024:4096
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#5. Next Steps
- Universal Service Map - Experience DeepFlow's AutoMetrics Capability
- Distributed Tracing - Experience DeepFlow's AutoTracing Capability
- Eliminate Data Silos - Learn About DeepFlow's AutoTagging and SmartEncoding Capabilities
- Say Goodbye to High Baseline Troubles - Integrate Metrics Data from Prometheus, etc.
- Full-Stack Distributed Tracing - Integrate Tracing Data from OpenTelemetry, etc.