花了点时间，将 pwru、skbtracer-iptables、 socketrace 和几个 TCP bpftrace 脚本缝合了起来，形成了一个新的工具，叫做 vista。
GitHub: vista[1]
vista 简介
$ vista -h
Usage: ./vista [options] [pcap-filter]
Available pcap-filter: see "man 7 pcap-filter"
Available options:
--all-kmods attach to all available kernel modules
--filter-addr string filter IP address
--filter-ifname string filter skb ifname in --filter-netns (if not specified, use current netns)
--filter-kprobe-batch uint batch size for kprobe attaching/detaching (default 10)
--filter-netns string filter netns ("/proc/<pid>/ns/net", "inode:<inode>")
--filter-port uint16 filter port
--filter-protocol string filter protocol, tcp, udp, icmp, empty for any
--filter-sk-func string filter kernel sk functions to be probed by name (exact match, supports RE2 regular expression)
--filter-sk-mark uint32 filter sk mark
--filter-skb-drop-stack trace kfree_skb and print skb drop stack
--filter-skb-func string filter kernel skb functions to be probed by name (exact match, supports RE2 regular expression)
--filter-skb-mark uint32 filter skb mark
--filter-tcp-lifetime duration filter TCP lifetime greater than or equal to the given duration (e.g., 100ms, 1s, 1m)
--filter-trace-iptables trace iptables
--filter-trace-sk trace sock
--filter-trace-skb trace skb
--filter-trace-tc trace TC bpf progs
--filter-trace-tcp trace tcp socket lifetime
--filter-trace-xdp trace XDP bpf progs
--filter-track-skb trace a packet even if it does not match given filters (e.g., after NAT or tunnel decapsulation)
-h, --help display this message and exit
--kernel-btf string specify kernel BTF file
--kmods strings list of kernel modules names to attach to
--kprobe-backend string Tracing backend('kprobe', 'kprobe-multi'). Will auto-detect if not specified.
--output-file string write traces to file
--output-iptables print iptables
--output-limit-lines uint exit the program after the number of events has been received/printed
--output-meta print skb metadata
--output-sk print sock
--output-skb print skb
--output-stack print stack
--output-tcp print TCP
--output-tuple print L4 tuple
--timestamp string print timestamp per event ("relative", "absolute", "none") (default "none")
--version show vista version and exit
除了 pwru 原有的功能，vista 还支持了更多的过滤选项，比如：
--filter-addr：过滤 IP 地址，为了支持--filter-trace-tcp。--filter-port：过滤端口，为了支持--filter-trace-tcp。--filter-skb-drop-stack：打印 skb drop 的 stack，便利性选项，效果等同于--filter-skb-func __kfree_skb --output-stack。--filter-tcp-lifetime：过滤 TCP 生命周期大于等于给定的时间，便于分析 tcp 长连接。--filter-trace-tcp：跟踪 TCP 生命周期，包括连接建立、连接关闭等事件。--filter-trace-iptables：跟踪 iptables 执行，包括规则匹配、规则动作等事件，即skbtracer-iptables的功能。--filter-trace-sk：类似--filter-trace-skb，跟踪 sock 结构体指针为参数的函数。--filter-trace-xdp: 跟踪 XDP bpf progs。
更多输出选项：
--output-iptables：打印 iptables 规则信息。--output-sk：打印 sock 信息。--output-tcp：打印 TCP 生命周期信息。
vista 使用示例
跟踪 TCP 生命周期
$ sudo ./vista --filter-trace-tcp --output-meta --output-tuple --output-tcp
2024/05/12 12:52:21 Attached tcp kprobes/tracepoints
2024/05/12 12:52:21 Listening for events..
SKB/SK CPU PROCESS FUNC
0xffff8c1f06684600 2 [gdbus(2000)] TCP netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:63174(tcp) rx=5.326KiB tx=5.747KiB lifetime=939.570803ms srtt=240µs retrans=0 sk_mark=0x0 cong=cubic
跟踪 iptables 执行
$ sudo ./vista --filter-trace-iptables --output-meta --output-tuple --output-iptables --output-limit-lines 10
2024/05/12 14:11:08 Attached iptables kprobes
2024/05/12 14:11:08 Listening for events..
SKB/SK CPU PROCESS FUNC
0xffff98ce06243700 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=0 proto=0x0800 mtu=0 len=184 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=OUTPUT verdict=ACCEPT cost=5.932µs
0xffff98ce06243700 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=184 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=POSTROUTING verdict=ACCEPT cost=930ns
0xffff98ce06242580 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=0 proto=0x0800 mtu=0 len=160 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=OUTPUT verdict=ACCEPT cost=6.258µs
0xffff98ce06242580 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=160 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=POSTROUTING verdict=ACCEPT cost=1.417µs
0xffff98ce06241680 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=0 proto=0x0800 mtu=0 len=344 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=OUTPUT verdict=ACCEPT cost=3.165µs
0xffff98ce06241680 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=344 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=POSTROUTING verdict=ACCEPT cost=979ns
0xffff98ce06242580 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=0 proto=0x0800 mtu=0 len=360 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=OUTPUT verdict=ACCEPT cost=3.019µs
0xffff98ce06242580 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=360 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=POSTROUTING verdict=ACCEPT cost=1.001µs
0xffff98ce06241680 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=0 proto=0x0800 mtu=0 len=344 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=OUTPUT verdict=ACCEPT cost=2.548µs
0xffff98ce06241680 1 [/usr/sbin/sshd(727984)] ipt_do_table netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=344 pkt_type=HOST 192.168.241.128:22->192.168.241.1:51170(tcp) pf=PF_INET table= hook=POSTROUTING verdict=ACCEPT cost=957ns
2024/05/12 14:11:08 Printed 10 events, exiting program..
跟踪 sock 结构体指针为参数的函数
$ sudo ./vista --filter-trace-sk --output-meta --output-tuple --output-sk --output-limit-lines 10
2024/05/12 13:30:11 Attaching sk kprobes (via kprobe-multi)...
1196 / 1196 [---------------------------------------------------------------------------------------------] 100.00% ? p/s
2024/05/12 13:30:11 Attached sk kprobes (ignored 0)
2024/05/12 13:30:11 Listening for events..
SKB/SK CPU PROCESS FUNC
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] inet_send_prepare netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] tcp_sendmsg netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] lock_sock_nested netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] tcp_sendmsg_locked netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] tcp_rate_check_app_limited netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] tcp_send_mss netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] tcp_current_mss netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] tcp_established_options netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] tcp_stream_alloc_skb netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
0xffff8c1f06683200 4 [/usr/sbin/sshd(1407)] sk_forced_mem_schedule netns=4026531840 mark=0x0 iface=0 proto=0x0000 mtu=0 len=0 pkt_type=HOST 192.168.241.133:22->192.168.241.1:60271(tcp) skc_state=ESTABLISHED skc_reuseport=false skc_bound_ifindex=0 sk_backlog=0 sk_rcv_buff=614849 sk_snd_buff=87040 sk_priority=6 sk_mark=0x0 sk_type=SOCK_STREAM socket_state=CONNECTED socket_file_inode=15100 socket_flags=32
2024/05/12 13:30:11 Printed 10 events, exiting program..
2024/05/12 13:30:11 Detaching sk kprobes...
5 / 5 [--------------------------------------------------------------------------------------------------] 100.00% 15 p/s
总结
vista 是一个集成了 pwru、skbtracer-iptables、socketrace 和几个 TCP bpftrace 脚本的工具，支持更多的过滤选项和输出选项，方便用户进行更多的网络跟踪和分析。
GitHub: vista: https://github.com/Asphaltt/vista
