网络观察方法

工具

netstat

• 默认：列出连接的套接字
• -a：所有套接字
• -s: 网络栈统计信息
• -i：网络接口信息
• -r：列出路由表
• -n：不解析IP为主机名
• -v：显示冗长的详细信息
• -c：连续模式
  
• OK：成功传输的数据包
• ERR：错误数据包
• DRP：丢包
• OVR：超限

netstat -s
Ip:
    903984797 total packets received
    0 forwarded
    0 incoming packets discarded
    903984742 incoming packets delivered
    903601997 requests sent out
    48 dropped because of missing route
Icmp:
    210617 ICMP messages received
    135814 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 209724
        timeout in transit: 707
        echo requests: 186
    209865 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 209865
IcmpMsg:
        InType3: 209724
        InType8: 186
        InType11: 707
        OutType3: 209865
Tcp:
    447882384 active connections openings
    557407 passive connection openings
    446943009 failed connection attempts
    163 connection resets received
    8 connections established
    902646251 segments received
    902815309 segments send out
    9470 segments retransmited
    0 bad segments received.
    445282817 resets sent
Udp:
    1045235 packets received
    206 packets to unknown port received.
    0 packet receive errors
    600846 packets sent
    0 receive buffer errors
    0 send buffer errors
UdpLite:
TcpExt:
    34 invalid SYN cookies received
    1 ICMP packets dropped because they were out-of-window
    621974 TCP sockets finished time wait in fast timer
    3197 TCP sockets finished time wait in slow timer
    19870 delayed acks sent
    2 delayed acks further delayed because of locked socket
    Quick ack mode was activated 1316 times
    77229 packets directly queued to recvmsg prequeue.
    168675 bytes directly in process context from backlog
    8606364 bytes directly received in process context from prequeue
    1589005 packet headers predicted
    52475 packets header predicted and directly queued to user
    2790664 acknowledgments not containing data payload received
    1694511 predicted acknowledgments
    696 congestion windows recovered without slow start after partial ack
    18 timeouts in loss state
    136610 other TCP timeouts
    49 connections reset due to unexpected data
    2 connections reset due to early user close
    93 connections aborted due to timeout
    TCPRcvCoalesce: 1526118
    TCPOFOQueue: 2019
    TCPOFOMerge: 109
    TCPChallengeACK: 1
    TCPSpuriousRtxHostQueues: 135894
    TCPAutoCorking: 33959
    TCPSynRetrans: 8620
    TCPOrigDataSent: 4521933
    TCPHystartTrainDetect: 55
    TCPHystartTrainCwnd: 1251
    TCPACKSkippedSynRecv: 6
IpExt:
    InMcastPkts: 2
    InBcastPkts: 82370
    InOctets: 39333180487
    OutOctets: 39018604444
    InMcastOctets: 72
    InBcastOctets: 16274146
    InNoECTPkts: 903973618
    InECT0Pkts: 11179
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

多项按协议分组的网络数据，主要是TCP，值得关注的指标：

• 相比接收的总数据包更高速的包转发率：检查服务器是否应该转发（路由）数据包
• 开放的被动连接：监视它们能显示客户机连接负载
• 相比发送的数据段更高的数据段重传率：能支持网络的不稳定
• 套接字缓冲超限导致的数据包从接收队列中删除：这是网络饱和的标志，增加套接字缓冲修复

/proc/net/snamp 统计信息

cat /proc/net/snmp

Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
Ip: 2 64 903985425 0 0 0 0 0 903985370 903602588 0 48 0 0 0 0 0 0 0
Icmp: InMsgs InErrors InCsumErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
Icmp: 210617 135814 0 209724 707 0 0 0 186 0 0 0 0 0 209865 0 209865 0 0 0 0 0 0 0 0 0 0
IcmpMsg: InType3 InType8 InType11 OutType3
IcmpMsg: 209724 186 707 209865
Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts InCsumErrors
Tcp: 1 200 120000 -1 447882801 557407 446943426 163 8 902646758 902815832 9470 0 445282817 0
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors
Udp: 1045318 206 0 600915 0 0 0
UdpLite: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors
UdpLite: 0 0 0 0 0 0 0

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

sar

ifconfig 逐渐被ip命令淘汰，总体与netstat -i结果类似

traceroute
发出一系列数据包实验性的探测到一个主机当前的路由

tcpdump/wireshark这个需要单独聊

systemtap/perf 这个也要单独聊

strace 跟踪套接字相关的系统调用并检查其使用的选项
lsof 按进程ID列出包括套接字细节在内的打开的文件
ss 套接字统计信息
nfsstat NFS服务器和客户机统计信息
iftop 按主机（嗅探）总结网络接口吞吐量
/proc/net网络统计信息文件

---

书中大篇幅的谈到用dtrace来做各种探针检测，但是dtrace还是有点学习成本，暂且先放着了，不过提到的一些概念倒是可以记录一下
套接字延时：

• 连接延时：对于同步的系统调用，是connect()消耗的时间；对于非阻塞的I/O，是执行connect() 至poll() 或者select()（或其他系统调用）报告套接字就绪的时间
• 首字节延时：自执行connect()或者从accept()返回，直到第一字节数据由任何一个I/O系统调用从套接字接收到的时间。
• 套接字持续时间：同一个文件描述符由socket()到close()的时间；要聚焦连接的持续时间，可以由connect()或者accept()开始计时。