使用zeek做HTTP RPC性能检测

Zeek是一个网络流量分析框架，开源的工具，遵循BSD协议；
框架的高级之处就是在于它支持使用脚本编写协议分析的扩展，
并提供了基础的协议分析，整个脚本的使用时基于事件驱动；用户可以在现有基础上做扩展。
github下载地址为：
下载地址：https://github.com/zeek/zeek/releases/tag/v5.0.0

**需求：**我的需求是在K8S的云上添加自己的非侵入式采集工具，并检测各种后端的RPC性能，比如: http_rpc, grpc, mysql, redis, mongodb, 以及自定义的rpc等网络交互过程的性能；

**解决方案：**直接在zeek现有基础上开发脚本，并开发日志读写工具将结果转存到自己的后端；
比如HTTP，则需要记录请求和应答的时间点，计算往返时延，记录URL, http请求类型等；

实验代码：
首先，参考HTTP的相关脚本，记录自己需要的数据，写个hello, world

@load base/protocols/http

module Robin;

export {
    # Create an ID for our new stream. By convention, this is
    # called "LOG".
    redef enum Log::ID += { LOG };

    # Define the record type that will contain the data to log.
    type Info: record {
        ts: time        &log;
        url: string     &log;
        met: string  &log;
        ver: string &log;
    };
}

event http_request(c: connection, method: string, original_URI: string,
                   unescaped_URI: string, version: string) &priority=6
{
    local rec: Robin::Info = [$ts=network_time(),$url=original_URI, $met=method, $ver=version];

    # Store a copy of the data in the connection record so other
    # event handlers can access it.
    #c$robin = rec;

    Log::write(Robin::LOG, rec);
}

event zeek_init()
{
	print "zeek_init_robin()";
    # Create the stream. This adds a default filter automatically.
    Log::create_stream(Robin::LOG, [$columns=Info, $path="robin"]);
}

event zeek_done()
{
	print "zeek_done_robin()";
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

在官方的使用网站上，https://try.bro.org/

更改脚本，并选择exercise_traffic.pcap ，点击Run 按钮

会有输出，在下面的LOG列表中，查看Robin表格：

在原来的连接基础上扩展一下自己用的东西，后面直接使用：

https://try.bro.org/#/tryzeek/saved/599238

@load base/protocols/http

module Robin;

export {
    # Create an ID for our new stream. By convention, this is
    # called "LOG".
    redef enum Log::ID += { LOG };

    # Define the record type that will contain the data to log.
    type Info: record {
        ts: time    &log;
        endTs: time &log &optional;
        delta: interval &log &optional;
        url: string &log;
        met: string &log;
        ver: string &log;
        host:string &log &optional &default="";
        code: count &log &optional &default=0;
       
    };
}

# Optionally, we can add a new field to the connection record so that
# the data we are logging (our "Info" record) will be easily
# accessible in a variety of event handlers.
redef record connection += {
    # By convention, the name of this new field is the lowercase name
    # of the module.
    robinField: Info &optional;
};

event http_request(c: connection, method: string, original_URI: string,
                   unescaped_URI: string, version: string) &priority=6
{
    local rec: Robin::Info = [$ts=network_time(),$url=original_URI, $met=method, $ver=version];

    # Store a copy of the data in the connection record so other
    # event handlers can access it.
    c$robinField = rec;

    
}

event http_header(c: connection, is_orig: bool, name: string, value: string) &priority=6
{
    if ( name == "HOST" )
			# Per #1844, we record the original host header, including any port
			# specification if present.
			c$robinField$host = value;
}
event http_reply(c: connection, version: string, code: count, reason: string) &priority=6
{
    c$robinField$code = code;
    c$robinField$endTs = network_time();
    c$robinField$delta = c$robinField$endTs - c$robinField$ts;
    
    Log::write(Robin::LOG, c$robinField);
}

event zeek_init()
{
	print "zeek_init_robin()";
    # Create the stream. This adds a default filter automatically.
    Log::create_stream(Robin::LOG, [$columns=Info, $path="robin"]);
}

event zeek_done()
{
	print "zeek_done_robin()";
}


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

这样就基本实现了我们的需求，后续的工作只是需要扩展各种协议，以及从日志流提取数据；
这个东西确实很NICE，不需要自己写底层的采集数据的逻辑了；避免反复造轮子；

备注：官网的几个链接

1）讲解了事件与事件队列的原理
https://docs.zeek.org/en/current/scripting/basics.html#the-event-queue-and-event-handlers
2）日志与脚本示例
https://docs.zeek.org/en/master/frameworks/logging.html