-
Pipeline aggregations管道聚合- parent-2
Pipeline aggregations
管道聚合工作于其他聚合产生的输出结果而不是文档集，用于向输出树添加信息。有不同类型的管道聚合，每一种从其他聚合中计算不同的信息，但这些类型可以分为以下两个类型：
(1)基于父聚集
基于父聚集的管道聚集包括moving_avg、moving_fn、bucket_script、bucket_selector、bucket_sort、derivative、cumulative_sum、serial_diff等。
(2)基于兄弟聚集
基于兄弟聚集的管道聚集包括avg_bucket、max_bucket、min_bucket、sum_bucket、stats_bucket、extended_ stats_ bucket、percentiles_bucket七种。如果将它们名称中的bucket去除，它们就与之前介绍的部分指标聚集同名了。事实上，它们不仅在名称上接近，而且在功能上也类似，只是聚集运算的范围由整个文档变成了另一个聚集结果
1. Moving average aggregation ====================================================
-
parent pipeline aggregation，moving_avg
-
窗口平均值聚合。基于已经排序过的数据，计算出处在当前出口中数据的平均值。比如窗口大小为 5 ，对数据 1-10 的部分窗口平均值如下：
(1 + 2 + 3 + 4 + 5) / 5 = 3
(2 + 3 + 4 + 5 + 6) / 5 = 4
(3 + 4 + 5 + 6 + 7) / 5 = 5
… -
移动平均值是一种平滑顺序数据的简单方法。
-
移动平均值通常应用于基于时间的数据，例如股票价格或服务器指标。 平滑可用于消除高频波动或随机噪声，从而使低频趋势更容易可视化，例如季节性。
-
moving_avg聚合必须嵌入histogram或date_histogram。也就是它必须使用另外一个 aggregation的结果，并在它的基础之上再进行聚合
-
参数
key Value buckets_path 用于计算均值的权值路径 gap_policy 空桶处理策略（skip/insert_zeros） model 移动模型 window 在直方图上“滑动”的窗口大小。 minimize settings 计算每月sales销售额的移动平均值:
-
一级聚合：sales_per_month，（按月）直方图date_histogam桶聚合。
-
二级子聚合：the_sum，在按月聚合的基础上，对每个月的文档求sum。
-
二级子聚合：the_avg/the_movavg/the_movfn_unweightedAvg/the_movfn_unweightedAvg，在按月聚合的基础上，对每个月的文档求各种平均值
-
“buckets_path”: “the_sum”
聚合需要得到sales_per_month数据直方图中计算sales聚合的统计数据
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "1M", "format": "yyyy-MM" }, "aggs": { "the_sum": { "sum": { "field": "price" } }, "the_avg": { "avg": { "field": "price" } }, "the_movavg": { "moving_avg": { "buckets_path": "the_sum", "window": 10, "model": "simple" } }, "the_movfn_unweightedAvg": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.unweightedAvg(values)" } }, "the_movfn_linearWeightedAvg": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.linearWeightedAvg(values)" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
响应结果：
- aggregations 由 sales_per_month 组成
- 按月份聚合bucket共3个。
key Value key_as_string 月份 doc_count 当月的doc数量 the_sum.value 当月计算字段price的总和sum值 the_movavg.value 当月price的the_sum的moving_avg the_sum [140, 30.0, 2110.0] doc_count [2, 2, 12] 平均数 the_avg [70, 15.0, 175.83] the_movavg [, 140.0, 85.0 ] the_movfn_unweightedAvg [null, 140.0, 85.0] the_movfn_linearWeightedAvg [null, 70.0, 50.0] { "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_avg" : { "value" : 70.0 }, "the_sum" : { "value" : 140.0 }, "the_movfn_unweightedAvg" : { "value" : null }, "the_movfn_linearWeightedAvg" : { "value" : null } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_avg" : { "value" : 15.0 }, "the_sum" : { "value" : 30.0 }, "the_movavg" : { "value" : 140.0 }, "the_movfn_unweightedAvg" : { "value" : 140.0 }, "the_movfn_linearWeightedAvg" : { "value" : 70.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_avg" : { "value" : 175.83333333333334 }, "the_sum" : { "value" : 2110.0 }, "the_movavg" : { "value" : 85.0 }, "the_movfn_unweightedAvg" : { "value" : 85.0 }, "the_movfn_linearWeightedAvg" : { "value" : 50.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
2. Moving function aggregation 滑动窗口上自定义函数 ====================================================
- parent pipeline aggregation，moving_fn
计算每月sales销售额的移动平均值:
- 一级聚合：sales_per_month，（按月）直方图date_histogam桶聚合。
- 二级子聚合：the_sum，在按月聚合的基础上，对每个月的文档求sum。
- 二级子聚合：moving_fn，在按月聚合的基础上，对每个月的文档求各种指标。
1.简单使用moving_fn
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "1M", "format": "yyyy-MM" }, "aggs": { "the_sum": { "sum": { "field": "price" } }, "the_movfn": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.unweightedAvg(values)" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
moving_fn响应结果
{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_sum" : { "value" : 140.0 }, "the_movfn" : { "value" : null } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_sum" : { "value" : 30.0 }, "the_movfn" : { "value" : 140.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_sum" : { "value" : 2110.0 }, "the_movfn" : { "value" : 85.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
2.moving_fn + script
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "1M", "format": "yyyy-MM" }, "aggs": { "the_sum": { "sum": { "field": "price" } }, "the_movavg": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "return values.length > 0 ? values[0] : Double.NaN" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
moving_fn + script 响应结果
- the_movavg [null,140.0,140.0]
{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_sum" : { "value" : 140.0 }, "the_movavg" : { "value" : null } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_sum" : { "value" : 30.0 }, "the_movavg" : { "value" : 140.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_sum" : { "value" : 2110.0 }, "the_movavg" : { "value" : 140.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
3.moving_fn + script + Pre-built Functions
key Value 参数 max() 取滑动窗口数据的最大值 max(double[] values) min() 取滑动窗口数据的最小值 min(double[] values) sum() 取滑动窗口中数据的和 sum(double[] values) stdDev() 取滑动窗口的标准差 stdDev(double[] values) unweightedAvg() 取滑动窗口非加权平均值 unweightedAvg(double[] values) linearWeightedAvg() 取滑动窗口线性加权的平均值 linearWeightedAvg(double[] values) ewma() 取滑动窗口的指数加权平均值 ewma(double[] values, double alpha) holt() 取滑动窗口的双指数加权平均值 holt(double[] values, double alpha) holtWinters() 取滑动窗口的三级指数平均值，“水平”、“趋势”和“季节性” holtWinters(double[] values, double alpha) 3.1 max min sum stdDev
key Value 参数 max() 取滑动窗口数据的最大值 max(double[] values) min() 取滑动窗口数据的最小值 min(double[] values) sum() 取滑动窗口中数据的和 sum(double[] values) stdDev() 取滑动窗口的标准差 stdDev(double[] values) POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "1M", "format": "yyyy-MM" }, "aggs": { "the_sum": { "sum": { "field": "price" } }, "the_movfn_max": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.max(values)" } }, "the_movfn_min": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.min(values)" } }, "the_movfn_sum": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.sum(values)" } }, "the_movfn_stdDev": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.stdDev(values, MovingFunctions.unweightedAvg(values))" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
max min sum stdDev 响应结果
key Value the_movfn_max [null, 140.0, 140.0] the_movfn_min [null, 140.0, 30.0] the_movfn_sum [0, 140.0, 170.0] the_movfn_stdDev [null, 0.0, 55.0] { "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_sum" : { "value" : 140.0 }, "the_movfn_max" : { "value" : null }, "the_movfn_min" : { "value" : null }, "the_movfn_sum" : { "value" : 0.0 }, "the_movfn_stdDev" : { "value" : null } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_sum" : { "value" : 30.0 }, "the_movfn_max" : { "value" : 140.0 }, "the_movfn_min" : { "value" : 140.0 }, "the_movfn_sum" : { "value" : 140.0 }, "the_movfn_stdDev" : { "value" : 0.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_sum" : { "value" : 2110.0 }, "the_movfn_max" : { "value" : 140.0 }, "the_movfn_min" : { "value" : 30.0 }, "the_movfn_sum" : { "value" : 170.0 }, "the_movfn_stdDev" : { "value" : 55.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
3.2 unweightedAvg linearWeightedAvg
key Value 参数 unweightedAvg() 取滑动窗口非加权平均值 unweightedAvg(double[] values) linearWeightedAvg() 取滑动窗口线性加权的平均值 linearWeightedAvg(double[] values) POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "1M", "format": "yyyy-MM" }, "aggs": { "the_sum": { "sum": { "field": "price" } }, "the_movfn_unweightedAvg": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.unweightedAvg(values)" } }, "the_movfn_linearWeightedAvg": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.linearWeightedAvg(values)" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
unweightedAvg linearWeightedAvg响应结果
key Value the_movfn_unweightedAvg [null, 140.0, 85.0] the_movfn_linearWeightedAvg [null, 70.0, 50.0] { "took" : 2, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_sum" : { "value" : 140.0 }, "the_movfn_unweightedAvg" : { "value" : null }, "the_movfn_linearWeightedAvg" : { "value" : null } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_sum" : { "value" : 30.0 }, "the_movfn_unweightedAvg" : { "value" : 140.0 }, "the_movfn_linearWeightedAvg" : { "value" : 70.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_sum" : { "value" : 2110.0 }, "the_movfn_unweightedAvg" : { "value" : 85.0 }, "the_movfn_linearWeightedAvg" : { "value" : 50.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
3.3 ewma holt holtWinters
key Value 参数 ewma() 取滑动窗口的指数加权平均值 ewma(double[] values, double alpha) holt() 取滑动窗口的双指数加权平均值 holt(double[] values, double alpha) holtWinters() 取滑动窗口的三级指数平均值，“水平”、“趋势”和“季节性” holtWinters(double[] values, double alpha) POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "1M", "format": "yyyy-MM" }, "aggs": { "the_sum": { "sum": { "field": "price" } }, "the_movfn_ewma": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.ewma(values, 0.3)" } }, "the_movfn_holt": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "MovingFunctions.holt(values, 0.3, 0.1)" } }, "the_movfn_holtWinters": { "moving_fn": { "buckets_path": "the_sum", "window": 10, "script": "if (values.length > 5*2) {MovingFunctions.holtWinters(values, 0.3, 0.1, 0.1, 5, false)}" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
ewma holt holtWinters 响应结果
key Value ewma 单指数加权平均值 holt 双指数加权平均值 holtWinters 三级指数平均值 key Value the_movfn_ewma [null, 140.0, 107.0] the_movfn_holt [null, 140.0, 107.0] the_movfn_holtWinters [0.0, 0.0, 0.0] { "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_sum" : { "value" : 140.0 }, "the_movfn_ewma" : { "value" : null }, "the_movfn_holt" : { "value" : null }, "the_movfn_holtWinters" : { "value" : 0.0 } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_sum" : { "value" : 30.0 }, "the_movfn_ewma" : { "value" : 140.0 }, "the_movfn_holt" : { "value" : 140.0 }, "the_movfn_holtWinters" : { "value" : 0.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_sum" : { "value" : 2110.0 }, "the_movfn_ewma" : { "value" : 107.0 }, "the_movfn_holt" : { "value" : 107.0 }, "the_movfn_holtWinters" : { "value" : 0.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
3. Moving percentiles aggregation ====================================================
- parent pipeline aggregation，moving_percentiles
- 该管道聚合在百分位数聚合之上工作，并使用滑动窗口计算累积百分位数。
计算累积百分位数
- 1,25,50,75,99 计算从聚合的文档中提取的数值上的一个或多个百分位值。这些值可以从文档中某些特定的数值字段中提取出来，也可以使用给定的脚本生成。
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "1M", "format": "yyyy-MM" }, "aggs": { "the_percentile": { "percentiles": { "field": "price", "percents": [ 1, 25, 50, 75, 99 ] } }, "the_movperc": { "moving_percentiles": { "buckets_path": "the_percentile", "window": 10 } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
moving_percentiles响应结果
the_percentile：
key 2022-04 2022-05 2022-06 1 59.9 10 10 25 60 10 90 50 70 15 170 75 80 20 260 99 80 20 380 the_movperc：
key 2022-04 2022-05 2022-06 1 59.9 10 25 60 15 50 70 40 75 80 70 99 80 80 { "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_percentile" : { "values" : { "1.0" : 59.99999999999999, "25.0" : 60.0, "50.0" : 70.0, "75.0" : 80.0, "99.0" : 80.0 } } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_percentile" : { "values" : { "1.0" : 10.0, "25.0" : 10.0, "50.0" : 15.0, "75.0" : 20.0, "99.0" : 20.0 } }, "the_movperc" : { "values" : { "1.0" : 59.99999999999999, "25.0" : 60.0, "50.0" : 70.0, "75.0" : 80.0, "99.0" : 80.0 } } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_percentile" : { "values" : { "1.0" : 10.0, "25.0" : 90.0, "50.0" : 170.0, "75.0" : 260.0, "99.0" : 380.0 } }, "the_movperc" : { "values" : { "1.0" : 10.000000000000002, "25.0" : 15.0, "50.0" : 40.0, "75.0" : 70.0, "99.0" : 80.0 } } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
4. Serial differencing aggregation 串行差异聚合 ====================================================
-
parent pipeline aggregation，serial_diff
-
串行差分是将在一个时间序列中的值减去其本身滞后一段时间或周期的值。例如，数据点 f(x) = f(xt) - f(xt-n)，其中n是所使用的周期。
-
serial_diff聚合必须嵌入到histogram或者date_histogram聚合中
-
参数
key 2022-04 buckets_path 度量的路径 lag 从当前的值减去历史bucket。比如，lag=7表示每次从当前的bucket的值减去其前面第7个bucket的值。必须是非零正整数。 gap_policy 空桶策略 format 格式化聚合输出的值 串行差分 lag=1
- date_histogram 在字段上指定 histogram 或者 date_histogram
- the_sum sum度量是用来计算字段的和。可以填很多值（sum, min, max, etc）
- thirtieth_difference 以the_sum作为输入的serial_diff聚合。
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "month", "format": "yyyy-MM" }, "aggs": { "the_sum": { "sum": { "field": "price" } }, "thirtieth_difference": { "serial_diff": { "buckets_path": "the_sum", "lag": 1 } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
serial_diff响应结果
bucket Value 2022-04： 140-0=0 2022-05： 30-140=-110.0 2022-06： 2110-30=2080.0 { "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_sum" : { "value" : 140.0 } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_sum" : { "value" : 30.0 }, "thirtieth_difference" : { "value" : -110.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_sum" : { "value" : 2110.0 }, "thirtieth_difference" : { "value" : 2080.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
5. Bucket script aggregation 桶脚本聚合====================================================
- parent pipeline aggregation， bucket_script
- 父级管道聚合，它执行一个脚本，它可以在父级多桶聚合中指定度量参数执行每个桶计算。指定的度量必须是数字，脚本必须返回一个数值。
计算每月sales不同类型销售额的占比:
-
一级聚合：sales_per_month，（按月）直方图date_histogam桶聚合。
-
二级子聚合：total_sales，在按月聚合的基础上，统计每个月的商品销售总额
-
二级子聚合：t-shirts，统计每个月的t-shirt类型销售总额
-
二级子聚合：hats，统计每个月的hat类型销售总额
-
二级子聚合：peppers，统计每个月的pepper类型销售总额
-
二级子聚合：t-shirt-percentage
-
1.buckets_path两个数据来源：
-
- （1）.“tShirtSales”: “t-shirts>sales”：
– 父子关系 （t-shirts 父 ；sales 子）
– 指向sales指标值，该值包含在t-shirts聚合桶内
– t-shirts分组聚合的sales指标作为pipeline aggregations的输入源。
- （1）.“tShirtSales”: “t-shirts>sales”：
-
- （2）.“totalSales”: “total_sales”
-
2.script
“script”: “params.tShirtSales / params.totalSales * 100” -
二级子聚合：hat-percentage，buckets_path两个数据来源：
-
1.buckets_path两个数据来源：
-
- （1）. “hatsSales”: “hats>hat_sales”
– 父子关系 （hats 父 ；hat_sales 子）
– 指向hat_sales指标值，该值包含在hats聚合桶内
– hats分组聚合的hat_sales指标作为pipeline aggregations的输入源。
- （1）. “hatsSales”: “hats>hat_sales”
-
- （2）.“totalSales”: “total_sales”
-
2.script
“script”: “params.hatsSales / params.totalSales * 100” -
二级子聚合：pepper-percentage，buckets_path两个数据来源：
-
1.buckets_path两个数据来源：
-
- （1）. “pepperSales”: “peppers>pepper_sales”
– 父子关系 （peppers 父 ；pepper_sales 子）
– 指向pepper_sales指标值，该值包含在peppers聚合桶内
– peppers分组聚合的pepper_sales指标作为pipeline aggregations的输入源。
- （1）. “pepperSales”: “peppers>pepper_sales”
-
- （2）. “totalSales”: “total_sales”
-
2.script
“script”: “params.pepperSales / params.totalSales * 100”
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "interval": "month", "format": "yyyy-MM" }, "aggs": { "total_sales": { "sum": { "field": "price" } }, "t-shirts": { "filter": { "term": { "type": "t-shirt" } }, "aggs": { "sales": { "sum": { "field": "price" } } } }, "hats": { "filter": { "term": { "type": "hat" } }, "aggs": { "hat_sales": { "sum": { "field": "price" } } } }, "peppers": { "filter": { "term": { "type": "pepper" } }, "aggs": { "pepper_sales": { "sum": { "field": "price" } } } }, "t-shirt-percentage": { "bucket_script": { "buckets_path": { "tShirtSales": "t-shirts>sales", "totalSales": "total_sales" }, "script": "params.tShirtSales / params.totalSales * 100" } }, "hat-percentage": { "bucket_script": { "buckets_path": { "hatsSales": "hats>hat_sales", "totalSales": "total_sales" }, "script": "params.hatsSales / params.totalSales * 100" } }, "pepper-percentage": { "bucket_script": { "buckets_path": { "pepperSales": "peppers>pepper_sales", "totalSales": "total_sales" }, "script": "params.pepperSales / params.totalSales * 100" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
bucket_script响应结果
- sales_per_month(3个bucket )
2022-04，2022-05，2022-06
项目 Value va total_sales value pepper的sum值 peppers doc_count 文档数量； pepper_sales pepper类型sum值 hats doc_count 文档数量 hat_sales hat类型sum值 t-shirts doc_count 文档数量 sales t-shirts类型sum值 t-shirt-percentage value pepper_sales/total_sales hat-percentage value hat-percentage/total_sales pepper-percentage value pepper-percentage/total_sales { "took" : 3, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "peppers" : { "doc_count" : 0, "pepper_sales" : { "value" : 0.0 } }, "hats" : { "doc_count" : 0, "hat_sales" : { "value" : 0.0 } }, "total_sales" : { "value" : 140.0 }, "t-shirts" : { "doc_count" : 2, "sales" : { "value" : 140.0 } }, "t-shirt-percentage" : { "value" : 100.0 }, "hat-percentage" : { "value" : 0.0 }, "pepper-percentage" : { "value" : 0.0 } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "peppers" : { "doc_count" : 2, "pepper_sales" : { "value" : 30.0 } }, "hats" : { "doc_count" : 0, "hat_sales" : { "value" : 0.0 } }, "total_sales" : { "value" : 30.0 }, "t-shirts" : { "doc_count" : 0, "sales" : { "value" : 0.0 } }, "t-shirt-percentage" : { "value" : 0.0 }, "hat-percentage" : { "value" : 0.0 }, "pepper-percentage" : { "value" : 100.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "peppers" : { "doc_count" : 2, "pepper_sales" : { "value" : 30.0 } }, "hats" : { "doc_count" : 5, "hat_sales" : { "value" : 960.0 } }, "total_sales" : { "value" : 2110.0 }, "t-shirts" : { "doc_count" : 2, "sales" : { "value" : 180.0 } }, "t-shirt-percentage" : { "value" : 8.530805687203792 }, "hat-percentage" : { "value" : 45.497630331753555 }, "pepper-percentage" : { "value" : 1.4218009478672986 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
6. Bucket selector aggregation 桶选择器聚合====================================================
- parent pipeline aggregation , bucket_selector
- 父管道聚合，它执行一个脚本，该脚本确定当前桶是否将保留在父多桶聚合中。指定度量必须是数字，脚本必须返回一个布尔值。如果脚本语言是表达式，则允许数字返回值。在这种情况下，0.0将被判定为false，所有的其他值将被判定为true。
- 注意：bucket_selector聚合，像所有管道聚合，在所有的其他同级聚合后执行。这意味着使用bucket_selector聚合过滤返回的响应数据中不会保存运行时的聚合。
1.buckets_path单个路径
过滤月额度超过200
- “script”: “params.totalSales > 100” 月额度超过100
- “script”: “params.totalSales > 200” 月额度超过200
- “script”: “params.totalSales < 200” 月额度小于200
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "month", "format": "yyyy-MM" }, "aggs": { "total_sales": { "sum": { "field": "price" } }, "sales_bucket_filter":{ "bucket_selector": { "buckets_path": { "totalSales": "total_sales" }, "script": "params.totalSales > 200" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
月额度超过200响应结果
- 过滤了小于200的记录，只显示200以上的
- script 总额度 > 0 ：buckets 3
- script 总额度 > 200 ：buckets 1
{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "total_sales" : { "value" : 140.0 } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "total_sales" : { "value" : 30.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
2.buckets_path多个路径
- “script”: “params.totalSales < params.totalSalesCount”
总额度 < 商品数量 - “script”: “params.totalSales > params.totalSalesCount”
总额度 > 商品数量
POST /sales/_search?size=0
{
“aggs”: {
“sales_per_month”: {
“date_histogram”: {
“field”: “date”,
“calendar_interval”: “month”,
“format”: “yyyy-MM”
},
“aggs”: {
“total_sales”: {
“sum”: {
“field”: “price”
}
},
“the_value_count”:{
“value_count”: {
“field”: “price”
}
},
“sales_bucket_filter”:{
“bucket_selector”: {
“buckets_path”: {
“totalSales”: “total_sales”,
“totalSalesCount”:“the_value_count”
},
“script”: “params.totalSales > params.totalSalesCount”
}
}
}
}
}
}bucket_selector响应结果
- script 总额度 < 商品数量 ：buckets 0
- script 总额度 > 商品数量 ：buckets 3
{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "the_value_count" : { "value" : 2 }, "total_sales" : { "value" : 140.0 } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "the_value_count" : { "value" : 2 }, "total_sales" : { "value" : 30.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "the_value_count" : { "value" : 12 }, "total_sales" : { "value" : 2110.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
7. Bucket sort aggregation====================================================
- parent pipeline aggregation , bucket_sort
- 父管道聚合，对其父多桶聚合的桶进行排序。可以将零个或多个排序字段与相应的排序顺序一起指定。每个 bucket 可以根据其_key、_count 或其子聚合进行排序。
- 此外，可以设置from和size的参数，以截断结果存储桶。
1.单字段排序
- { “total_sales”:{“order”:“desc”}} buckets返回的doc的排序desc
- { “total_sales”:{“order”:“asc”}} buckets返回的doc的排序asc
- “size”: 3 buckets返回的数量
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "month", "format": "yyyy-MM" }, "aggs": { "total_sales": { "sum": { "field": "price" } }, "sales_bucket_sort":{ "bucket_sort": { "sort": [ { "total_sales":{"order":"asc"}} ], "size": 3 } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
bucket_sort响应结果：
- sales_per_month： buckets 3个
- “order”:“desc” [2022-06,2022-04,2022-05]
- “order”:“asc” [2022-05,2022-04,2022-06]
{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "total_sales" : { "value" : 30.0 } }, { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "total_sales" : { "value" : 140.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "total_sales" : { "value" : 2110.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
2.多字段排序
- buckets返回的doc的排序desc
- _count排序desc
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "interval": "week", "format": "yyyy-MM-dd" }, "aggs": { "total_sales": { "sum": { "field": "price" } }, "sales_bucket_sort":{ "bucket_sort": { "sort": [ { "total_sales":{"order":"desc"}}, { "_count":{"order":"desc"}} ], "size": 3 } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
bucket_sort响应结果：
{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-06-06", "key" : 1654473600000, "doc_count" : 10, "total_sales" : { "value" : 1550.0 } }, { "key_as_string" : "2022-05-30", "key" : 1653868800000, "doc_count" : 4, "total_sales" : { "value" : 590.0 } }, { "key_as_string" : "2022-03-28", "key" : 1648425600000, "doc_count" : 2, "total_sales" : { "value" : 140.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
8. Cumulative cardinality aggregation====================================================
- parent pipeline aggregation ,cardinality + cumulative_cardinality
计算每月总计的累计基数types
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "month", "format": "yyyy-MM" }, "aggs": { "distinct_sales_type": { "cardinality": { "field": "type" } }, "total_new_sales_type": { "cumulative_cardinality": { "buckets_path": "distinct_sales_type" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
cumulative_cardinality响应结果
bucket Value Value Value 2022-04 doc:2; type:t-shirt; cardinality:1; 2022-05 doc:2; type:pepper; cardinality:1; 2022-05 doc:12; type:[pepper,t-shirt,hat,bag]; cardinality:4; { "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "distinct_sales_type" : { "value" : 1 }, "total_new_sales_type" : { "value" : 1 } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "distinct_sales_type" : { "value" : 1 }, "total_new_sales_type" : { "value" : 2 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "distinct_sales_type" : { "value" : 4 }, "total_new_sales_type" : { "value" : 4 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
9. Cumulative sum aggregation 累积汇总聚合 ====================================================
- parent pipeline aggregation , cumulative_sum
- 父级管道聚合，计算父级直方图（或日期直方图）聚合中指定度量的累积和。指定的度量必须是数字，并且闭合min_doc_count直方图必须设置为0（默认为histogram聚合）。
累计三个月的总和
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "month", "format": "yyyy-MM" }, "aggs": { "sales": { "sum": { "field": "price" } }, "cumulative_sales": { "cumulative_sum": { "buckets_path": "sales" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
cumulative_sales响应结果
bucket Value Value 2022-04 sales：140.0 cumulative_sales：140.0+0 2022-05 sales：30.0 cumulative_sales：30+140.0 2022-06 sales：2110.0 cumulative_sales：2110.0+30+140.0 { "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "sales" : { "value" : 140.0 }, "cumulative_sales" : { "value" : 140.0 } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "sales" : { "value" : 30.0 }, "cumulative_sales" : { "value" : 170.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "sales" : { "value" : 2110.0 }, "cumulative_sales" : { "value" : 2280.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
10. Derivative aggregation====================================================
- parent pipeline aggregation , derivative
- 导数管道聚合，其计算父直方图（或日期 - 图形）聚合中指定度量的导数。指定的度量必须是数字，并且必须设置直方图min_doc_count为0（默认为直方图聚合）。
求导
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "month", "format": "yyyy-MM" }, "aggs": { "sales": { "sum": { "field": "price" } }, "sales_deriv":{ "derivative": { "buckets_path": "sales" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
derivative响应结果
{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "sales" : { "value" : 140.0 } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "sales" : { "value" : 30.0 }, "sales_deriv" : { "value" : -110.0 } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "sales" : { "value" : 2110.0 }, "sales_deriv" : { "value" : 2080.0 } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
11.Normalize aggregation====================================================
- parent pipeline aggregation , normalize
- 父管道聚合，计算特定存储段值的特定规范化/重缩放值。将使用跳空政策 .
- method参数
Methods [5, 5, 10, 50, 10, 20]表示 Value rescale_0_1 [0, 0, .1111, 1, .1111, .3333] x’ = (x - min_x) / (max_x - min_x) escale_0_100 [0, 0, 11.11, 100, 11.11, 33.33] x’ = 100 * (x - min_x) / (max_x - min_x) percent_of_sum [5%, 5%, 10%, 50%, 10%, 20%] x’ = x / sum_x mean [4.63, 4.63, 9.63, 49.63, 9.63, 9.63, 19.63] x’ = (x - mean_x) / (max_x - min_x) zscore [-0.68, -0.68, -0.39, 1.94, -0.39, 0.19] x’ = (x - mean_x) / stdev_x softmax [2.862E-20, 2.862E-20, 4.248E-18, 0.999, 9.357E-14, 4.248E-18] x’ = e^x / sum_e_x 百分比
- “method”:“percent_of_sum”,
- “format”:“00.00%” 百分制格式
POST /sales/_search?size=0 { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "month", "format": "yyyy-MM" }, "aggs": { "sales": { "sum": { "field": "price" } }, "percent_of_total_sales":{ "normalize":{ "buckets_path": "sales", "method":"percent_of_sum", "format":"00.00%" } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
normalize+percent_of_sum响应结果
bucket Value 2022-04 06.14% 2022-05 01.32% 2022-06 92.54% { "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 16, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "sales_per_month" : { "buckets" : [ { "key_as_string" : "2022-04", "key" : 1648771200000, "doc_count" : 2, "sales" : { "value" : 140.0 }, "percent_of_total_sales" : { "value" : 0.06140350877192982, "value_as_string" : "06.14%" } }, { "key_as_string" : "2022-05", "key" : 1651363200000, "doc_count" : 2, "sales" : { "value" : 30.0 }, "percent_of_total_sales" : { "value" : 0.013157894736842105, "value_as_string" : "01.32%" } }, { "key_as_string" : "2022-06", "key" : 1654041600000, "doc_count" : 12, "sales" : { "value" : 2110.0 }, "percent_of_total_sales" : { "value" : 0.9254385964912281, "value_as_string" : "92.54%" } } ] } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
12. Inference bucket aggregation====================================================
- parent pipeline aggregation
GET kibana_sample_data_logs/_search { "size": 0, "aggs": { "client_ip": { "composite": { "sources": [ { "client_ip": { "terms": { "field": "clientip" } } } ] }, "aggs": { "url_dc": { "cardinality": { "field": "url.keyword" } }, "bytes_sum": { "sum": { "field": "bytes" } }, "geo_src_dc": { "cardinality": { "field": "geo.src" } }, "geo_dest_dc": { "cardinality": { "field": "geo.dest" } }, "responses_total": { "value_count": { "field": "timestamp" } }, "success": { "filter": { "term": { "response": "200" } } }, "error404": { "filter": { "term": { "response": "404" } } }, "error503": { "filter": { "term": { "response": "503" } } }, "malicious_client_ip": { "inference": { "model_id": "malicious_clients_model", "buckets_path": { "response_count": "responses_total", "url_dc": "url_dc", "bytes_sum": "bytes_sum", "geo_src_dc": "geo_src_dc", "geo_dest_dc": "geo_dest_dc", "success": "success._count", "error404": "error404._count", "error503": "error503._count" } } } } } } }- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
数据集
DELETE sales PUT sales { "mappings": { "properties": { "name": { "type": "keyword" }, "type": { "type": "keyword" }, "price": { "type": "integer" }, "country": { "type": "keyword" }, "tags": { "type": "keyword" }, "date": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis", "fields": { "keyword": { "type": "keyword", "ignore_above": 20 } } } } } } POST sales/_bulk {"index":{"_id":1}} {"name":"yellow pepper","type":"pepper","price":20,"date":"2022-06-08 21:41:27","timestamp":1654695687000,"promoted":0,"country":"CN","tags":["vegetable","pepper","fresh","kick","yellow"]} {"index":{"_id":2}} {"name":"red pepper","type":"pepper","price":10,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"EN","tags":["vegetable","pepper","fresh","hot","red"]} {"index":{"_id":3}} {"name":"green pepper","type":"pepper","price":20,"date":"2022-05-31 21:41:27","timestamp":1654004487000,"promoted":1,"country":"TH","tags":["vegetable","pepper","fresh","fiery","green"]} {"index":{"_id":4}} {"name":"bule pepper","type":"pepper","price":10,"date":"2022-05-31 21:41:27","timestamp":1654004487000,"promoted":1,"country":"CN","tags":["vegetable","pepper","pecial discount","hot","blue"]} {"index":{"_id":5}} {"name":"balck t-shirt","type":"t-shirt","price":60,"date":"2022-04-01 12:41:27","timestamp":1648788087000,"promoted":1,"country":"TH","tags":["clothes","t-shirt","grandad shirt","balck","hot","sport","cotton"]} {"index":{"_id":6}} {"name":"green t-shirt","type":"t-shirt","price":80,"date":"2022-04-01 12:41:27","timestamp":1648788087000,"promoted":1,"country":"CN","tags":["clothes","t-shirt","green","v-neck","blouse","pure cotton"]} {"index":{"_id":7}} {"name":"blue t-shirt","type":"t-shirt","price":100,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"EN","tags":["clothes","t-shirt","blue","navy collar","100% cotton"]} {"index":{"_id":8}} {"name":"red t-shirt","type":"t-shirt","price":80,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":0,"country":"CN","tags":["clothes","t-shirt","red","sweetheart (heart shaped) neckline","pure cotton"]} {"index":{"_id":9}} {"name":"balck hat","type":"hat","price":260,"date":"2022-06-02 12:41:27","timestamp":1654144887000,"promoted":1,"country":"CN","tags":["clothes","hat","balck","streetwear","sunscreen","Bucket hat","Screen Printing","polyester 190t","Kaitlyn Bristow"]} {"index":{"_id":10}} {"name":"red hat","type":"hat","price":200,"date":"2022-06-08 11:41:27","timestamp":1654659687000,"promoted":1,"country":"EN","tags":["clothes","hat","red","streetwear","sunscreen","straw hat","90% straw"]} {"index":{"_id":11}} {"name":"white hat","type":"hat","price":180,"date":"2022-06-10 11:41:27","timestamp":1654832487000,"promoted":1,"country":"CN","tags":["clothes","hat","white","streetwear","sunscreen","cowboy cap","felt","Bollman Hat","80% wool,20% N"]} {"index":{"_id":12}} {"name":"balck hat","type":"hat","price":160,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"TH","tags":["clothes","hat","balck","streetwear","Bucket hat","Baseball cap","hot","sport","30% cotton"]} {"index":{"_id":13}} {"name":"blue hat","type":"hat","price":160,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"TH"} {"index":{"_id":14}} {"name":"red bag","type":"bag","price":300,"date":"2022-06-02 15:41:27","timestamp":1654155687000,"promoted":1,"country":"EN","tags":["bag","red","sport","90% straw"]} {"index":{"_id":15}} {"name":"white bag","type":"bag","price":380,"date":"2022-06-10 11:41:27","timestamp":1654832487000,"promoted":1,"country":"CN","tags":["bag","white","sport","80% wool,20% N"]} {"index":{"_id":16}} {"name":"balck bag","type":"bag","price":260,"date":"2022-06-12 21:41:27","timestamp":1655041287000,"promoted":1,"country":"TH","tags":["clothes","bag","balck","hot","sport","30% cotton"]}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
-
-
相关阅读:
xindong科技面试回顾
ElementUI浅尝辄止35：Checkbox 多选框
Redis 学习笔记 3：黑马点评
Android13 按键kl文件优先级详解
qt波位图
RocketMQ 消息重复消费
JdbcTemplate的基本使用
mysql数据库连接后缀的作用
vue3 el-table表格数据过多加载卡死优化问题
Xmake v2.7.1 发布，更好的 C++ Modules 支持
- 原文地址：https://blog.csdn.net/tsj11514oo/article/details/125494926
