Extension for Redis Monitoring 原理分析

1、概述

Extension for Redis Monitoring 是 AppDynamics machine Agent 的监控扩展插件,可以直接监控Redis的运行状态信息。

2、监控原理

Redis 自身是提供监控相关的指令的,此插件通过 Jedis 连接 redis,发送监控指令获取监控内容。主要使用监控指令为:infoSLOWLOG

3、源码分析

3.1 性能指标

metrics 下是所有的性能指标,metrics > 指令类型 > 项指标分类(如果只有一种类型不需要) > 性能指标。
alias 为性能指标别名,控制台上需要呈现的名称。

插件所有的需要采集的性能指标配置在 config.yml 文件中。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
#This will populate the metrics in all the tiers, under this path(not recommended)
#metricPrefix: "Custom Metrics|Redis"
#The following prefix will populate the metrics under one tier
metricPrefix: "Server|Component:<TIER NAME OR TIER ID>|Custom Metrics|Redis"

#Add your list of Redis servers here.
servers:
- name: "Server1"
host: "localhost"
port: "6379"
password: ""
encryptedPassword: ""

#Encryption key for Encrypted password.
encryptionKey: ""

# Each server instance needs 3 threads, one for the server instance itself, one for info call to the Redis server and one for slowlog call to the server.
# So, please change the value accordingly(Based on the number of server instances you are monitoring).
numberOfThreads: 21

# List of metrics
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#Glossary of terms(These terms are used as properties for each metric):
# alias
# aggregationType
# timeRollUpType
# clusterRollUpType }
# multiplier -->not for derived metrics
# convert --> not for derived metrics
# delta --> not for derived metrics
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
metrics:
Slowlog:
- no_of_new_slow_logs:
alias: "no_of_new_slow_logs"
Info:
Clients: #Information about the client connections
- connected_clients: #Number of client connections (excluding connections from slaves)
alias: "connected_clients"
- blocked_clients: #Number of clients pending on a blocking call(BLPOP, BRPOP, BRPOPLPUSH)
alias: "blocked_clients"

Memory: #Information about the memory consumption
- used_memory: #Total number of bytes allocated by Redis using its allocator (either standard libc, jemalloc, or an alternative allocator such as tcmalloc
alias: "used_memory"
- used_memory_rss: #Number of bytes that Redis allocated as seen by the operating system (a.k.a resident set size). This is the number reported by tools such as top(1) and ps(1)
alias: "used_memory_rss"
- used_memory_peak: #Peak memory consumed by Redis (in bytes)
alias: "used_memory_peak"
- used_memory_lua: #Number of bytes used by the Lua engine
alias: "used_memory_lua"
- mem_fragmentation_ratio: #Ratio between used_memory_rss and used_memory
alias: "mem_fragmentation_ratio"


Persistence: #Information related to RDB and AOF
- rdb_changes_since_last_save: #Number of changes since the last dump
alias: "rdb_changes_since_last_save"
- rdb_last_bgsave_time_sec: #Duration of the last RDB save operation in seconds
alias: "rdb_last_bgsave_time_sec"
- rdb_current_bgsave_time_sec: #Duration of the on-going RDB save operation if any
alias: "rdb_current_bgsave_time_sec"
- aof_last_rewrite_time_sec: #Duration of the last AOF rewrite operation in seconds
alias: "aof_last_rewrite_time_sec"
- aof_current_rewrite_time_sec: #Duration of the on-going AOF rewrite operation if any
alias: "aof_current_rewrite_time_sec"

Stats: #General statistics
- total_connections_received: #Total number of connections accepted by the server
alias: "total_connections_received"
- total_commands_processed: #Total number of commands processed by the server
alias: "total_commands_processed"
- instantaneous_ops_per_sec: #Number of commands processed per second
alias: "instantaneous_ops_per_sec"
- rejected_connections: #Number of connections rejected because of maxclients limit
alias: "rejected_connections"
- expired_keys: #Total number of key expiration events
alias: "expired_keys"
- evicted_keys: #Number of evicted keys due to maxmemory limit
alias: "evicted_keys"
- keyspace_hits: #Number of successful lookup of keys in the main dictionary
alias: "keyspace_hits"
- keyspace_misses: #Number of failed lookup of keys in the main dictionary
alias: "keyspace_misses"
- pubsub_channels: #Global number of pub/sub channels with client subscriptions
alias: "pubsub_channels"
- pubsub_patterns: #Global number of pub/sub pattern with client subscriptions
alias: "pubsub_patterns"
- latest_fork_usec: #Duration of the latest fork operation in microseconds
alias: "latest_fork_usec"

Replication: #Information related to Master/Slave replication
- role:
alias: "role"
convert:
master: "1"
slave: "0"
- connected_slaves: #Number of connected slaves
alias: "connected_slaves"

CPU: #Information related to CPU consumption
- used_cpu_sys: #System CPU consumed by the Redis server
alias: "used_cpu_sys"
- used_cpu_user: #User CPU consumed by the Redis server
alias: "used_cpu_user"
- used_cpu_sys_children: #System CPU consumed by the background processes
alias: "used_cpu_sys_children"
- used_cpu_user_children: #User CPU consumed by the background processes
alias: "used_cpu_user_children"

derivedMetrics:
- derivedMetricPath: "{x}|Stats|keyspace_hit_ratio"
formula: "{x}|Stats|keyspace_hits / ({x}|Stats|keyspace_hits + {x}|Stats|keyspace_misses)"

3.2 核心程序

  • com.appdynamics.extensions.redis.RedisMonitor 是插件的入口方法, machine agent 通过此类加载插件。
1
2
3
4
5
6
7
8
9
10
11
@Override
protected void doRun(TasksExecutionServiceProvider serviceProvider) {
previousTimeStamp = currentTimeStamp;
currentTimeStamp = System.currentTimeMillis();
List<Map<String,String>> servers = (List<Map<String,String>>)configuration.getConfigYml().get("servers");
AssertUtils.assertNotNull(servers, "The 'servers' section in config.yml is not initialised");
for (Map<String, String> server : servers) {
RedisMonitorTask task = new RedisMonitorTask(serviceProvider, server, previousTimeStamp, currentTimeStamp);
serviceProvider.submit(server.get("name"),task);
}
}
  • com.appdynamics.extensions.redis.RedisMonitorTask
    插件定时执行的Task,用于采集性能指标。

  • com.appdynamics.extensions.redis.metrics.SlowLogMetrics

  • com.appdynamics.extensions.redis.metrics.InfoMetrics

分别用于获取 Slow Log Metrics 和 Info Metrics

获取的方式就是 Jedis 调用相关的指令获取。

3.3 Metric 前缀

  • info metric 前缀处理
1
String metricPrefix = configuration.getMetricPrefix() + METRIC_SEPARATOR + server.get("name");

格式为配置文件前缀|配置的Server名称。

参考 InfoMetrics

  • SlowLogMetrics
1
String metricPath = configuration.getMetricPrefix() + METRIC_SEPARATOR + server.get("name") + METRIC_SEPARATOR + "SlowLog" + METRIC_SEPARATOR + "no_of_new_slow_logs";

格式为配置文件前缀|配置的Server名称|SlowLog|no_of_new_slow_logs

参考 SlowLogMetrics

4、插件使用

4.1、监控多个Server

在配置文件中config.yaml 中配置多个Server监控,注意不同的Server使用唯一的名称,其会体现在Metric Path 中 ,可以使用IP-Port方式,也是使用有意义的名称使用。

1
2
3
4
5
6
7
8
9
10
11
12
#Add your list of Redis servers here.
servers:
- name: "Server1"
host: "localhost"
port: "6379"
password: ""
encryptedPassword: ""
- name: "Server2"
host: "localhost"
port: "6380"
password: ""
encryptedPassword: ""

参考《AppDynamics Extension for Redis Monitoring