

在三种链路监控组件中,skywalking的探针对吞吐量的影响最小,zipkin的吞吐量居中。pinpoint的探针对吞吐量的影响较为明显
skywalking是一个国产开源框架,2015年由吴晟开源 , 2017年加入Apache孵化器。skywalking是分布式系统的应用程序性能监视工具,专为微服务、云原生架构和基于容器(Docker、K8s、Mesos)架构而设计。SkyWalking 是观察性分析平台和应用性能管理系统,提供分布式追踪、服务网格遥测分析、度量聚合和可视化一体化解决方案。
下载:http://skywalking.apache.org/downloads/
Github:https://github.com/apache/skywalking
文档:https://skywalking.apache.org/docs/main/v9.1.0/readme/
中文文档: https://skyapm.github.io/document-cn-translation-of-skywalking/
版本: v9.2.0
采集数据——》传输数据——》存储数据——》分析数据——》监控报警

接受探针发送过来的数据,进行度量分析,调用链分析和存储。后端主要分为两部分:



SkyWalking APM
Java Agent
1)先使用默认的H2数据库存储,不用修改配置
config/application.yml
2)启动脚本bin/startup.sh
➜ apache-skywalking-apm-bin ./bin/startup.sh
SkyWalking OAP started successfully!
SkyWalking Web Application started successfully!
日志信息存储在logs目录
启动成功后会启动 skywalking-oap-server 和 skywalking-web-ui
skywalking-oap-server服务启动后会暴露端口,修改端口可以修改config/applicaiton.yml
skywalking-web-ui服务会占用 8080 端口, 修改端口可以修改webapp/webapp.yml

#!/bin/sh
# SkyWalking Agent配置
export SW_AGENT_NAME=springboot-skywalking-demo #Agent名字,一般使用`spring.application.name`
export SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800 #配置 Collector 地址。
export SW_AGENT_SPAN_LIMIT=2000 #配置链路的最大Span数量,默认为 300。
export JAVA_AGENT=-javaagent:/root/skywalking-agent/skywalking-agent.jar
java $JAVA_AGENT -jar springboot-skywalking-demo-0.0.1-SNAPSHOT.jar #jar启动
java -javaagent:/root/skywalking-agent/skywalking-agent.jar
-DSW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800
-DSW_AGENT_NAME=springboot-skywalking-demo -jar springboot-skywalking-demo-0.0.1-SNAPSHOT.jar
-javaagent:/root/skywalking-agent/skywalking-agent.jar
-Dskywalking.agent.service_name=springboot-skywalking-demo
-Dskywalking.collector.backend_service=127.0.0.1:11800
使用skywalking.+配置文件中的配置名作为系统配置项来进行覆盖。 javaagent参数配置方式优先级更高
-javaagent:/root/skywalking-agent/skywalking-agent.jar
-DSW_AGENT_NAME=springboot-skywalking-demo
-DSW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800
解决方案:拷贝agent/optional-plugins目录下的gateway插件和webflux插件到agent/plugins目录


引入依赖
<dependency>
<groupId>org.apache.skywalkinggroupId>
<artifactId>apm-toolkit-logback-1.xartifactId>
<version>8.11.0version>
dependency>
微服务添加logback-spring.xml文件,并配置 %tid 占位符
<configuration>
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%tid] [%thread] %-5level %logger{36} -%msg%nPattern>
layout>
encoder>
appender>
<appender name="grpc-log" class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.log.GRPCLogClientAppender">
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="org.apache.skywalking.apm.toolkit.log.logback.v1.x.TraceIdPatternLogbackLayout">
<Pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%tid] [%thread] %-5level %logger{36} -%msg%nPattern>
layout>
encoder>
appender>
<root level="INFO">
<appender-ref ref="console"/>
<appender-ref ref="grpc-log"/>
root>
configuration>
Skywalking通过grpc上报日志 (需要v8.4.0以上)
gRPC报告程序可以将收集到的日志转发到SkyWalking OAP服务器上

兼容ELK 配置 Logstash

skywalking告警的核心由一组规则驱动,这些规则定义在config/alarm-settings.yml文件中,告警规则的定义分为三部分:
为了方便,skywalking发行版中提供了默认的alarm-setting.yml文件,包括一些规则,每个规则有英文注释,可以根据注释得知每个规则的作用:
只要我们的服务请求符合alarm-setting.yml文件中的某一条规则就会触发告警。
比如service_resp_time_rule规则:
该规则表示服务{name}的响应时间在最近10分钟的3分钟内超过1000ms

demo

@RequestMapping("/notify")
public String notify(@RequestBody Object obj) {
//TODO 告警信息,给技术负责人发短信,钉钉消息,邮件,微信通知等
System.err.println(obj.toString());
return "notify successfully";
}



storage:
#选择使用mysql 默认使用h2,不会持久化,重启skyWalking之前的数据会丢失
selector: ${SW_STORAGE:mysql}
#使用mysql作为持久化存储的仓库
mysql:
properties:
#数据库连接地址 创建swtest数据库
jdbcUrl: ${SW_JDBC_URL:"jdbc:mysql://1ocalhost:3306/swtest"}
#用户名
dataSource.user: ${SW_DATA_SOURCE_USER:root}
#密码
dataSource.password: ${SW_DATA_SOURCE_PASSWORD:root}
➜ apache-skywalking-apm-bin ls oap-libs/mysql-connector-java-8.0.19.jar
oap-libs/mysql-connector-java-8.0.19.jar
注意:需要添加mysql数据驱动包,因为在lib目录下是没有mysql数据驱动包的,所以修改完配置启动是会报错,启动失败的。
4. 启动Skywalking 查看swtest库生成的表


<dependency>
<groupId>org.apache.skywalkinggroupId>
<artifactId>apm-toolkit-traceartifactId>
<version>8.11.0version>
dependency>
@RequestMapping("/list")
public List<User> list(){
//TraceContext可以绑定key-value
TraceContext.putCorrelation("name", "fox");
Optional<String> op = TraceContext.getCorrelation("name");
log.info("name = {} ", op.get());
//获取追踪的traceId
String traceId = TraceContext.traceId();
log.info("traceId = {} ", traceId);
return userService.list();
}
如果一个业务方法想在ui界面的追踪链路上显示出来,只需要在业务方法上加上@Trace注解即可
加入@Tags或@Tag为追踪链路增加其他额外的信息,比如记录参数和返回信息
@Service
public class UserServiceImpl implements UserService {
@Autowired
private UserMapper userMapper;
@Trace
@Tag(key = "list", value = "returnedObj")
@Override
public List<User> list() {
return userMapper.list();
}
@Trace
@Tags({@Tag(key = "param", value = "arg[0]"),
@Tag(key = "user", value = "returnedObj")})
@Override
public User getById(Integer id) {
return userMapper.getById(id);
}
}

Skywalking集群是将skywalking oap作为一个服务注册到nacos上,只要skywalking oap服务没有全部宕机,保证有一个skywalking oap在运行,就能进行追踪。
搭建一个skywalking oap集群需要:
(1)至少一个Nacos(也可以是nacos集群)
(2)至少一个ElasticSearch(也可以是es集群)
(3)至少2个skywalking oap服务;
(4)至少1个UI(UI也可以集群多个,用Nginx代理统一入口)
cluster:
selector: ${SW_CLUSTER:nacos}
nacos:
serviceName: ${SW_SERVICE_NAME:"SkyWalking_OAP_Cluster"}
hostPort: ${SW_CLUSTER_NACOS_HOST_PORT:localhost:8848}
# Nacos Configuration namespace
namespace: ${SW_CLUSTER_NACOS_NAMESPACE:"public"}
# Nacos auth username
username: ${SW_CLUSTER_NACOS_USERNAME:""}
password: ${SW_CLUSTER_NACOS_PASSWORD:""}
# Nacos auth accessKey
accessKey: ${SW_CLUSTER_NACOS_ACCESSKEY:""}
secretKey: ${SW_CLUSTER_NACOS_SECRETKEY:""}
internalComHost: ${SW_CLUSTER_INTERNAL_COM_HOST:""}
internalComPort: ${SW_CLUSTER_INTERNAL_COM_PORT:-1}
-Dskywalking.collector.backend_service=ip1:11800,ip2:11800-javaagent:instrument-demo/target/instrument-demo-1.0-SNAPSHOT.jar
<dependencies>
<dependency>
<groupId>javassistgroupId>
<artifactId>javassistartifactId>
<version>3.12.1.GAversion>
<type>jartype>
dependency>
dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-jar-pluginartifactId>
<version>2.2version>
<configuration>
<archive>
<manifestEntries>
<Project-name>${project.name}Project-name>
<Project-version>${project.version}Project-version>
<Premain-Class>com.demo.TestAgentPremain-Class>
manifestEntries>
archive>
<skip>trueskip>
configuration>
plugin>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-shade-pluginartifactId>
<executions>
<execution>
<phase>packagephase>
<goals>
<goal>shadegoal>
goals>
execution>
executions>
<configuration>
<artifactSet>
<includes>
<include>javassist:javassist:jar:include>
includes>
artifactSet>
configuration>
plugin>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-compiler-pluginartifactId>
<configuration>
<source>6source>
<target>6target>
configuration>
plugin>
plugins>
build>
FoxAgent.java
public class TestAgent {
public static void premain(String args, Instrumentation instrumentation){
System.out.println("premain:获取方法调用时间");
// 添加 Transformer
ClassFileTransformer transformer = new PerformMonitorTransformer();
instrumentation.addTransformer(transformer);
}
}
PerformMonitorTransformer.java
public class PerformMonitorTransformer implements ClassFileTransformer {
private static final Set<String> classNameSet = new HashSet<String>();
static {
// 直接定位
classNameSet.add("com.demo.HelloService");
// 间接定位 注解 继承关系
}
@Override
public byte[] transform(ClassLoader loader,
String className,
Class<?> classBeingRedefined,
ProtectionDomain protectionDomain,
byte[] classfileBuffer) throws IllegalClassFormatException {
try {
String currentClassName = className.replaceAll("/", ".");
// 只增强Set中含有的类 过滤
if (!classNameSet.contains(currentClassName)) {
return null;
}
System.out.println("transform: [" + currentClassName + "]");
CtClass ctClass = ClassPool.getDefault().get(currentClassName);
CtBehavior[] methods = ctClass.getDeclaredBehaviors();
for (CtBehavior method : methods) {
//增强方法
enhanceMethod(method);
}
return ctClass.toBytecode();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
private void enhanceMethod(CtBehavior method) throws Exception {
if (method.isEmpty()) {
return;
}
String methodName = method.getName();
// 不增强main方法
if (methodName.equalsIgnoreCase("main")) {
return;
}
// 增强@Test修饰的方法
if(null == method.getAnnotation(Test.class)){
return;
}
final StringBuilder source = new StringBuilder();
source.append("{")
// 前置增强: 加入时间戳
.append("long start = System.currentTimeMillis();\n")
// 保留原有的代码处理逻辑
.append("$_ = $proceed($$);\n")
.append("System.out.print(\"method:[" + methodName + "]\");")
.append("\n")
// 后置增强
.append("System.out.println(\" cost:[\" +(System.currentTimeMillis() -start)+ \"ms]\");")
.append("}");
ExprEditor editor = new ExprEditor() {
@Override
public void edit(MethodCall methodCall) throws CannotCompileException {
methodCall.replace(source.toString());
}
};
method.instrument(editor);
}
}
<dependencies>
<dependency>
<groupId>javassistgroupId>
<artifactId>javassistartifactId>
<version>3.12.1.GAversion>
<type>jartype>
dependency>
<dependency>
<groupId>net.bytebuddygroupId>
<artifactId>byte-buddyartifactId>
<version>1.5.13version>
dependency>
<dependency>
<groupId>net.bytebuddygroupId>
<artifactId>byte-buddy-agentartifactId>
<version>1.5.13version>
dependency>
dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-jar-pluginartifactId>
<version>2.2version>
<configuration>
<archive>
<manifestEntries>
<Project-name>${project.name}Project-name>
<Project-version>${project.version}Project-version>
<Premain-Class>com.demo.TestAgentPremain-Class>
manifestEntries>
archive>
<skip>trueskip>
configuration>
plugin>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-shade-pluginartifactId>
<executions>
<execution>
<phase>packagephase>
<goals>
<goal>shadegoal>
goals>
execution>
executions>
<configuration>
<artifactSet>
<includes>
<include>javassist:javassist:jar:include>
<include>net.bytebuddy:byte-buddy:jar:include>
<include>net.bytebuddy:byte-buddy-agent:jar:include>
includes>
artifactSet>
configuration>
plugin>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-compiler-pluginartifactId>
<configuration>
<source>6source>
<target>6target>
configuration>
plugin>
plugins>
build>
public class TestAgent {
public static void premain(String agentArgs, Instrumentation inst) {
System.out.println("premain:获取方法调用时间");
AgentBuilder.Transformer transformer = new AgentBuilder.Transformer() {
@Override
public DynamicType.Builder<?> transform(DynamicType.Builder<?> builder,
TypeDescription typeDescription,
ClassLoader classLoader) {
return builder
// 拦截任意方法
.method(ElementMatchers.<MethodDescription>any())
// 指定方法拦截器,此拦截器中做具体的操作
.intercept(MethodDelegation.to(TimeInterceptor.class));
}
};
AgentBuilder.Listener listener = new AgentBuilder.Listener() {
@Override
public void onTransformation(TypeDescription typeDescription, ClassLoader classLoader, JavaModule module, DynamicType dynamicType) {}
@Override
public void onIgnored(TypeDescription typeDescription, ClassLoader classLoader, JavaModule module) { }
@Override
public void onError(String typeName, ClassLoader classLoader, JavaModule module, Throwable throwable) { }
@Override
public void onComplete(String typeName, ClassLoader classLoader, JavaModule module) { }
};
new AgentBuilder
.Default()
// 指定需要拦截的类
.type(ElementMatchers.nameStartsWith("com.demo"))
.transform(transformer)
.with(listener)
.installOn(inst);
}
}
TimeInterceptor
public class TimeInterceptor {
@RuntimeType
public static Object intercept(@Origin Method method,
@SuperCall Callable<?> callable) throws Exception {
long start = System.currentTimeMillis();
try {
// 原方法执行
return callable.call();
} finally {
System.out.println(method + ": cost " + (System.currentTimeMillis() - start) + "ms");
}
}
}
<dependencies>
<dependency>
<groupId>javassistgroupId>
<artifactId>javassistartifactId>
<version>3.12.1.GAversion>
<type>jartype>
dependency>
dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-jar-pluginartifactId>
<version>2.2version>
<configuration>
<archive>
<manifestEntries>
<Project-name>${project.name}Project-name>
<Project-version>${project.version}Project-version>
<Premain-Class>com.demo.TestAgentPremain-Class>
manifestEntries>
archive>
<skip>trueskip>
configuration>
plugin>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-shade-pluginartifactId>
<executions>
<execution>
<phase>packagephase>
<goals>
<goal>shadegoal>
goals>
execution>
executions>
<configuration>
<artifactSet>
<includes>
<include>javassist:javassist:jar:include>
includes>
artifactSet>
configuration>
plugin>
<plugin>
<groupId>org.apache.maven.pluginsgroupId>
<artifactId>maven-compiler-pluginartifactId>
<configuration>
<source>6source>
<target>6target>
configuration>
plugin>
plugins>
build>
public class TestAgent {
public static void premain(String agentArgs, Instrumentation inst) {
//每隔5秒打印JVM内存和GC信息
Executors.newScheduledThreadPool(1).scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
Metric.printMemoryInfo();
Metric.printGCInfo();
}
}, 0, 5000, TimeUnit.MILLISECONDS);
}
}
Metric
public class Metric {
private static final long MB = 1048576L;
public static void printMemoryInfo() {
MemoryMXBean memory = ManagementFactory.getMemoryMXBean();
MemoryUsage headMemory = memory.getHeapMemoryUsage();
String info = String.format("\nHeapMemory init: %s\t max: %s\t used: %s\t committed: %s\t use rate: %s\n",
headMemory.getInit() / MB + "MB",
headMemory.getMax() / MB + "MB", headMemory.getUsed() / MB + "MB",
headMemory.getCommitted() / MB + "MB",
headMemory.getUsed() * 100 / headMemory.getCommitted() + "%"
);
System.out.print(info);
MemoryUsage nonheadMemory = memory.getNonHeapMemoryUsage();
info = String.format("NonHeapMemory init: %s\t max: %s\t used: %s\t committed: %s\t use rate: %s\n",
nonheadMemory.getInit() / MB + "MB",
nonheadMemory.getMax() / MB + "MB", nonheadMemory.getUsed() / MB + "MB",
nonheadMemory.getCommitted() / MB + "MB",
nonheadMemory.getUsed() * 100 / nonheadMemory.getCommitted() + "%"
);
System.out.println(info);
}
public static void printGCInfo() {
List<GarbageCollectorMXBean> garbages = ManagementFactory.getGarbageCollectorMXBeans();
for (GarbageCollectorMXBean garbage : garbages) {
String info = String.format("name: %s\t count:%s\t took:%s\t pool name:%s",
garbage.getName(),
garbage.getCollectionCount(),
garbage.getCollectionTime(),
Arrays.deepToString(garbage.getMemoryPoolNames()));
System.out.println(info);
}
}
}
