Ease Monitor

• 1. Product Principles
• 2. Design Principles
• 3. Architecture
• 4. Prerequisite and Limitation
• 5. Features Showcase
  • 5.1 Overview Dashboard
  • 5.2 Service SLA Daily Report
  • 5.3 Service Dashboard
  • 5.4 Tracing
  • 5.5 Top N List
  • 5.6 Customized Dashboard
• 6. Technical Details
  • 6.1 EaseAgent
    • 6.1.1 Design Principles
    • 6.1.2 Compatibility and Requirement
    • 6.1.3 Data Collection
    • 6.1.4 Usage
  • 6.2 Event
  • 6.3 Data Store Schema
    • 6.3.1 Indices Schema
    • 6.3.2 Document Types Schema

1. Product Principles

The Ease Monitor is designed by the following principles:

• Focus on the SLA. Monitoring the API which consumed by end-user.

• Metrics Aggregation. Connecting the infrastructure and middleware metrics with application.

• Quick Fault Location. The failure always happens, quickly address and recovery the fault is the key.

In other words, the Ease Monitor is designed for two major scenarios:

1. Health Checking.

   • Capacity Management. By comparing the data trend, it can help engineering team decides whether add more resource.

   • Performance Management. Managing the application stack performance, make sure every pieces of stack works fine on production.

2. Diagnosis.

   • Locating the Failure. Once the failure or exception happens, it helps developer find the root location of failure quickly.

   • Performance Analysis. It helps find the software bottle neck and hot spot which developer can dive into the code.

The following is a case usually happens. A slow SQL or Java Full GC could cause the the whole site running very very slow.

2. Design Principles

The Ease Monitor is a kind of APM - Application Performance Management, but it’s a bit different with the traditional APM software.

There are two aspects impacts the design of the Ease Monitor:

• Different Engineering Angle. We know, there are several engineers role in a company, and they looking at the whole system from different angles. For example,

  • Manager likes the whole system’s health.
  • Software Developer likes to know the application’s running status.
  • Operation Team likes to know the infrastructure and the middleware’s running status.

• No reinvent the wheels. Developing a monitor system looks like reinvent another wheel. So, we won’t want to reinvent everything, and need make use the Ease Monitor is opening and flexible enough to be compatible with the current mainstream monitoring technologies.

So, the Ease Monitor had the following design principles:

• Using Mainstream Technology. Most of engineering teams in this world can operate and maintain it.

• Every Components can be Replaced or Tailored. People has different requirement and business, so, the design must give the enough flexibility that anyone can modify it easily.

• Tracing the Services Requests. The monitor must trace request crossing the distributed system from end to end.

• Guiding the Engineering. The monitor must can guide the engineers at least two things, 1) Easily address the issues, 2) Easily make the engineering decision.

• Leverage the Automation. The monitor could connect with other control system to do the automated operations, such as: auto-scaling, auto-scheduling, etc.

• The Whole Stack Metrics Monitoring. We must monitoring the three layers softwares:

  • Service Layer. The key index of service, such as, HTTP request, Status code, Throughput, Latency …, etc.
  • Platform Layer. The key index of a number of middleware, such as, Nginx, Redis, Tomcat, Kafka, MySQL…, etc.
  • Infrastructure Layer. The key index of operation system, such as, CPU, Memory, Disk, Network etc.

• Customized Dashboard. The dashboard can be configured by everyone who have different interests.

3. Architecture

The whole architecture based on the open source technology.

• Agents
  • Metrics Collection - Telegraf
  • Logs Collection - Filebeat or Fluentd
  • Java Agent - EaseAgent (This is developed by us & open source with Apache 2.0 license, See the Technical Details section)
• Backend Pipeline
  • Data Bus - Apache Kafka
  • Data ETL - Logstash
  • Data Store - ElasticSearch
  • Event Data - InfluxDB
  • Event Trigger & Handler - This is developed by us. (See the Technical Details section)
  • UI Portal - This is developed by us. (See the Technical Details section)

The whole architecture not only can monitor big clusters, but also every components can be flexible to replaced or tailored.

4. Prerequisite and Limitation

Currently, Ease Monitor only supports

• Java Language Service, and the version need be >= 1.6
• Linux Operation System

5. Features Showcase

5.1 Overview Dashboard

The Overview dashboard shows the overall health and capacity.

5.2 Service SLA Daily Report

The following diagram shows the daily SLA report, it could be the whole site or individual service.

5.3 Service Dashboard

The following Service Dashboard put the service traffic, the upstream and downstream services, TOP API，Top 5 slowest tracing request，and the related the resource and the metrics.

5.4 Tracing

The real-time topology could let us understand the architecture of the services.

The Tracing could let us understand the chains of the services call and its performance.

5.5 Top N List

Top N lists show the operations or APIs consumed the time most.

Service Top API List

JDBC Top Operation List

5.6 Customized Dashboard

The customized dashboard.

Infrastructure Dashboard

6. Technical Details

6.1 EaseAgent

• EaseAgent is a Java agent for APM(Application Performance Management) system.
• EaseAgent majorly focuses on the Spring Boot development environments.
• EaseAgent is compatible with mainstream monitoring ecosystems, such as Kafka, ElasticSearch, Prometheus, Zipkin, etc.
• EaseAgent collects the basic metrics and the service tracking logs, which is very helpful for performance analysis and troubleshooting.

6.1.1 Design Principles

• Design for Micro-Service architecture, collecting the data from a service perspective.
• Instrumenting a Java application in a non-intrusive way.
• Lightweight and very low CPU, memory, and I/O resource usage.
• Safe to Java application/service.

6.1.2 Compatibility and Requirement

• Collecting Metric & Tracing Logs.
  • JDBC 4.0
  • HTTP Servlet、HTTP Filter
  • Spring Boot 2.2.x: WebClient 、 RestTemplate、FeignClient
  • RabbitMQ Client 5.x、 Kafka Client 2.4.x
  • Jedis 3.5.x、 Lettuce 5.3.x
• Collecting Access Logs.
  • HTTP Servlet、HTTP Filter
  • Spring Cloud Gateway
• Instrumenting the traceId and spanId automatically
• Supplying the health check endpoint
• Supplying the readiness check endpoint for SpringBoot2.2.x

6.1.3 Data Collection

1. HTTP Requests Metrics.
2. JDBC Connection and Statement Metrics, and related context information (such as, URL, SQL statement, etc.)
3. Compatible Zipkin protocol to trace the distributed services. which includes:
   • HTTP receive and send
   • SQL execution
   • Redis access

6.1.4 Usage

Downloads easeagent.jar from release , and just simply add the follow arguments for Java application running:

-javaagent=easeagent.jar

6.2 Event

Currently, the Ease Monitor event handling would deal with the following cases.

• Metric - Duration - Threshold. A metric keeps exceeding the threshold in certain duration. (e.g. cpu > 80% lasts 2mins)

• Metric - Duration - Percentile - Threshold. A metric’s percentile(e.g. P99) exceed the threshold in certain duration. (e.g. response time P90 > 300ms lasts 2mins)

• Metric - Duration - Function - Threshold. Support some simple functions - Sum/Average/Min/Max/Count to trigger the event.

• Logs - Duration - Keywords - Times. Monitor a certain keyword(support the regular expression), if the keyword matched the configured times, then report the event.

6.3 Data Store Schema

The following data schema is used for ElasticSearch storing.

6.3.1 Indices Schema

6.3.2 Document Types Schema

The Document Types Schema include the following things:

• Index mapping template

  • ease-monitor-metrics-* - for metrics data
  • ease-monitor-aggregate-metrics-* - for java agent metrics data
  • ease-monitor-logs-* - for logs

• Category

  • application - for Java Agent metrics data.
  • platform - for a number of middleware metrics - such as: nginx, redis, tomcat, mysql, kafka … etc.
  • infrastructure - for CPU, MEM, DISK, NET metrics.

• Document Type

  • The name of the component.
  • The group of the performance counters and statistics.

For example: