Spring Cloud Sleuth is a distributed tracing tool for Spring Cloud. It borrows from Zipkin, and Quick Start
Add sleuth to the classpath of a Spring Boot application (see "Adding Sleuth to the Project" for Maven and Gradle examples), and you can see the correlation data being collected in logs, as long as you are logging requests.
For example, consider the following HTTP handler:
@RestController public class DemoController { private static Logger log = LoggerFactory.getLogger(DemoController.class); @RequestMapping("/") public String home() { log.info("Handling home"); ... return "Hello World"; } }
If you add that handler to a controller, you can see the calls to home() being traced in the logs and in Zipkin, if Zipkin is configured.
logging.level.org.springframework.web.servlet.DispatcherServlet=DEBUG.
spring.application.name=myService (for instance) to see the service name as well as the trace and span IDs.
spring.sleuth.sampler.probability
(default: 0.1, which is 10 percent). Otherwise, you might think that Sleuth is not working be cause it omits some spans.
Trace: A set of spans forming a tree-like structure.
For example, if you run a distributed big-data store, a trace might be formed by a PUT request.
Annotation: Used to record the existence of an event in time. With Zipkin to understand who the client and server are, where the request started, and where it ended. For learning purposes, however, we mark these events to highlight what kind of an action took place.
cs: Client Sent. The client has made a request. This annotation indicates the start of the span.
sr: Server Received: The server side got the request and started processing it.
Subtracting the cs timestamp from this timestamp reveals the network latency.
ss: Server Sent. Annotated upon completion of request processing (when the response got sent back to the client).
Subtracting the sr timestamp from this timestamp reveals the time needed by the server side to process the request.
cr> Client Received. Signifies the end of the span.
The client has successfully received the response from the server side.
Subtracting the cs timestamp from this timestamp reveals the whole time needed by the client to receive the response from the server.
The following image shows how Span and Trace look in a system, together with the Zipkin annotations:
Each color of a note signifies a span (there are seven spans - from A to G). Consider the following note:
Trace Id = X Span Id = D Client Sent
This note indicats thatthe current span has Trace Id set to X and Span Id set to D.
Also, the Client Sent event took place.
The following image shows how parent-child relationships of spans look:
The following sections refer to the example shown in the preceding image.
This example has seven spans. If you go to traces in Zipkin, you can see this number in the second trace, as shown in the following image:
However, if you pick a particular trace, you can see four spans, as shown in the following image:
Why is there a difference between the seven and four spans in this case?
Two spans come from the http:/start span. It has the Server Received (sr) and Server Sent (ss) annotations.
Two spans come from the RPC call from service1 to service2 to the http:/foo endpoint.
The Client Sent (cs) and Client Received (cr) events took place on the service1 side.
Server Received (sr) and Server Sent (ss) events took place on the service2 side.
These two spans form one logical span related to an RPC call.
Two spans come from the RPC call from service2 to service3 to the http:/bar endpoint.
The Client Sent (cs) and Client Received (cr) events took place on the service2 side.
The Server Received (sr) and Server Sent (ss) events took place on the service3 side.
These two spans form one logical span related to an RPC call.
Two spans come from the RPC call from service2 to service4 to the http:/baz endpoint.
The Client Sent (cs) and Client Received (cr) events took place on the service2 side.
Server Received (sr) and Server Sent (ss) events took place on the service4 side.
These two spans form one logical span related to an RPC call.
So, if we count the physical spans, we have one from http:/start, two from service1 calling service2, two from service2
calling service3, and two from service2 calling service4. In sum, we have a total of seven spans.
Logically, we see the information of four total Spans because we have one span related to the incoming request
to service1 and three spans related to RPC calls.
Zipkin lets you visualize errors in your trace. When an exception was thrown and was not caught, we set proper tags on the span, which Zipkin can then properly colorize. You could see in the list of traces one trace that is red. That appears because an exception was thrown.
If you click that trace, you see a similar picture, as follows:
If you then click on one of the spans, you see the following
The span shows the reason for the error and the whole stack trace related to it.
Starting with version 2.0.0, Spring Cloud Sleuth uses Live examples
The dependency graph in Zipkin should resemble the following image:
When using grep to read the logs of those four applications by scanning for a trace ID equal to (for example) 2485ec27856c56f4, you get output resembling the following:
service1.log:2016年02月26日 11:15:47.561 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Hello from service1. Calling service2 service2.log:2016年02月26日 11:15:47.710 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Hello from service2. Calling service3 and then service4 service3.log:2016年02月26日 11:15:47.895 INFO [service3,2485ec27856c56f4,1210be13194bfe5,true] 68060 --- [nio-8083-exec-1] i.s.c.sleuth.docs.service3.Application : Hello from service3 service2.log:2016年02月26日 11:15:47.924 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service3 [Hello from service3] service4.log:2016年02月26日 11:15:48.134 INFO [service4,2485ec27856c56f4,1b1845262ffba49d,true] 68061 --- [nio-8084-exec-1] i.s.c.sleuth.docs.service4.Application : Hello from service4 service2.log:2016年02月26日 11:15:48.156 INFO [service2,2485ec27856c56f4,9aa10ee6fbde75fa,true] 68059 --- [nio-8082-exec-1] i.s.c.sleuth.docs.service2.Application : Got response from service4 [Hello from service4] service1.log:2016年02月26日 11:15:48.182 INFO [service1,2485ec27856c56f4,2485ec27856c56f4,true] 68058 --- [nio-8081-exec-1] i.s.c.sleuth.docs.service1.Application : Got response from service2 [Hello from service2, response from service3 [Hello from service3] and from service4 [Hello from service4]]
If you use a log aggregating tool (such as Splunk, and others), you can order the events that took place. An example from Kibana would resemble the following image:
filter { # pattern matching logback pattern grok { match => { "message" => "(?m)OUT\s+%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:severity}\s+\[%{DATA:service},%{DATA:trace},%{DATA:span},%{DATA:exportable}\]\s+%{DATA:pid}\s+---\s+\[%{DATA:thread}\]\s+%{DATA:class}\s+:\s+%{GREEDYDATA:rest}" } } }
Often, you do not want to store your logs in a text file but in a JSON file that Logstash can immediately pick.
To do so, you have to do the following (for readability, we pass the dependencies in the groupId:artifactId:version notation).
Dependencies Setup
Ensure that Logback is on the classpath (ch.qos.logback:logback-core).
Add Logstash Logback encode. For example, to use version 4.6, add net.logstash.logback:logstash-logback-encoder:4.6.
Logback Setup
That Logback configuration file:
Logs information from the application in a JSON format to a build/${spring.application.name}.json file.
Has commented out two additional appenders: console and standard log file.
Has the same logging pattern as the one presented in the previous section.
logback-spring.xml, you must pass the spring.application.name in the bootstrap rather than the application property file.
Otherwise, your custom logback file does not properly read the property.
The span context is the state that must get propagated to any child spans across process boundaries. Part of the Span Context is the Baggage. The trace and span IDs are a required part of the span context. Baggage is an optional part.
Baggage is a set of key:value pairs stored in the span context.
Baggage travels together with the trace and is attached to every span.
Spring Cloud Sleuth understands that a header is baggage-related if the HTTP header is prefixed with baggage- and, for messaging, it starts with baggage_.
The following example shows setting baggage on a span:
Span initialSpan = this.tracer.nextSpan().name("span").start(); try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(initialSpan)) { ExtraFieldPropagation.set("foo", "bar"); ExtraFieldPropagation.set("UPPER_CASE", "someValue"); }
Baggage travels with the trace (every child span contains the baggage of its parent). Zipkin has no knowledge of baggage and does not receive that information.
Tags are attached to a specific span. In other words, they are presented only for that particular span. However, you can search by tag to find the trace, assuming a span having the searched tag value exists.
If you want to be able to lookup a span based on baggage, you should add a corresponding entry as a tag in the root span.
The following listing shows integration tests that use baggage:
initialSpan.tag("foo", ExtraFieldPropagation.get(initialSpan.context(), "foo")); initialSpan.tag("UPPER_CASE", ExtraFieldPropagation.get(initialSpan.context(), "UPPER_CASE"));
This section addresses how to add Sleuth to your project with either Maven or Gradle.
spring.application.name property in bootstrap.yml.
If you want to use only Spring Cloud Sleuth without the Zipkin integration, add the spring-cloud-starter-sleuth module to your project.
The following example shows how to add Sleuth with Maven:
<dependencyManagement> (1) <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>${release.train.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependency> (2) <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-sleuth</artifactId> </dependency>
We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.
Add the dependency to spring-cloud-starter-sleuth.
The following example shows how to add Sleuth with Gradle:
dependencyManagement { (1) imports { mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}" } } dependencies { (2) compile "org.springframework.cloud:spring-cloud-starter-sleuth" }
We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.
Add the dependency to spring-cloud-starter-sleuth.
If you want both Sleuth and Zipkin, add the spring-cloud-starter-zipkin dependency.
The following example shows how to do so for Maven:
<dependencyManagement> (1) <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>${release.train.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependency> (2) <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-zipkin</artifactId> </dependency>
We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.
Add the dependency to spring-cloud-starter-zipkin.
The following example shows how to do so for Gradle:
dependencyManagement { (1) imports { mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}" } } dependencies { (2) compile "org.springframework.cloud:spring-cloud-starter-zipkin" }
We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.
Add the dependency to spring-cloud-starter-zipkin.
If you want to use RabbitMQ or Kafka instead of HTTP, add the spring-rabbit or spring-kafka dependency.
The default destination name is zipkin.
spring-cloud-sleuth-stream is deprecated and incompatible with these destinations.
If you want Sleuth over RabbitMQ, add the spring-cloud-starter-zipkin and spring-rabbit
dependencies.
The following example shows how to do so for Gradle:
<dependencyManagement> (1) <dependencies> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-dependencies</artifactId> <version>${release.train.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> <dependency> (2) <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-zipkin</artifactId> </dependency> <dependency> (3) <groupId>org.springframework.amqp</groupId> <artifactId>spring-rabbit</artifactId> </dependency>
We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.
Add the dependency to spring-cloud-starter-zipkin. That way, all nested dependencies get downloaded.
To automatically configure RabbitMQ, add the spring-rabbit dependency.
dependencyManagement { (1) imports { mavenBom "org.springframework.cloud:spring-cloud-dependencies:${releaseTrainVersion}" } } dependencies { compile "org.springframework.cloud:spring-cloud-starter-zipkin" (2) compile "org.springframework.amqp:spring-rabbit" (3) }
We recommend that you add the dependency management through the Spring BOM so that you need not manage versions yourself.
Add the dependency to spring-cloud-starter-zipkin. That way, all nested dependencies get downloaded.
To automatically configure RabbitMQ, add the spring-rabbit dependency.
You can watch a video of Marcin Grzejszczak talking about Spring Cloud Sleuth and Zipkin:
Adds trace and span IDs to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator, as shown in the following example logs:
2016年02月02日 15:30:57.902 INFO [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ... 2016年02月02日 15:30:58.372 ERROR [bar,6bfd228dc00d216b,6bfd228dc00d216b,false] 23030 --- [nio-8081-exec-3] ... 2016年02月02日 15:31:01.936 INFO [bar,46ab0d418373cbc9,46ab0d418373cbc9,false] 23030 --- [nio-8081-exec-4] ...
Notice the [appname,traceId,spanId,exportable] entries from the MDC:
spanId: The ID of a specific operation that took place.
appname: The name of the application that logged the span.
traceId: The ID of the latency graph that contains the span.
exportable: Whether the log should be exported to Zipkin.
When would you like the span not to be exportable?
When you want to wrap some operation in a Span and have it written to the logs only.
Provides an abstraction over common distributed tracing data models: traces, spans (forming a DAG), annotations, and key-value annotations. Spring Cloud Sleuth is loosely based on HTrace but is compatible with Zipkin (Dapper).
Sleuth records timing information to aid in latency analysis. By using sleuth, you can pinpoint causes of latency in your applications.
Sleuth is written to not log too much and to not cause your production application to crash. To that end, Sleuth:
Propagates structural data about your call graph in-band and the rest out-of-band.
Includes opinionated instrumentation of layers such as HTTP.
Includes a sampling policy to manage volume.
Can report to a Zipkin system for query and visualization.
Instruments common ingress and egress points from Spring applications (servlet filter, async endpoints, rest template, scheduled actions, message channels, Zuul filters, and Feign client).
Sleuth includes default logic to join a trace across HTTP or messaging boundaries. For example, HTTP propagation works over Zipkin-compatible request headers.
Sleuth can propagate context (also known as baggage) between processes. Consequently, if you set a baggage element on a Span, it is sent downstream to other processes over either HTTP or messaging.
Provides a way to create or continue spans and add tags and logs through annotations.
If spring-cloud-sleuth-zipkin is on the classpath, the app generates and collects Zipkin-compatible traces.
By default, it sends them over HTTP to a Zipkin server on localhost (port 9411).
You can configure the location of the service by setting spring.zipkin.baseUrl.
If you depend on spring-rabbit or spring-kafka, your app sends traces to a broker instead of HTTP.
**
spring-cloud-sleuth-stream is deprecated and should no longer be used.
logging.pattern.level set to %5p [${spring.zipkin.service.name:${spring.application.name:-}},%X{X-B3-TraceId:-},%X{X-B3-SpanId:-},%X{X-Span-Export:-}]
(this is a Spring Boot feature for logback users).
If you do not use SLF4J, this pattern is NOT automatically applied.
To build the source you will need to install JDK 1.7.
Spring Cloud uses Maven for most build-related activities, and you should be able to get off the ground quite quickly by cloning the project you are interested in and typing
$ ./mvnw install
mvn command
in place of ./mvnw in the examples below. If you do that you also
might need to add -P spring if your local Maven settings do not
contain repository declarations for spring pre-release artifacts.
MAVEN_OPTS environment variable with
a value like -Xmx512m -XX:MaxPermSize=128m. We try to cover this in
the .mvn configuration, so if you find you have to do it to make a
build succeed, please raise a ticket to get the settings added to
source control.
For hints on how to build the project look in .travis.yml if there
is one. There should be a "script" and maybe "install" command. Also
look at the "services" section to see if any services need to be
running locally (e.g. mongo or rabbit). Ignore the git-related bits
that you might find in "before_install" since they’re related to setting git
credentials and you already have those.
The projects that require middleware generally include a
docker-compose.yml, so consider using
scripts demo
repository for specific instructions about the common cases of mongo,
rabbit and redis.
.travis.yml (usually
./mvnw install).
The spring-cloud-build module has a "docs" profile, and if you switch
that on it will try to build asciidoc sources from
src/main/asciidoc. As part of that process it will look for a
README.adoc and process it by loading all the includes, but not
parsing or rendering it, just copying it to ${main.basedir}
(defaults to ${basedir}, i.e. the root of the project). If there are
any changes in the README it will then show up after a Maven build as
a modified file in the correct place. Just commit it and push the change.
If you don’t have an IDE preference we would recommend that you use Eclipse when working with the code. We use the Importing into eclipse with m2eclipse
settings.xml. Alternatively you can
copy the repository settings from the "spring" profile of the parent
pom into your settings.xml.
If you prefer not to use m2eclipse you can generate eclipse project metadata using the following command:
$ ./mvnw eclipse:eclipse
The generated eclipse projects can be imported by selecting import existing projects
from the file menu.
ide Maven profile to turn on
Java 1.8 for both main and test sources. You MUST NOT use Java 1.8 features in the main sources. If you do
so, your app breaks during the Maven build.
Spring Cloud is released under the non-restrictive Apache 2.0 license, and follows a very standard Github development process, using Github tracker for issues and merging pull requests into master. If you want to contribute even something trivial please do not hesitate, but follow the guidelines below.
Before we accept a non-trivial patch or pull request we will need you to sign the Code of Conduct
This project adheres to the Contributor Covenant spring-code-of-conduct@pivotal.io.
None of these is essential for a pull request, but they will all help. They can also be added after the original pull request but before a merge.
Use the Spring Framework code format conventions. If you use Eclipse
you can import formatter settings using the
eclipse-code-formatter.xml file from the
Eclipse Code Formatter
Plugin to import the same file.
Make sure all new .java files to have a simple Javadoc class comment with at least an
@author tag identifying you, and preferably at least a paragraph on what the class is
for.
Add the ASF license header comment to all new .java files (copy from existing files
in the project)
Add yourself as an @author to the .java files that you modify substantially (more
than cosmetic changes).
Add some Javadocs and, if you change the namespace, some XSD doc elements.
A few unit tests would help a lot as well — someone has to do it.
If no-one else is using your branch, please rebase it against the current master (or other target branch in the main project).
When writing a commit message please follow /springpro/spring-cloud-sleuth