Welcome to YARP – 8. Distributed Tracing

Welcome to YARP – 8. Distributed Tracing

Welcome to YARP – 1. Understand YARP and build a reverse proxy service

Welcome to YARP – 2. Configuration function

• 2.1 – Configuration Files
• 2.2 – Configuration Providers
• 2.3 – Configuration Filters

Welcome to YARP – 3. Load balancing

Welcome to YARP – 4. Current Limit

Welcome to YARP – 5. Authentication and Authorization

Welcome to YARP – 6. Compression and caching

Welcome to YARP – 7. Target Health Check

Welcome to YARP – 8. Distributed Tracing

At the end of this article, the study of YARP has come to an end. There are also many omitted chapters (such as: middleware, HTTPS and TLS, GRPC, etc.). Friends who want to know more can go to the documentation on the official website to find out.

Introduction

Before introducing YARP‘s distributed tracing, let’s first understand what distributed tracing is.

When we build a large application or system, we usually split it into multiple parts, which may run on different computers or processes. This distributed architecture helps improve system scalability and performance, but also increases the difficulty of fault diagnosis. Distributed tracing acts like a detective tool for the application, helping engineers identify problems in the application, especially those that may span multiple computers or processes.

For example, suppose we have a typical web service. After the user sends a request, the request may go through the load balancer and then be passed to the back-end Web server process. In the end, it may involve multiple queries to the database. Using distributed tracing, engineers can track the entire request process just like we are investigating a case. They can tell whether each step was successful, how much time each step took, and even record the details each step produced.

As a component of ASP.NET Core, YARP can be easily integrated into different tracking systems like any other Web application. Distributed tracing can be configured using the following program, see for details:

• OpenTelemetry (is a vendor-neutral library that supports multiple services)
• Application Insights Application Insights (a full-featured service provided by Microsoft)

.NET has built-in configurable support for distributed tracing, which YARP leverages to enable such scenarios out of the box.

Use custom tracking header

When using a propagation mechanism that .NET does not natively support, you need to create a special propagator (DistributedContextPropagator) to handle the context information transfer of the mechanism.

YARP will remove DistributedContextPropagator.Fields(This is an attribute or field in DistributedContextPropagator that is used to store context propagation-related information) so that the propagator can re-add them to the request during the Inject call. This step is to effectively manage header information throughout the propagation process to ensure they are processed and delivered appropriately.

Passthrough proxy

If you do not want the proxy to actively participate in tracking, and wish to preserve all tracking headers, you can do so by setting SocketsHttpHandler.ActivityHeadersPropagator to null. This means that the proxy will maintain transparent delivery of tracking headers without actively intervening.

services.AddReverseProxy()
     .ConfigureHttpClient((context, handler) => handler.ActivityHeadersPropagator = null);
 

example

1. Create project

dotnet new web -n YARP.Metrics -f net6.0
 

2. Add project reference

This is a library provided by YARP that listens to various stages of proxy operations to collect detailed information and performance metrics about request processing.

Internally, YARP uses an EventSource to collect telemetry events and metrics from the many subsystems used to handle requests.

To listen to these metrics, classes implementing each functional interface need to be registered in DI (dependency injection). The following are the functions provided by this class library:

Function Overview:

1. Proxy: Represents the entire proxy operation, including success or failure.
   • Events include:
     • When starting and stopping proxy requests
     • When processing the request/response body
   • Metrics include:
     • Number of requests initiated
     • Number of ongoing requests
     • If failed pleaseFind the quantity
2. Kestrel: The web server that handles incoming requests.
   • Events include:
     • When a request starts/stops or fails
   • Metrics include:
     • Connection rate – connections opened per second
     • Total number of connections
     • TLS handshakes
     • Inbound queue length
3. Http: HttpClient used to make outbound requests to the target server.
   • Events include:
     • When the connection is created
     • When a request starts/stops or fails
     • Headers/content when sent/received
     • When a request is dequeued when a connection is available
   • Metrics include:
     • Number of outbound requests initiated
     • Number of failed requests
     • Number of active requests
     • Number of outbound connections
4. Sockets: Events involving connection attempts and metrics about the amount of data sent and received.
5. NameResolution: Events involving name resolution attempts and metrics about DNS queries for the target.
6. NetSecurity: Events involving SslStream handshakes and metrics about handshake number and latency for each protocol.

3. Key files

• ForwarderTelemetryConsumer (Listens for events from proxy telemetry, recording timing and information related to the high-level process of proxy request processing.)
• HttpClientTelemetryConsumer (Listens for events from HttpClient telemetry, recording timing and information related to outbound requests and responses from the target server.)
• PerRequestMetrics (Class that stores metrics calculated on a per-request basis. Instances are stored in AsyncLocal storage throughout the lifetime of the request.)
• PerRequestYarpMetricCollectionMiddleware (The first and last step in processing requests. It initializes the metrics for each request and logs the results at the end of the request.)

4.Register in DI

using YARP.Metrics;

 var builder = WebApplication.CreateBuilder(args);

 var services = builder.Services;

 services.AddControllers();

 services.AddReverseProxy()
     .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));

 services.AddHttpContextAccessor();

 // Interface for collecting general metrics about proxy forwarding
 services.AddMetricsConsumer();

 //Register the user to the event telemetered by the proxy forwarder
 services.AddTelemetryConsumer();

 // Register the user to HttpClient telemetry events
 services.AddTelemetryConsumer();

 services.AddTelemetryConsumer();

 var app = builder.Build();

 // Custom middleware to collect and report agent metrics
 // Placed at the beginning so it's the first and last thing run on every request
 app.UsePerRequestMetricCollection();

 // Middleware for intercepting WebSocket connections and collecting telemetry exposed to WebSocketsTemetryConsumer
 app.UseWebSocketsTelemetry();

 app.MapReverseProxy();

 app.Run();
 

5.Appsettings.json configuration

{
   "Logging": {
     "LogLevel": {
       "Default": "Information",
       // "Microsoft": "Warning",
       "Microsoft.Hosting.Lifetime": "Information"
     }
   },
   "AllowedHosts": "*",
   "ReverseProxy": {
     "Routes": {
       "route1": {
         "ClusterId": "cluster1",
         "Match": {
           "Path": "{**catch-all}"
         }
       }
     },
     "Clusters": {
       "cluster1": {
         "Destinations": {
           "cluster1/destination1": {
             "Address": "https://www.baidu.com/"
           }
         }
       }
     }
   }
 }
 

5.Run Project

Next, when we run the project, we can see some indicator data of the proxy request:

Summary

This is the end of the Distributed Tracing chapter. It is particularly important in distributed systems and can analyze performance bottlenecks and locate errors and exceptions. And the telemetry data (metrics) collected can be exported to a number of different backend storage or visualization tools. For example: Zipkin, Jaeger, Prometheus, these are all things for later. Interested friends can do their own research. The relevant code has been uploaded to Github, and key files are also commented. At this point, the study of YARP has come to an end. There are also many omitted chapters (such as: middleware, HTTPS and TLS, GRPC, etc.). Friends who want to know more can go to the documentation on the official website to find out.