12 Configuring the DyeInjection Monitor to Manage Diagnostic Contexts

This chapter describes the WLDF Instrumentation component in WebLogic Server 10.3.6 and how it provides a way to uniquely identify requests (such as HTTP or RMI requests) and track them as they flow through the system. You can configure WLDF to check for certain characteristics (such as the originating user or client address) of every request that enters the system and attach a diagnostic context to the request. This allows you to take measurements (such as elapsed time) of specific requests to get an idea of how all requests are being processed as they flow through the system.

The diagnostic context consists of two pieces: a unique Context ID and a 64-bit dye vector that represents the characteristics of the request. The Context ID associated with a given request is recorded in the Event Archive and can be used to:

Throttle instrumentation event generation, that is determine how often events are generated when specified conditions are met

Contents, Life Cycle, and Configuration of a Diagnostic Context

A diagnostic context contains a unique Context ID and a 64-bit dye vector. The dye vector contains flags which are set to identify the characteristics of the diagnostic context associated with a request. Currently, 32 bits of the dye vector are used, one for each available dye flag (see Table 12-1).

Context Life Cycle and the Context ID

The diagnostic context for a request is created and initialized when the request enters the system (for example, when a client makes an HTTP request). The diagnostic context remains attached to the request, even as the request crosses thread boundaries and Java Virtual Machine (JVM) boundaries. The diagnostic context lives for the duration of the life cycle of the request.

Every diagnostic context is identified by a Context ID that is unique in the domain. Because the Context ID travels with the request, it is possible to determine the events and log entries associated with a given request as it flows through the system.

Dyes, Dye Flags, and Dye Vectors

Contextual information travels with a request as a 64-bit dye vector, where each bit is a flag to identify the presence of a dye. Each dye represents one attribute of a request; for example, an originating user, an originating client IP address, access protocol, and so on.

When a dye flag for a given attribute is set, it indicates that the attribute is present. When the flag is not set, it indicates the attribute is not present.

For example, consider a configuration where:

the flag ADDR1 is configured to indicate a request that originated from IP address 127.0.0.1.

the flag ADDR2 is configured to indicate a request that originated from IP address 127.0.0.2.

the flag USER1 is configured to indicate a request that originated from user admin@avitek.com.

If a request from IP address 127.0.0.1 enters the system from a user other than admin@avitek.com, the ADDR1 flag in the dye vector for the request is set. The ADDR2 and USER1 dye flags remain unset.

If a request from admin@avitek.com enters the system from an IP address other than 127.0.0.1 or 127.0.0.2, the USER1 flag in the dye vector for the request is set. The ADDR1 and ADDR2 dye flags remain unset.

If a request from admin@avitek.com from IP address 127.0.0.2 enters the system, both the USER1 and ADDR2 flags in the dye vector for this request are set. The ADDR1 flag remains unset.

Diagnostic and monitoring features that take advantage of the diagnostic context can examine the dye vector to determine if one or more attributes are present (that is, the associated flag is set). In the example above, you could configure a diagnostic monitor to trace every request that is dyed with ADDR1; that is, every request originating from IP address 127.0.0.1. You could also configure a diagnostic monitor that traces every request that is dyed with both ADDR1 and USER1; that is, every request originating from user admin@avitek.com at IP address 127.0.0.1 (requests from other users at 127.0.0.1 would not be traced).

The dye vector also contains a THROTTLE dye, which is used to set how often incoming requests are dyed. For more information about this special dye, see THROTTLE Dye Flag.

For a list of the available dyes and the attributes they represent, see Dyes Supported by the DyeInjection Monitor. The process of configuring dye vectors and using them is discussed throughout the rest of this chapter.

Where Diagnostic Context Is Configured

Diagnostic context is configured as part of a diagnostic module. You use the DyeInjection monitor at the server level to configure the diagnostic context. The DyeInjection monitor is a standard diagnostic monitor, so you cannot modify its behavior. The joinpoints where the DyeInjection monitor is woven into the code are those locations where a request can enter the system.

The diagnostic action is to check every request against the DyeInjection monitor's configuration, then create and attach a diagnostic context to the request, setting the dye flags as appropriate. If the dye flags that are set for a request match the dye flags that are configured for a downstream diagnostic monitor, an event with the request's associated Context ID is added to the Event Archive. So, for example, if a request has only the USER1 and ADDR1 dye flags set, and there is a diagnostic monitor configured to trace requests with both the USER1 and ADDR1 flags set (but no other flags set), an event is added to the Event Archive.

When any request enters the system, WLDF creates and instantiates a diagnostic context for the request. The context includes a unique Context ID and a dye vector.

The DyeInjection monitor, if enabled at the server level within a WLDF diagnostic module, examines the request to see if any of the configured dye values in the dye vector match attributes of the request. For example, it checks to see if the request originated from the user associated with USER1 or USER2, and it checks to see if the request came from the IP address associated with ADDR1 or ADDR2.

For each dye value that matches a request attribute, the DyeInjection monitor sets the associated dye bits within the diagnostic context. For example, if the DyeInjection monitor is configured with USER1=weblogic, USER2=admin@avitek.com, ADDR1=127.0.0.1, ADDR2=127.0.0.2, and the request originated from user weblogic at IP address 127.0.0.2, it will set the USER1 and ADDR2 dye bits within the dye vector.

As the request flows through the system, the diagnostic context (which includes the dye vector) flows with it as well. This 64-bit dye vector contains only flags, not values. So, in this example, the dye vector contains only two flags that are explicitly set (USER1 and ADDR2). It does not contain the actual user name and IP address associated with USER1 and ADDR2.

The diagnostic monitor will perform its associated action(s) if the dye flags that are set in the diagnostic context's dye vector match the dye mask of the diagnostic monitor. See How Dye Masks Filter Requests to Pass to Monitors, for more details. In this example, the monitor will perform its action(s) if the USER1 and ADDR2 flags are set in the dye vector. In addition, an event associated with the request will be written to the Event Archive.

Configuring the Dye Vector via the DyeInjection Monitor

To create diagnostic contexts for all requests coming into the system, you must:

Create and enable a diagnostic module for the server (or servers) you want to monitor.

Enable Instrumentation for the diagnostic module.

Configure and enable the DyeInjection monitor for the module. (Only one DyeInjection monitor can be used with a diagnostic module at any one time.)

You configure the DyeInjection monitor by assigning values to dyes. The available dye flags are described in Table 12-1.

For example, you could set the flags as follows: USER1=weblogic, USER2=admin@avitek.com, ADDR1=127.0.0.1, ADDR2=127.0.0.2, and so forth. Basically, you want to set the values of one or more flags to the user(s), IP address(es) whose requests you want to monitor.

For example, to monitor all requests initiated by a user named admin@avitek from a client at IP address 127.0.0.1, assign the value admin@avitek to USER1 and assign the value 127.0.0.1 to ADDR1.

Dyes Supported by the DyeInjection Monitor

The dyes available in the dye vector are listed and explained in the following table.

Table 12-1 Request Protocols for Supported Diagnostic Context Dyes

Dye Flags

Description

ADDR1

ADDR2

ADDR3

ADDR4

Use the ADDR1, ADDR2, ADDR3 and ADDR4 dyes to specify the IP addresses of clients that originate requests. These dye flags are set in the diagnostic context for a request if the request originated from an IP address specified by the respective property (ADDR1, ADDR2, ADDR3, ADDR4) of the DyeInjection monitor.

These dyes cannot be used to specify DNS names.

CONNECTOR1

CONNECTOR2

CONNECTOR3

CONNECTOR4

Use the CONNECTOR1, CONNECTOR2, CONNECTOR3 and CONNECTOR4 dyes to identify characteristics of connector drivers.

These dye flags are set by the connector drivers to identify request properties specific to their situations. You do not configure these directly in the Administration Console or in the descriptor files. The connector drivers can assign values to these dyes (using the Connector API), so information about the connections can be carried in the diagnostic context.

COOKIE1

COOKIE2

COOKIE3

COOKIE4

COOKIE1, COOKIE2, COOKIE3 and COOKIE4 are set in the diagnostic context for an HTTP/S request, if the request contains the cookie named weblogic.diagnostics.dye and its value is equal to the value of the respective property (COOKIE1, COOKIE2, COOKIE3, COOKIE4) of the DyeInjection monitor.

The DyeInjection monitor implicitly identifies the protocol used for a request and sets the appropriate dye(s) in the dye vector, according to the protocol(s) used.

PROTOCOL_HTTP is set in the diagnostic context of a request if the request uses HTTP or HTTPS protocol.

PROTOCOL_IIOP is set in the diagnostic context of a request if it uses Internet Inter-ORB Protocol (IIOP).

PROTOCOL_JRMP is set in the diagnostic context of a request if it uses the Java Remote Method Protocol (JRMP).

PROTOCOL_RMI is set in the diagnostic context of a request if it uses the Java Remote Method Invocation (RMI) protocol.

PROTOCOL_SSL is set in the diagnostic context of a request if it uses the Secure Sockets Layer (SSL) protocol.

PROTOCOL_T3 is set in the diagnostic context of a request if the request uses T3 or T3s protocol

THROTTLE

The THROTTLE dye is set in the diagnostic context of a request if it satisfies requirements specified by THROTTLE_INTERVAL and/or THROTTLE_RATE properties of the DyeInjection monitor.

USER1

USER2

USER3

USER4

Use the USER1, USER2, USER3 and USER4 dyes to specify the user names of clients that originate requests. These dye flags are set in the diagnostic context for a request if the request was originated by a user specified by the respective property (USER1, USER2, USER3, USER4) of the DyeInjection monitor.

PROTOCOL Dye Flags

You must explicitly set the values for the dye flags USERn, ADDRn, COOKIEn, and CONNECTORn. in the DyeInjection monitor. However, the flags PROTOCOL_HTTP, PROTOCOL_IIOP, ROTOCOL_JRMP, PROTOCOL_RMI, PROTOCOL_SOAP, PROTOCOL_SSL, and PROTOCOL_T3 are set implicitly by WLDF. When the DyeInjection monitor is enabled, every request is injected with the appropriate protocol dye. For example, every request that arrives via HTTP is injected with the PROTOCOL_HTTP dye.

When Diagnostic Contexts Are Created

When the DyeInjection monitor is enabled in a diagnostic module, a diagnostic context is created for every incoming request. The DyeInjection monitor is enabled by default when you enable instrumentation in a diagnostic module. This ensures that a diagnostic Context ID is available so that events can be correlated. Even if no properties are explicitly set in the DyeInjection monitor, the diagnostic context for every request will contain a unique Context ID and a dye vector with one of the implicit PROTOCOL dyes.

If the DyeInjection monitor is disabled, no diagnostic contexts will be created for any incoming requests.

Configuring Delegating Monitors to Use Dye Filtering

You can use the DyeInjection monitor as a mechanism to restrict when a delegating or custom diagnostic monitor in the diagnostic module is triggered. This process is called dye filtering.

Each monitor can have a dye mask, which specifies a selection of the dyes from the DyeInjection monitor. When dye filtering is enabled for a diagnostic monitor, the monitor's diagnostic action is triggered and a diagnostic event is generated only for those requests that meet the criteria set by the mask.

Figure 12-2 shows an example of diagnostic events that were generated when a configured diagnostic action was triggered. Notice that the Context ID is the same for all of the events, indicating that they are related to the same request. You can use this Context ID to query for log records that are associated with the request. Note that the user ID associated with a request may not always be the same as the USER value you configured in the DyeInjection monitor; as a request is processed through the system, the user associated with the request may change to allow the system to perform certain functions (for example, the User ID may change to kernel).

Consider a Servlet_Around_Service application-scoped diagnostic monitor that has a TraceElapsedTimeAction action attached to it. Without dye filtering, any request that is handled by Servlet_Around_Service will trigger a TraceElapsedTimeAction. However, you could use dye filtering to trigger TraceElapsedTimeAction only for requests that originated from user admin@avitek.com at IP address 127.0.0.1.

After adding the monitor, click Save on the Settings for <application_name> page.

Click the Servlet_Around_Service link to display the Settings for Servlet_Around_Service page.

Select the Enabled check box to enable the monitor.

Under Actions, move TraceElapsedTimeAction from the Available list to the Chosen list.

In the Dye Mask section, move USER1 and ADDR1 from the Available list to the Chosen list.

Select the EnableDyeFiltering check box.

Click Save.

Redeploy the application.

Configurations added via the Administration Console are not persisted to the weblogic-diagnostics.xml file in the application's META-INF directory or to the DIAG_MODULE.xml file; they are saved in the application's deployment plan.

You can also manually update your DIAG_MODULE.xml file to add diagnostic monitors, as shown in Example 12-2, but this is not recommended. It is better to change the configuration via the Administration Console on a running server. Any changes you make to DIAG_MODULE.xml will not take effect until you redeploy the application.

Example 12-2 Sample Configuration for Using Dye Filtering in a Delegating Monitor, in DIAG_MODULE.xml

With this configuration, the TraceElapsedTimeAction action will be triggered for the Servlet_Around_Service diagnostic monitor only for those requests that originate from IP address 127.0.0.1 and user admin@avitek.com.

The flags that are enabled in the diagnostic monitor must exactly match the bits set in the request's dye vector for an action to be triggered and an event to be written to the Event Archive. For example, if the diagnostic monitor has both the USER1 and ADDR1 flags enabled, and only the USER1 flag is set in the request's dye vector, no action will be triggered and no event will be generated.

Note:

When configuring a diagnostic monitor, do not enable multiple flags of the same type. For example, don't enable both the USER1 and USER2 flags, as the dye vector for a given request will never have both the USER1 and USER2 flags set.

How Dye Masks Filter Requests to Pass to Monitors

A dye vector attached to a request can contain multiple dyes, and a dye mask attached to a delegating monitor can contain multiple dyes. For a delegating monitor's dye mask to allow a monitor to take action on a request, all of the following must be true:

Dye filtering for the delegating or custom diagnostic monitor is enabled in the application's weblogic-diagnostics.xml descriptor, or is enabled via the Administration Console.

The request's dye vector contains all of the dyes that are defined in the monitor's dye mask. (The dye vector can also contain dyes that are not in the dye mask.)

Dye Filtering Example

Figure 12-3 illustrates how dye filtering works, using a diagnostic module with three diagnostic monitors:

The DyeInjection monitor is configured as follows:

ADDR1=127.0.0.1
USER1=weblogic

The Servlet_Around_Service monitor is configured with a dye mask containing only ADDR1.

The EJB_Around_SessionEjbBusinessMethods monitor is configured with a dye mask containing USER1 only.

A request initiated by user guest from IP address 127.0.0.1 enters the system. The user guest does not match the value for USER1 in the DyeInjection monitor, so the request is not dyed with the dye vector USER1. The originating IP address (127.0.0.1) matches the value for ADDR1 defined in the DyeInjection monitor, so the request is dyed with the dye vector ADDR1.

The request (dyed with ADDR1) enters the Servlet component, where the diagnostic monitor Servlet_Around_Service is woven into the code. (Servlet_Around_Service triggers diagnostic actions at the entry of and exit of certain servlet and JSP methods.) Dye monitoring is enabled for the monitor, and the dye mask is defined with the single value ADDR1.

When the request enters or exits a method instrumented with Servlet_Around_Service, the diagnostic monitor checks the request for dye vector ADDR1, which it finds. Therefore, the monitor triggers a diagnostic action, which generates a diagnostic event, for example, writing data to the Events Archive.

The request enters the SessionEJB component, where the diagnostic monitor EJB_Around_SessionEjbBusinessMethods is woven into the code. (EJB_Around_SessionEjbBusinessMethods triggers diagnostic actions at the entry and exit of all SessionBean methods). Dye monitoring is enabled for the monitor, and the dye mask is defined with the single value USER1.

When the request enters or exits a SessionBean method (instrumented with EJB_Around_SessionEjbBusinessMethods), the diagnostic monitor checks the request for dye vector USER1, which it does not find. Therefore, the monitor does not trigger a diagnostic action, and therefore does not generate a diagnostic event.

Using Throttling to Control the Volume of Instrumentation Events

Throttling is used to control the number of requests that are processed by the monitors in a diagnostic module. Throttling is configured using the THROTTLE dye, which is defined in the DyeInjection monitor.

Note:

The USERn and ADDRn dyes allow inspection of requests from specific users or IP addresses. However, they do not provide a means to look at arbitrary user transactions. The THROTTLE dye provides that functionality by allowing sampling of requests.

Configuring the THROTTLE Dye

Unlike other dyes in the dye vector, the THROTTLE dye is configured through two properties.

THROTTLE_INTERVAL sets an interval (in milliseconds) after which a new incoming request is dyed with the THROTTLE dye.

If the THROTTLE_INTERVAL is greater than 0, the DyeInjection monitor sets the THROTTLE dye flag in the dye vector of an incoming request if the last request dyed with THROTTLE arrived at least THROTTLE_INTERVAL before the new request. For example, if THROTTLE_INTERVAL=3000, the DyeInjection monitor waits at least 3000 milliseconds before it will dye an incoming request with THROTTLE.

THROTTLE_RATE sets the rate (in terms of the number of incoming requests) by which new incoming requests are dyed with the THROTTLE dye.

If THROTTLE_RATE is greater than 0, the DyeInjection monitor sets the THROTTLE dye flag in the dye vector of an incoming request when the number of requests since the last request dyed with THROTTLE equals THROTTLE_RATE. For example, if THROTTLE_RATE = 6, every sixth request is dyed with THROTTLE.

You can use THROTTLE_INTERVAL and THROTTLE_RATE together. If either condition is satisfied, the request is dyed with the THROTTLE dye.

If you assign a value to either THROTTLE_INTERVAL or THROTTLE_RATE (or both, or neither), you are configuring the THROTTLE dye. A THROTTLE configuration setting in the Administration Console is shown in the following figure.

How Throttling is Handled by Delegating and Custom Monitors

Dye masks and dye filtering provide a mechanism for restricting which requests are passed to delegating and custom monitors for handling, based on properties of the requests. The presence of a property in a request is indicated by the presence of a dye, as discussed in Configuring the Dye Vector via the DyeInjection Monitor. One of those dyes can be the THROTTLE dye, so that you can filter on THROTTLE, just like any other dye.

The items in the following list explain how throttling is handled:

If dye filtering for a delegating or custom monitor is enabled and that monitor has a dye mask, filtering is performed based on the dye mask. That mask may include the THROTTLE dye, but it does not have to. If THROTTLE is included in a dye mask, then THROTTLE must also be included in the request's dye vector for the request to be passed to the monitor. However, if THROTTLE is not included in the dye mask, all qualifying requests are passed to the monitor, whether their dye vectors include THROTTLE or not.

If dye filtering for a delegating or custom monitor is not enabled and neither THROTTLE property is set in the DyeInjection monitor, dye filtering will not take place and throttling will not take place.

If dye filtering for a delegating or custom monitor is not enabled and THROTTLE is configured in the DyeInjection monitor, delegating monitors ignore dye masks but do check for the presence of the THROTTLE dye in all requests. Only those requests dyed with THROTTLE are passed to the delegating monitors for handling. Therefore, by setting a THROTTLE_RATE and/or THROTTLE_INTERVAL in the DyeInjection monitor, you reduce the number of requests handled by all delegating monitors. You do not have to configure dye masks for all your delegating monitors to take advantage of throttling.

If dye filtering for a delegating or custom monitor is enabled and the only dye set in a dye mask is THROTTLE, only those requests that are dyed with THROTTLE are passed to the delegating monitor. This behavior is the same as when dye filtering is not enabled and THROTTLE is configured in the DyeInjection monitor.

Using weblogic.diagnostics.context

The weblogic.diagnostics.context package provides applications with limited access to a diagnostic context.

An application can use the weblogic.diagnostics.context.DiagnosticContextHelper APIs to perform the following functions:

Inspect a diagnostics context's immutable context ID.

Inspect the settings of the dye flags in a context's dye vector.

Retrieve an array of valid dye flag names.

Set, or unset, the DYE_0 through DYE_7 flags in a context's dye vector. (Note that there is no way to set these flag bits via XML. You can configure DyeInjection monitor <properties> to set the non-application-specific flag bits via XML, but setDye() is the only method for setting DYE_0 through DYE_7 in a dye vector.)

Attach a payload (a String) to a diagnostic context, or read an existing payload.

An application cannot:

Set any flags in a dye vector other than the eight flags reserved for applications.

Prevent another application from setting the same application flags in a dye vector. A well-behaved application can test whether a dye flag is set before setting it.

Prevent another application from replacing a payload. A well-behaved application can test for the presence of a payload before adding one.

Note:

The diagnostic context payload can be viewed by other code in the same execution context; it can flow out of the process along with the Work instance; and it can be overwritten by other code running in the same execution context. Therefore, you should ensure the following behavior in your applications:

Avoid including any sensitive data in the payload that, for example, could be returned by the getPayload() method.

Do not create a dependency on any particular data being available in the context payload. For example, applications should not rely on a particular context ID being present. If an application uses the contents of the payload, the application should first verify that the contents match what is expected.

A monitor, or another application, that is downstream from the point where an application has set one or more of the DYE_0 through DYE_7 flags can set a dye mask to check for those flags, and take an action when the flag(s) are present in a context's dye vector. If a payload is attached to the diagnostics context, any action taken by that monitor will result in the payload being archived, and thus available through the accessor component.

Example 12-5 is a short example which (implicitly) creates a diagnostic context, prints the context ID, checks the value of the DYE_0 flag, and then sets the DYE_0 flag.