NGINX Agent troubleshooting | Dynatrace communityhttps://www.dynatrace.com/support/doc/appmon/
Fri, 22 Feb 2019 10:02:28 +0100Mon, 12 Nov 2018 23:22:42 +0100Metalsmith v2.3.0NGINX Agent troubleshooting updated on Mon, 12 Nov 2018 23:22:42 +0100<h2 id="troubleshooting-startup-issues">Troubleshooting startup issues
<span class="shortlink-copy shortlink-copy-js" data-clipboard-text="https://www.dynatrace.com/support/doc/appmon/shortlink/id-nginx-agent-troubleshooting#troubleshooting-startup-issues">
</span></h2>
<section class="expandable expandable--separated" id="expand-1767offset-file-availability">
<a class="expandable__trigger" href="#expand-1767offset-file-availability">Offset file availability</a>
<article class="expandable__content"><p>If the NGINX Agent won&apos;t start, it is almost always a question of the offset file. The Agent should be able to handle that automatically in the following cases:</p>
<ul>
<li>NGINX installed from distribution package, in APT or RPM (yum)-based distributions.</li>
<li>Custom-compiled unstripped NGINX binaries.</li>
</ul>
<p>With an additional manual step, the Agent can support an arbitrary NGINX binary when debug symbols are available. To check whether your NGINX binary contains debug symbols, simply run the command <code>file</code> on it. If the output ends with <code>Not stripped</code>, the debug symbols are included.</p>
<p>When you attempt to load the Agent with a binary that is not supported out of the box, it prints the required actions to <code>stdout</code>.</p>
</article>
</section>
<section class="expandable expandable--separated" id="expand-1768agent-cannot-read-offset-file">
<a class="expandable__trigger" href="#expand-1768agent-cannot-read-offset-file">Agent cannot read offset file</a>
<article class="expandable__content"><p>The most common reason for errors of this type is file system permissions. Though NGINX starts using root privileges, workers almost always run as a special non-privileged user, <code>www-data</code> or similar. The NGINX&#xA0;Agent configuration directory (the one where it looks for the offset file) must be readable for this user. This also involves all its parent directories that must be executable for this user, otherwise it won&apos;t be possible to <code>chdir</code> to this directory).</p>
<h4 id="classic-agent">Classic Agent</h4>
<p>To use the offset self-generation functionality, the configuration directory must be writable for the non-privileged NGINX user when the file is generated. Later it can be read-only.</p>
<p>The easiest way to verify that the offset file is actually accessible for this user is:</p>
<p><code>$ sudo su www-data</code>
<code>$ cat /opt/dynatrace-7.0/agent/conf/dtnginx_offsets.json | wc -l</code></p>
<h4 id="generating-an-offset-file-manually">Generating an offset file manually</h4>
<p>You can extract the script for offset generation from the agent library. This way you always have the correct version for your Agent.</p>
<p>To dump the script to a file, set an environment variable when using <code>LD_PRELOAD</code> to load the Agent library. The actual command (<strong>ls</strong> in the example) run with the Agent preloaded is not important, as it won&apos;t even get executed &#x2010; the&#xA0;Agent terminates after dumping the script:</p>
<p><code>$ DT_DUMP_SCRIPT=1 LD_PRELOAD=/opt/dynatrace-7.0/agent/lib64/libdtnginxagent.so ls</code></p>
<p>The script is written to file <code>generate_offsets.sh</code> in the working directory. It can be used in these modes:</p>
<ul>
<li>To generate offsets for a specific unstripped NGINX binary or a stripped binary + debug symbol file pair.</li>
<li>To generate a single offset file covering multiple NGINX binaries or stripped binary + debug symbol file pairs included in specified directory tree.</li>
<li>To verify that all system requirements for NGINX Agent are met and the specified NGINX binary can be supported.</li>
</ul>
<p>You can run <code>generate_offsets.sh --help</code> for syntax details. Once the JSON file is produced, place it in the&#xA0;AppMon <code>conf</code> directory. For example:</p>
<p><code>/opt/dynatrace-7.0/agent/conf/</code></p>
<p>If offset generation fails, the script collects required system information and prints it to <code>stdout</code>. Forward it to AppMon Support, if possible accompanied by the NGINX binary itself, either unstripped or together with separate debug symbol file.</p>
<h4 id="validating-offset-files">Validating offset files</h4>
<p>If uncertain whether the given JSON offset file is compatible with their NGINX Agent version, the Agent can be ordered to validate a file. To do this, set an environment variable when using <code>LD_PRELOAD</code> to preload the Agent library. The actual command (<strong>ls</strong> in the example) run with the Agent preloaded is not important, as it won&apos;t even get executed &#x2010; the&#xA0;Agent terminates after validating the offset file:</p>
<p><code>$ DT_TEST_OFFSETS=/opt/dynatrace-7.0/agent/conf/dtnginx_offsets.json LD_PRELOAD=/opt/dynatrace-7.0/agent/lib64//libdtnginxagent.so ls</code></p>
<h4 id="bypassing-nginx-binary-hash-check">Bypassing NGINX binary hash check</h4>
<p>In rare cases, you can alter binaries on disk during system lifetime (RedHat prelink functionality comes to mind). This changes the md5 hash value used for identifying binaries within an offset file and would require regenerating the offset file, which is not always feasible.</p>
<p>To support these cases, NGINX Agent bypasses the hash check and manually identifies the offset file binary. Set an environment variable DT_NGINX_MD5 to the value of md5 hash describing the desired section in the offset file:</p>
<p><code>$ DT_NGINX_MD5=785ff0b8da9b061eb0dc4d0692d66a9d LD_PRELOAD=/opt/dynatrace-7.0/agent/lib64/libdtagent.so nginx</code></p>
<p>You must also change NGINX&apos;s configuration file (in most cases <code>nginx.conf</code>) by adding the following line in the main section:
<code>env DT_NGINX_MD5;</code></p>
<p>This makes the Agent use offsets specified in the section&#xA0;<strong>785ff0b8da9b061eb0dc4d0692d66a9d</strong> in the JSON file.</p>
<p>Ensure that this functionality is required before using it, as it bypasses several checks built into the Agent.</p>
</article>
</section>
<section class="expandable expandable--separated" id="expand-1769upstart-configuration">
<a class="expandable__trigger" href="#expand-1769upstart-configuration">Upstart configuration</a>
<article class="expandable__content"><p>Edit&#xA0;<code>/etc/init/nginx.conf</code> to attach NGINX&apos;s agent to NGINX with an Upstart startup script. Adapt the example below to your script:</p>
<pre><code class="language-text">description &quot;nginx - small, powerful, scalable web/proxy server&quot;
start on filesystem and static-network-up
stop on runlevel [016]
expect fork
respawn
pre-start script
[ -x /usr/sbin/nginx ] || { stop; exit 0; }
/usr/sbin/nginx -q -t -g &apos;daemon on; master_process on;&apos; || { stop; exit 0; }
end script
script
set -a
LD_PRELOAD=&quot;/opt/dynatrace-7.0/agent/lib64/libdtagent.so&quot;
exec /usr/sbin/nginx -g &apos;daemon on; master_process on;&apos;
end script
pre-stop exec /usr/sbin/nginx -s quit
</code></pre>
</article>
</section>
<section class="expandable expandable--separated" id="expand-1770systemd-configuration">
<a class="expandable__trigger" href="#expand-1770systemd-configuration">Systemd configuration</a>
<article class="expandable__content"><p>For <code>systemd</code> configuration <code>/lib/systemd/system/nginx.service</code> should look similar to below:</p>
<pre><code class="language-text">[Unit]
Description=A high performance web server and a reverse proxy server
After=network.target
[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g &apos;daemon on; master_process on;&apos;
Environment=LD_PRELOAD=/opt/dynatrace-7.0/agent/lib64/libdtagent.so
ExecStart=/usr/sbin/nginx -g &apos;daemon on; master_process on;&apos;
ExecReload=/usr/sbin/nginx -g &apos;daemon on; master_process on;&apos; -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed
[Install]
WantedBy=multi-user.target
</code></pre>
</article>
</section>
<h2 id="third-party-plugin-issues-and-workarounds">Third party plugin issues and workarounds
<span class="shortlink-copy shortlink-copy-js" data-clipboard-text="https://www.dynatrace.com/support/doc/appmon/shortlink/id-nginx-agent-troubleshooting#third-party-plugin-issues-and-workarounds">
</span></h2>
<section class="expandable expandable--separated" id="expand-1771google-pagespeed">
<a class="expandable__trigger" href="#expand-1771google-pagespeed">Google PageSpeed</a>
<article class="expandable__content"><p>Google <a href="https://github.com/pagespeed/ngx_pagespeed">nginx_pagespeed</a> is a module that optimizes page loading times by applying several techniques like image optimization and JavaScript and CSS inlining. The module is highly configurable by several filters and can interfere with&#xA0;AppMon UEM injection in following ways:</p>
<ul>
<li>
<p><strong>Blocked page loading</strong>: As PageSpeed intercepts the loading of JavaScript, UEM injection can cause a page not to load because of blocked HTTP requests to <code>dtagent*.js</code> and <code>dynaTraceMonitor</code>.</p>
</li>
<li>
<p><strong>Solution</strong>: Use the &apos;pagespeed Disallow&apos; filter as a solution to make it skip all requests related to AppMon UEM:</p>
</li>
</ul>
<pre><code class="language-text">pagespeed Disallow &quot;*dtagent*.js&quot;;
pagespeed Disallow &quot;*dynaTraceMonitor*&quot;;
</code></pre>
<p>The default script names can be overridden in the AppMon Client, so make sure that the pattern matches your configuration.</p>
<ul>
<li><strong>Segmentation faults</strong>: Using automatic UEM injection with PageSpeed can lead to segmentation fault errors visible in your servers error log.</li>
</ul>
<p>As a solution disable automatic UEM injection by the instrumented server and use manual injection instead. Place&#xA0;<code>&lt;script src=&apos;/&lt;http://&lt;instrumentedWebOrAppServer&gt;[/path]/dtagent_bootstrap.js&apos;&gt;&lt;/script&gt;</code> manually. See <a href="https://www.dynatrace.com/support/doc/appmon/application-monitoring/agents/javascript-agent/#anchor_manual-injection-details">Manual Injection Details</a>&#xA0;on the JavaScript Agent page to learn how.</p>
<div class="callout information">
<span class="callout__title">Tip</span>
<p>The JavaScript&#xA0;Agent&#xA0;bootstrapping is available for manual injection.</p>
</div>
</article>
</section>
Mon, 12 Nov 2018 23:22:42 +0100https://www.dynatrace.com/support/doc/appmon/installation/set-up-agents/nginx-agent-configuration/nginx-agent-troubleshooting/?updated=mon-12-nov-2018-23-22-42-0100
https://www.dynatrace.com/support/doc/appmon/installation/set-up-agents/nginx-agent-configuration/nginx-agent-troubleshooting/