Reading and Writing Application Logs

Overview

When a request is sent to your app, a request log is automatically written by
App Engine. During the handling of the request, your app can also write
application logs. In this page, you'll learn how to write application logs
from your application, how to read both application and request logs
programmatically using the Logs API, how to
view logging in the
Google Cloud Platform Console, and how to understand the request log data that App Engine
writes during the request. To view the contents of the logs
package, see the logs package reference.

Request logs vs application logs

There are two categories of log data: request logs and application logs. A request log is automatically written by App Engine for each request handled by your app, and contains information such as the project ID, HTTP version, and so forth. For a complete list of available properties for request logs, see Record. See
also the request log table for descriptions of the request log fields.

Each request log contains a list of application logs (AppLog) associated with that request, returned in the Record.AppLog field. Each app log contains the time the log was written, the log message, and the log level.

Note: A request log can only hold 1000 app logs. If you add more than 1000, the older logs will not be retained.

Writing application logs

Important: Individual log entries from an application have an 8KB limit.
Nothing beyond this limit will be written to the app log.
The App Engine Go SDK allows a developer to log the following levels of severity:

Reading logs in the console

Note: the logs described here are shown as visible in the Log Viewer.

To view logs written by apps running in the flexible environment, use the Logs
Viewer.

To filter log entries by label or text search in Logs Viewer, see
Basic Logs Filters. To write advanced logs
filters using expressions specifying a set of log entries from any number of
logs, see Advanced Logs Filters.

A typical App Engine log contains data in the
Apache combined log format,
along with some special App Engine fields, as shown in the following sample log:

Understanding request log fields

The following table lists the fields in order of occurrence along with a
description:

Field Order

Field Name

Always Present?

Description

1

Client address

Yes

Client IP address. Example: 192.0.2.0

2

RFC 1413 identity

No

RFC 1413 identity of the client. This is nearly always the character -

3

User

No

Present only if the app uses the Users API and the user is logged in. This value is the "nickname" portion of the Google Account, for example, if the Google Account is test@example.com, the nickname that is logged in this field is test.

4

Timestamp

Yes

Request timestamp. Example: [27/Jun/2014:09:11:47 -0700]

5

Request querystring

Yes

First line of the request, containing method, path, and HTTP version. Example: GET / HTTP/1.1

6

HTTP Status Code

Yes

Returned HTTP status code. Example: 200

7

Response size

Yes

Response size in bytes. Example: 414

8

Referrer path

No

If there is no referrer, the log contains no path, but only -. Example referrer path: "http://www.example.com/index.html".

The hostname used by the client to connect to the App Engine application. Example : (1-dot-calm-sylph-602.appspot.com)

11

Wallclock time

Yes

Total clock time in milliseconds spent by App Engine on the request. This time duration does not include time spent between the client and the server running the instance of your application. Example: ms=195.

12

CPU milliseconds

Yes

CPU milliseconds required to fulfill the request. This is the number of milliseconds spent by the CPU actually executing your application code, expressed in terms of a baseline 1.2 GHz Intel x86 CPU. If the CPU actually used is faster than the baseline, the CPU milliseconds can be larger than the actual clock time defined above. Example: cpu_ms=42

13

Exit code

No

Only present if the instance shut down after getting the request. In the format exit_code=XXX where XXX is a 3 digit number corresponding to the reason the instance shut down. The exit codes are not documented since they are primarily intended to help Google spot and fix issues.

14

Estimated cost

Yes

DEPRECATED. Estimated cost of 1000 requests just like this one, in USD. Example: cpm_usd=0.000046

15

Queue name

No

The name of the task queue used. Only present if request used a task queue. Example: queue_name=default

16

Task name

No

The name of the task executed in the task queue for this request. Only present if the request resulted in the queuing of a task. Example: task_name=7287390692361099748

17

Pending queue

No

Only present if a request spent some time in a pending queue. If there are many of these in your logs and/or the values are high, it might be an indication that you need more instances to serve your traffic. Example: pending_ms=195

18

Loading request

No

Only present if the request is a loading request. This means an instance had to be started up. Ideally, your instances should be up and healthy for as long as possible, serving large numbers of requests before being recycled and needing to be started again. Which means you shouldn't see too many of these in your logs. Example: loading_request=1.

19

Instance

Yes

Unique identifier for the instance that handles the request. Example: instance=00c61b117cfeb66f973d7df1b7f4ae1f064d

20

Version

Yes

The current App Engine release version used in production App Engine: 1.9.68

Quotas and limits

Quota for data retrieved

The first 100 megabytes of logs data retrieved per day via the Logs API calls
are free. After this amount is exceeded, no further Logs API calls will succeed
unless billing is enabled for your app. If billing is enabled for your app, data
in excess of 100 megabytes results in charges of $0.12/GB.

Logs ingestion allotment

Logging for App Engine apps is provided by Stackdriver.
See Stackdriver Pricing for
more information on logging costs and limits. For long-term storage of logs, you
can export logs from Stackdriver
to Cloud Storage, BigQuery, and Cloud Pub/Sub.

The development server and Logs API

By default, logs are stored in memory only in the development server and are accessible if you wish to test the Logs API feature. If you wish to persist logs from the development server to disk at the default location /tmp/dev_appserver.logs, supply the
--persist_logs command line option as follows:

dev_appserver.py --persist_logs your-app-directory

If you wish to persist the logs from the development server to disk at a location of your own choosing, supply the desired path and filename to the --logs_path command line option as follows: