Running Tests

一如大多数基于 Perl 的测试框架，`Test:Nginx` 依靠命令行工具 `prove` 来运行测试文件。

== Running Tests

Like most Perl-based testing frameworks, Test:Nginx relies on Perl’s
prove command-line utility to run the test files. The prove utility is usually
shipped with the standard perl distribution so we should already have it
when we have perl installed.

Test::Nginx always invokes a real NGINX server and a real socket client
to run the tests. It automatically uses the nginx program found in the
system environment PATH. It is your responsibility to specify the right
nginx in your PATH environment for the test suite. Usually we just
specify the path of the nginx program inside the OpenResty installation
tree. For example,

[source,bash]

Here we assume that OpenResty is installed to the default prefix, i.e.,
/usr/local/openresty/.

这里我们假设 OpenResty 安装在默认路径，即 /usr/local/openresty/。

You can always use the which command to verify if the PATH environment
is indeed set properly:

你可以使用which 命令验证 PATH 环境变量是否设置正确：

[source,console]

$ which nginx
/usr/local/openresty/nginx/sbin/nginx
—-

For convenience, we usually wrap such environment settings in a custom
shell script so that we do not risk polluting the system-wide or account-wide
environment settings nor take on the burden of manually setting the environments
manually for every shell session. For example, I usually have a local bash
script named go in each project I work on. A typical go script might
look like below

[source,bash]

exec prove “$@”

Then we can use this ./go script to substitute the prove utility in
any of the subsequent commands involving prove.

然后我们可以在要用到 prove 工具的场景中，用这个 ./go 脚本替换掉它。

Because Test::Nginx makes heavy use of environment variables for the
callers to fine tune the testing behaviors (as we shall see in later sections),
such shell wrapper scripts also make it easy to manage all these environment
variable settings and hard to get things wrong.

This is a very concise summary. The first line tells you all tests were
passed while the second line gives you a summary of the number of test
files (1 in this case), the number of tests (2 in this case), and the wallclock
and CPU times used to run all the tests.

这段总结惜字如金。第一行告诉你所有测试都通过了，第二行则给出测试文件数（1）、测试数（2）以及运行全部测试所花费的墙上时间和CPU用时。

It is interesting to see that we have only one test block in the sample
test file but in the test summary output by prove we see that the number
of tests are 2. Why the difference? We can easily find it out by asking
prove to generate a detailed test report for all the individual tests.
This is achieved by passing the -v option (meaning “verbose”) to the
prove command we used earlier:

Obviously, the first test is doing the status code check, which is dictated
by the error_code data section in the test block, and the second test
is doing the response body check, required by the response_body section.
Now the mystery is solved.

It is worth mentioning that the --- error_code: 200 section is automatically
assumed when no error_code section is explicitly provided in the test
block. So our test block above can be simplified by removing the --- error_code:
200 line without affecting the number of tests. This is because that checking
200 response status code is so common that Test::Nginx makes it the default.
If you expect a different status code, like 500, then just add an explicit
error_code section.

From this example, we can see that one test block can contain multiple
tests and the number of tests for any given test block can be determined
or predicted by looking at the data sections performing output checks.
This is important when we provide a “test plan” ourselves to the test file
where a “test plan” is the exact number of tests we expect the current
test file to run. If a different number of tests than the plan were actually
run, then the test result would be considered malicious even when all the
tests are passed successfully. Thus, a test plan adds a strong constraint
on the total number of tests expected to be run. For our t/foo.t file
here, however, we intentionally avoid providing any test plans by passing
the 'no_plan' argument to the use statement that loads the Test::Nginx::Socket
module. We will revisit the “test plan” feature and explain how to provide
one in a later section.

Running multiple test files are straightforward; just specify the file
names on the prove command line, as in

运行多个文件的方式跟单个文件的差不多，像这样在 prove 命令中指定它们的名字即可：

[source,bash]

prove -v t/foo.t t/bar.t t/baz.t
—-

If you want to run all the test files directly under the t/ directory,
then using a shell wildcard can be handy:

如果你打算一口气运行 t/ 文件夹下的所有测试文件，需要借助通配符的力量：

[source,bash]

prove -v t/*.t
—-

In case that you have sub-directories under t/, you can specify the -r
option to ask prove to recursively traverse the while directory tree
rooted
at t/ to find test files:

如果在 /t 下有子目录，你可以指定 -r 选项，告诉 prove 从根目录 t/ 递归查找测试文件：

[source,bash]

prove -r t/
—-

This command is also the standard way to run the whole test suite of a
project.

这个命令也是运行一个项目的所有测试用例的标准方法。

=== Running Individual Test Blocks 运行单个测试块

Test::Nginx makes it easy to run an individual test block in a given
file. Just add the special data section ONLY to that test block you want
to run individually and prove will skip all the other test blocks while
running that test file. For example,

Now prove won’t run any other test blocks (if any) in the same test file.

现在 prove 不再会运行同一测试文件中的其他测试（如果有的话）。

This is very handy while debugging a particular test block. You can focus
on one test case at a time without worrying about other unrelated test
cases stepping in your way.

这一特性有助于你调试特定的测试块。你可以仅关注某一测试的运行结果，把其他不相关的用例都抛到九宵云外。

When using the link:http://www.vim.org/[Vim] editor, we can quickly insert
a --- ONLY line to the test block we are viewing in the vim file buffer,
and then type :!prove % in the command mode of vim without leaving the
editor window. This works because vim automatically expands the special
% placeholder with the path of the current active file being edited.
This workflow is great since you never leave your editor window and you
never have to type the title (or other IDs) of your test block nor the
path of the containing test file. You can quickly jump between test blocks
even across different files. Test-driven development usually demands very
frequent interactions and iterations, and Test::Nginx is particularly
optimized to speed up this process.

Sometimes you may forget to remove the --- ONLY line from some test files
even after debugging, this will incorrectly skip all the other tests in
those files. To catch such mistakes, Test::Nginx always reports a warning
for files using the ONLY special section, as in

Similar to ONLY, Test::Nginx also provides the LAST data section
to make the containing test block become the last test block being run
in that test file.

类似于ONLY，Test::Nginx 也提供了 LAST 数据节，可以让所在的测试块运行在测试文件的最后。

NOTE: The special data sections ONLY and LAST are actually features
inherited from the Test::Base module.

NOTE: ONLY 和 LAST 这样的特殊的数据节实际上都是来自于 Test::Base 模块的特性。

=== Skipping Tests 跳过测试

We can specify the special SKIP data section to skip running the containing
test block unconditionally. This is handy when we write a test case that
is for a future feature or a test case for a known bug that we haven’t
had the time to fix right now. For example,

[source,test-base]

It is also possible to skip a whole test file in the prologue part. Just
replace the use statement with the following form.

在序言部分跳过整个测试文件也是可行的。仅需把 use 语句替换成下面的形式。

[source,Perl]

use Test::Nginx::Socket skip_all => “some reasons”;
—-

Then running the test file gives something like follows.

然后运行这个测试文件会有如下的输出。

….
t/foo.t .. skipped: some reasons
….

NOTE: It is also possible to conditionally skip a whole test file but it
requires a little bit of Perl programming. Interested readers can try using
a BEGIN {} before the use statement to calculate the value of
the skip_all option on the fly.

Test files are usually run by the alphabetical order of their file names.
Some people prefer explicitly controlling the running order of their test
files by prefixing the test file names with number sequences like 001-,
002-, and etc.

测试文件通常按文件名的字母表顺序依序运行。
有些人会在测试名前面添加数字前缀，比如 001-、002- 等等，来控制测试文件运行顺序。

The test suite of the link:https://github.com/openresty/lua-nginx-module#readme[ngx_http_lua]
module follows this practice, for example, which has test file names like
below

Although the prove utility supports running test files in multiple parallel
jobs via the -jN option, Test::Nginx does not really support this mode
since all the test cases share exactly the same test server directory,
t/servroot/, and the same listening ports, as we have already seen, while
parallel running requires strictly isolated running environments for each
individual thread of execution. One can still manually split the test files
into different groups and run each group on a different (virtual) machine
or an isolated environment like a Linux container.

By default, the Test::Nginx scaffold shuffles the test blocks in each
file and run them in a random order. This behavior encourages writing
self-contained and independent test cases and also increases the chance
of hitting a bug by actively mutating the relative running order of the
test cases. This may, indeed, confuse new comers, coming from a more traditional
testing platform.

We can always disable this test block shuffling behavior by calling the
Perl function, no_shuffle(), imported by the Test::Nginx::Socket module,
before the run_tests() call in the test file prologue. For example,