--- wikisrc/tutorials/atf.mdwn 2010/09/03 16:32:26 1.9
+++ wikisrc/tutorials/atf.mdwn 2010/09/03 21:29:36 1.10
@@ -1,6 +1,8 @@
[[!meta title="Creating atf-based tests for NetBSD src"]]
[[!toc ]]
+# Introduction
+
This quick tutorial provides a guideline on how to start creating new test
programs and/or test cases, how these tests are tied to the NetBSD source tree
and includes a short reference of the most commonly used functions.
@@ -81,12 +83,14 @@ The following properties are commonly us
* require.user: Set to 'root' to tell the atf runtime that this test requires
root privileges. The test will later be skipped if you are running atf as
- non-root, and the test will be executed otherwise.
+ non-root, and the test will be executed otherwise. Please do not use this
+ unless absolutely necessary! You can most likely make your tests run as a
+ regular user if you use puffs and rump.
* use.fs: Set to 'true' if the test case creates temporary files in the "current
directory". If set to false, the atf runtime will set the "current directory"
to an unwritable directory, which will disallow the creation of the temporary
- files and will make your test misteriously fail.
+ files and will make your test mysteriously fail.
## The body
@@ -165,7 +169,7 @@ to execute the tests of that particular
tests have been installed into the destdir.
Please note that this is only provided for convenience but it is completely
-unsupported. Tests run this way may fail misteriously, and that is perfectly
+unsupported. Tests run this way may fail mysteriously, and that is perfectly
fine as long as they work from their canonical locations in /usr/tests.
# Adding a new test
@@ -315,15 +319,17 @@ For example:
The following functions are commonly used from within a test case body:
-* ATF_CHECK(boolean_expression): Checks if the given boolean expression is true
- and, if not, records a failure for the test case but *execution continues*.
- Using ATF_REQUIRE aborts execution immediately after a failure.
-
-* ATF_CHECK_EQ(expected_expression, actual_expression): Checks if the two
- expressions match and, if not, records a failure. Similarly, ATF_REQUIRE_EQ
- aborts immediately if the check fails.
+* ATF_REQUIRE(boolean_expression): Checks if the given boolean expression is
+ true and, if not, aborts exectuion and markts the test as failed. Similarly
+ ATF_CHECK performs the same test but does not abort execution: it records the
+ failure but keeps processing the test case. For an explanation on when to use
+ which, refer to the FAQ question below.
+
+* ATF_REQUIRE_EQ(expected_expression, actual_expression): Checks if the two
+ expressions match and, if not, aborts marking the test as failed. Similarly,
+ ATF_CHECK_EQ records the error but does not abort execution.
-* ATF_CHECK_STREQ(expected_string, actual_string): Same as ATF_CHECK_EQ but
+* ATF_REQUIRE_STREQ(expected_string, actual_string): Same as ATF_REQUIRE_EQ but
performs string comparisons with strcmp.
* atf_tc_skip(const char *format, ...): Marks the test case as skipped with the
@@ -338,7 +344,7 @@ The following functions are commonly use
* atf_expect_fail(const char *format, ...): Tells the atf runtime that the code
following this call is expected to raise one or more failures (be it with
- atf_tc_fail, ATF_CHECK_*, etc.). Use this to mark a block of code that is
+ atf_tc_fail, ATF_REQUIRE_*, etc.). Use this to mark a block of code that is
known to be broken (e.g. a test that reproduces a known bug). Use the string
parameter to provide an explanation about why the code is broken; if possible,
provide a PR number. Lastly, to terminate the "expected failure" code block
@@ -364,6 +370,13 @@ The following functions are commonly use
the test program binary. This must be used to locate any data/auxiliary files
stored alongside the binary.
+* RL(integer_expression, integer): Used to evaluate a call to a libc function
+ that updates errno when it returns an error and to provide correct error
+ reporting. The integer expression is the call to such function, and the
+ literal integer provides the expected return value when there is an error.
+ For example: RL(open("foo", O_RDONLY), -1). This would fail the test case if
+ open returns -1, and would record the correct error message returned by libc.
+
# Shell test programs
## Template
@@ -563,14 +576,14 @@ Which can later be invoked as any of:
Use the "expectations" mechanism to define part of the test case as faulty,
crashy, etc. This is for two reasons:
-* As long as the bug still exists, he test case will be reported as an "expected
- failure". Such expected failures do not count towards the success or failure
- of the whole test suite.
+* As long as the bug still exists, the test case will be reported as an
+ "expected failure". Such expected failures do not count towards the success
+ or failure of the whole test suite.
* When the bug gets fixed, the bug will not trigger any more in the test case,
and thus the expectation of failure will not be met any more. At this point
the test case will start raising a regular failure, which is usually addressed
- by either just removing the expect_* calls.
+ by just removing the expect_* calls (but add a comment with the PR number!).
For example, suppose we have PR lib/1 that reports a condition in which
snprintf() does the wrong formatting when using %s, and PR lib/2 that mentions