Update status: all planned work items for 0.6 have been completed. Import
will start "soon"!

[[!meta title="Kyua: The way into NetBSD"]]
[[!toc levels=2]]
**Project owner: [Julio Merino](mailto:jmmv@NetBSD.org).**
**Status: Approved by core as of 2013-02-09. Import of Lutok and a new
release of ATF to begin soon.**
The import of Kyua into NetBSD to replace the deprecated ATF tools is
planned to happen in NetBSD 7.0. The ATF libraries will remain in place,
and as such no changes will happen to the tests that already live in
`src/tests`.
The transition from ATF to Kyua includes steps and tools to offer backwards
compatibility with users that may rely on the ATF tools shipped since
NetBSD 5.0. These backwards compatibility tools cover the most common use
cases but might not be perfect initially. Despite these provisions, this
plan should be executed well in advance the creation of the 7.0 branch to
ensure there is enough time in NetBSD-current to flesh out any major
problems.
**Before moving on, please read the
[[Kyua: An introduction for NetBSD users|/kyua]] page. You should be
familiar with the structure of Kyua and its major components to be able to
review this plan.**
# Proposed changes
The final user visible changes that this project will bring to NetBSD are
as follows. Because these change the availability of tools that have
already been shipped since 5.0, the modifications are staged to happen
across two major releases:
## For NetBSD 7.0
* Addition of the new `kyua` command-line tool. This provides the runtime
engine for the test programs (`atf-run`'s functionality) and the ability
to generate reports of the results in plain text form and HTML form
(`atf-report`'s functionality). This tool is able to execute all the
existing test programs without modifications.
* Replacement of the `Atffile`s files in `/usr/tests` with `Kyuafile`s.
* Ability to generate HTML reports right form the base system, with such
reports hooked into the various continuous build systems.
* Replacement of the `atf-run` and `atf-report` tools with shell scripts
that provide backwards compatibility implementations based on `kyua`.
These can deal with old-style `Atffile`s so that users with custom test
suites can continue to run them.
* Introduction of a new `atf2kyua` script to convert old `Atffile`s to
`Kyuafile`s. This is used internally by the compatibility `atf-run`
script, but can also be invoked by the end user by hand to deal with his
own test suites. Could be placed into `libexec` if we do not want to
make this public.
## For NetBSD 8.0
* Removal of the backwards compatibility `atf-run` and `atf-report`
scripts, as well as the supporting `atf2kyua` tool.
# Why?
As mentioned in [[another page|/kyua]], Kyua should be seen as ATF 2.x even
though it carries a different name. It is the evolution of the previous
ATF tools (*the tools only*), but written in a more modular and flexible
way, and with a more reliable codebase. Therefore, you should consider
this project as the update of ATF to a newer version.
The `atf-run` and `atf-report` tools have effectively been in maintenance
mode for over a year already. None of the desired features (see the list
of open PRs) have been implemented on top of them, mostly because doing so
is building upon a broken implementation. Additionally, several developers
have had to implement their own test results dashboards due to
defficiencies in `atf-report`, effectively reinventing the wheel.
This update will permit the real removal of the obsolete tools, thus
allowing us to build additional features on top of Kyua without having to
worry about being compatible with `atf-run` (and thus adjusting this tool
to behave in the same manner).
Some possible answers to "Why not?" can be found later in this page.
## And why now?
Kyua has existed for almost 3 years already, so you may be wondering why
this import is being proposed now. The major reasons are:
* Since 0.5, Kyua has now feature-parity with ATF. Replacing the ATF tools
with Kyua should not introduce functionality regressions.
* NetBSD-6 has been branched relatively recently, so we can expect NetBSD-7
to be far away enough in the future to provide plenty of time to
stabilize Kyua in NetBSD-current.
* The existence of the deprecated `atf-run` and `atf-report` tools hinders
the development of new features that require changes to the inteface of
ATF test programs. As an example: a highly requested feature is the
ability to change the timeout settings of the test programs (for the
benefit of old, slow platforms); doing this in the ATF codebase is
tricky.
* Some developers continue to add old-style tests to the tree. While this
is suboptimal and against the policy dictated by core@, Kyua brings in a
mechanism to allow running these tests unmodified (i.e. without adding
any ATF calls in their code). This, in turn, will let such developers
add their tests more easily, and thus increase the tests coverage.
# The plan
This section details what the transition plan looks like and an estimate
timeline. As all things that depend on free time (opensource software
hacking, in this case), take the estimates with a grain of salt.
## Discuss this plan in tech-userlevel@
If you are here it's possibly because a review request for this plan has
already been published and thus the plan has already begun.
This plan will be sent to the tech-userlevel@ mailing list asking for
comments. **Two weeks shall be allowed for initial discussion.** Depending
on the outcome, the plan and/or the software will need to be adjusted
(which in turn may require significant amounts of time not yet accounted
for). I'll be optimistic for now.
## Get core@ approval
By policy, the import of any new software component into src requires core@
approval. Even though this will have been discussed at length in
tech-userlevel@ (as per the previous step), the final decider on this issue
will be core@. The corollary of this is that, if no consensus can be
reached in tech-userlevel@ regarding this plan, core@ will be asked to come
up with a decision.
If core@ approves the plan, the next steps shall start immediately. If
core@ disagrees, core@ will be asked to provide advice on the corrections
that should be made before the plan can be approved.
It is hard to tell how long this step will last, but possibly account for 2
to 4 weeks.
## Address feedback as a new release
Publish a new Kyua release that collects all the feedback from the reviews
above.
The list of issues to be addressed can be found by querying the
[bug tracker](http://code.google.com/p/kyua/issues/list) for the
[Milestone=Release0.6](http://code.google.com/p/kyua/issues/list?can=2&q=Milestone%3DRelease0.6&colspec=ID+Type+Status+Priority+Milestone+Owner+Summary&cells=tiles)
keyword. In particular, the following are the issues that have arisen from
the review:
* [Issue 36](http://code.google.com/p/kyua/issues/detail?id=36): Fix the
`help` command to not fail if the configuration file is bogus.
* [Issue 37](http://code.google.com/p/kyua/issues/detail?id=37): Simplify
the syntax definition of configuration and `Kyuafile` files by removing
the format name.
* [Issue 40](http://code.google.com/p/kyua/issues/detail?id=40): Provide
manpages instead of an info document. **DONE**
* [Issue 47](http://code.google.com/p/kyua/issues/detail?id=47): Implement
independent testers, which reduces the amount of C++ code and avoids the
need of modifying `bsd.dep.mk`. **DONE**
* [Issue 57](http://code.google.com/p/kyua/issues/detail?id=57): Generalized
support for all metadata properties to plain test programs. This is to
make plain test programs more versatile by bringing them closer to feature
parity with ATF test programs.
## Import Kyua into src
As the [[introductory page to Kyua|/kyua]] describes, Kyua has been
available in pkgsrc for a while and can be readily installed and used to
run the tests from `/usr/tests`.
However, because ATF lives in src, and because NetBSD aims to provide the
best environment for testing "out of the box", Kyua should be imported into
src just like ATF was. The major reasons for this, as have been explained
in the past, are: first, to allow any new deployment of NetBSD to be
validated right after installation and continuously afterwards; and,
second, to permit the execution of tests during development without having
to install any additional software.
The specific steps to perform this import are:
1. Import Lutok into `external/bsd/lutok/`. This is a shared library that
wraps the native Lua C interface in C++. Lutok was originally part of
Kyua, and was split into its own package per the request of some users
that found this component useful on its own.
1. Import the Kyua testers into `external/bsd/kyua-testers/`. This yields two
new binaries in `/usr/libexec` (`kyua-atf-tester` and
`kyua-plain-tester`) and a bunch of tests in `/usr/tests/kyua-testers`.
1. Import the Kyua frontend into `external/bsd/kyua-cli/`. This yields a
new kyua binary in `/usr/bin`, a lot of test programs in
`/usr/tests/kyua-cli` (around 100) and some auxiliary files in
`/usr/share/kyua`.
1. Protect all products of Lutok and Kyua with a new `MKKYUA` knob. **Set
`MKKYUA=no` by default.** Once the ATF tools are removed, the existence
of both the `MKATF` and `MKKYUA` knows will probably be confusing. When
that happens, we can revisit this decision by possibly replacing both
with an `MKTESTS`.
1. Update `bsd.test.mk` to generate `Kyuafile`s *in addition to*
`Atffile`s.
There is no real need to do this import in a branch given that this import
only adds new functionality without touching existing stuff, and the new
code is disabled by default.
All the preparatory work for the import can be done offline (in about two
weeks at most, given that I have mot of this ready). Aside from the code
changes, this will involve the validation of NetBSD/amd64, NetBSD/i386 and
NetBSD/macppc builds (which are the ports I have access to). If you
consider that some other tricky architecture should be build-tested
(sparc64?), let me know and I'll include it in the list.
The submission to CVS will be prepared locally and performed on a package
basis (i.e. `lutok`, `kyua-testers` and `kyua-cli`, in this order). These
are to be imported separately to simplify the review of the changes and to
allow me to better test every individual change locally. There may be an
arbitrary amount of time between the submission of each package: this
should not be a problem because these modules are still disabled due to
`MKKYUA` being set to `no` by default.
## Adjust continuous testing systems to use Kyua
With `kyua` being part of the release sets, it is possible to adjust the
continuous test systems to make use of this tool in the test environments
without having to take any additional step.
I'll work with gson@ and pgoyette@ to adapt their continuous testing
machines to use the new built-in `kyua` binary instead of `atf-run` and
`atf-report`. I'm planning to do the necessary work to change `anita`
myself, and I expect to help them deploy the changes to their own systems.
Because Kyua and ATF will cohexist in the base system at this point,
migrating the continuous testing systems to Kyua can happen at its own
peace.
It might happen that Kyua misses some little detail needed by these
systems. In that case, this may require a new release of Kyua and a
reimport into the tree. Incremental imports with new features are much
easier than the original import described in here. Also, it will be
possible to cherry-pick any external changes into the tree without a
reimport (as has often been done in ATF).
This step can take a few weeks of time, mostly due to the back and forth
between different people in different timezones.
## Flip MKKYUA to yet
The previous steps imported Kyua but didn't enable its build by default so
that proper testing can be performed by the only people that care. Once
basic testing (particularly build testing on a variety of platforms) is
performed, flip `MKKYUA=yes`.
## Update documentation
The [[Kyua: An introduction for NetBSD users|/kyua]] wiki page will be
updated to explain how Kyua is bundled in NetBSD and how to use the bundled
version.
The [[Creating atf-based tests for NetBSD src|/tutorials/atf]] wiki page
will be updated to account for the differences in test programs execution
with Kyua instead of ATF.
The afterboot(8) and tests(7) manpages will be adjusted to mention `kyua`
instead of the ATF tools.
## User validation period
At this stage, **at least one month shall be given to the community** to
test the new tools and the new test results dashboards. Collect feedback
and address requests as appropriate (possibly by releasing and importing a
new version of Kyua).
One important thing to validate is that the results reported by `atf-run`
and `kyua test` match with each other. I have already been validating this
with every public release of Kyua, but it has been a manual process. To
make this more reliable, I will set up a continuous testing machine of my
own in which I will execute `atf-run` and `kyua test` in sequence (possibly
within anita) and will add an automatic comparison of the exit status of
each other.
*Because the old `atf-run` and `atf-report` tools have not yet been dropped
at this point, we can spend as much time as necessary on this phase to get
things right.*
## Replace atf-run and atf-report with kyua-atf-compat
Import the Kyua-based `atf-run` and `atf-report` compatibility tools and
stop building the deprecated versions of these. The compatibility versions
are shipped in a separate `kyua-atf-compat` package, and thus this will be
imported into `external/bsd/kyua-atf-compat/`.
Once this is done, change `bsd.test.mk` to not generate `Atffile`s any
more, as the compatibility tools do not need them.
## Drop atf-run and atf-report's code
Delete the native `atf-run` and `atf-report` utilities from the source tree
(and from the upstream repository). This also gets rid of a lot of
supporting helper code, which makes the various ATF libraries leaner (and
therefore benefits all test programs!).
## Get rid of some ATF wrappers as a proof of concept
There are some test programs (e.g. the ones in the ipf test suite) that are
not ATF-based. To plug them into the NetBSD test suite, we have
implemented ATF-based wrappers that invoke the original plain tests. This
is certainly suboptimal, as this level of indirection hinders readability
and makes development harder. (In particular, it prevents some key
developers from contributing tests in the first place.)
Because Kyua has the ability to run these "plain" (non-ATF) test programs
directly, and because a bunch of developers are really looking forward to
this feature, I will convert a few tests to not include ATF wrappers as a
proof of concept of this feature. The ipf tests are probably a good choice
for this.
## Remove atf-run and atf-report compatibility tools
This is still really far away in the future (needs to happen after NetBSD
7.0 is branched), but I'm listing it for completeness of the plan. The
idea is to get rid of any ATF compatibility scripts by NetBSD 8.0. This
gives end users a full release cycle (that of 7) to adapt to Kyua while at
the same time being able to use the old-style tools.
# Possible concerns
This section attempts to collect the list of possible concerns and/or
objections that may come up during the review, together with an attempt of
rebuttal from my side.
## Increased compilation time
The addition of any new component to src increases the build time of the
whole system. In the case of Kyua, this increase might be noticeable
*because of the large amount of test programs provided (roughly 100)*, all
of which are in C++.
Note that ATF also had a relatively large codebase and a bunch of tests
(although not as many). The "trick" was that the majority of these tests
were written in shell, and as such they did not increase the build time by
much. However, they significantly increased the run time of the test
suite, and they were less detailed (mostly integration tests, few unit
tests) than the Kyua tests.
In order to mitigate this issue, the build of all pieces of Kyua will be
protected by `MKKYUA` so that people allergic to C++ can avoid the whole
thing. Even more, there are some other additional provisions described
below.
## Introduction of more C++ code in base
First of all, why did I use C++? To make the implementation simpler and
safer (the RAII programming pattern is really useful in avoiding memory and
resource leaks with minimal effort). And C++ is part of the base system
and a supported language, so there was no reason not to do so. But that's
not the point of this item: if you dislike like C++, this is not going to
make you think I did right.
It's true that, if we count the number of lines, Kyua brings in more C++
code than what will eventually be dropped by the removal of the ATF tools.
However, because ATF was also C++, the import of Kyua itself does not make
the situation significantly worse.
Additionally, the two compilers we can really use (GCC and LLVM) already
use, or will soon use, C++ in their code base. It is unlikely that we will
be able to remove all C++ support from base anytime soon due to this, while
at the same time keeping support for all the ports that NetBSD has.
Thanks to the
[testers project](http://code.google.com/p/kyua/wiki/TestersDesign), and
starting with Kyua 0.6, a lot of the tricky OS-specific code in Kyua has
been rewritten in plain C. This paves the way to rewriting parts of the
now-simpler frontend in C or Lua, if the use of C++ proves to be a serious
problem in the future.
In the short term, the replacement of ATF with Kyua does not make things
worse: this project just changes one chunk of code with another.
## No need for Lutok as a public library
Kyua depends on Lutok, which is a C++ interface to Lua. The code of this
library was originally part of Kyua, but I split it into its own project
because some users asked for it.
If there is no desire to ship Lutok as a shared library in `/usr/lib`, we
can build Lutok as a static private library and link Kyua against it.
There is no need to install this as a shared library.