Annotation of wikisrc/kyua/import.mdwn, revision 1.13

1.1 jmmv 1: [[!meta title="Kyua: The way into NetBSD"]]
1.11 wiki 2:
3: **Contents**
4:
1.1 jmmv 5: [[!toc levels=2]]
6:
7: **Project owner: [Julio Merino](mailto:jmmv@NetBSD.org).**
1.12 jmmv 8: **Status: Import of Kyua done as of 2013-02-26. All new code still
1.13 ! jmmv 9: disabled by default. Call for testing sent on 2013-02-28.**
1.1 jmmv 10:
11: The import of Kyua into NetBSD to replace the deprecated ATF tools is
12: planned to happen in NetBSD 7.0. The ATF libraries will remain in place,
13: and as such no changes will happen to the tests that already live in
14: `src/tests`.
15:
16: The transition from ATF to Kyua includes steps and tools to offer backwards
17: compatibility with users that may rely on the ATF tools shipped since
18: NetBSD 5.0. These backwards compatibility tools cover the most common use
19: cases but might not be perfect initially. Despite these provisions, this
20: plan should be executed well in advance the creation of the 7.0 branch to
21: ensure there is enough time in NetBSD-current to flesh out any major
22: problems.
23:
1.3 jmmv 24: **Before moving on, please read the
1.2 jmmv 25: [[Kyua: An introduction for NetBSD users|/kyua]] page. You should be
1.1 jmmv 26: familiar with the structure of Kyua and its major components to be able to
1.3 jmmv 27: review this plan.**
1.1 jmmv 28:
29: # Proposed changes
30:
31: The final user visible changes that this project will bring to NetBSD are
32: as follows. Because these change the availability of tools that have
33: already been shipped since 5.0, the modifications are staged to happen
34: across two major releases:
35:
36: ## For NetBSD 7.0
37:
38: * Addition of the new `kyua` command-line tool. This provides the runtime
39: engine for the test programs (`atf-run`'s functionality) and the ability
40: to generate reports of the results in plain text form and HTML form
41: (`atf-report`'s functionality). This tool is able to execute all the
42: existing test programs without modifications.
43:
44: * Replacement of the `Atffile`s files in `/usr/tests` with `Kyuafile`s.
45:
46: * Ability to generate HTML reports right form the base system, with such
47: reports hooked into the various continuous build systems.
48:
49: * Replacement of the `atf-run` and `atf-report` tools with shell scripts
50: that provide backwards compatibility implementations based on `kyua`.
51: These can deal with old-style `Atffile`s so that users with custom test
52: suites can continue to run them.
53:
54: * Introduction of a new `atf2kyua` script to convert old `Atffile`s to
55: `Kyuafile`s. This is used internally by the compatibility `atf-run`
56: script, but can also be invoked by the end user by hand to deal with his
57: own test suites. Could be placed into `libexec` if we do not want to
58: make this public.
59:
60: ## For NetBSD 8.0
61:
62: * Removal of the backwards compatibility `atf-run` and `atf-report`
63: scripts, as well as the supporting `atf2kyua` tool.
64:
1.3 jmmv 65: # Why?
66:
67: As mentioned in [[another page|/kyua]], Kyua should be seen as ATF 2.x even
68: though it carries a different name. It is the evolution of the previous
69: ATF tools (*the tools only*), but written in a more modular and flexible
70: way, and with a more reliable codebase. Therefore, you should consider
71: this project as the update of ATF to a newer version.
72:
73: The `atf-run` and `atf-report` tools have effectively been in maintenance
74: mode for over a year already. None of the desired features (see the list
75: of open PRs) have been implemented on top of them, mostly because doing so
76: is building upon a broken implementation. Additionally, several developers
77: have had to implement their own test results dashboards due to
78: defficiencies in `atf-report`, effectively reinventing the wheel.
79:
80: This update will permit the real removal of the obsolete tools, thus
81: allowing us to build additional features on top of Kyua without having to
82: worry about being compatible with `atf-run` (and thus adjusting this tool
83: to behave in the same manner).
84:
85: Some possible answers to "Why not?" can be found later in this page.
86:
87: ## And why now?
1.1 jmmv 88:
89: Kyua has existed for almost 3 years already, so you may be wondering why
90: this import is being proposed now. The major reasons are:
91:
92: * Since 0.5, Kyua has now feature-parity with ATF. Replacing the ATF tools
93: with Kyua should not introduce functionality regressions.
94:
95: * NetBSD-6 has been branched relatively recently, so we can expect NetBSD-7
96: to be far away enough in the future to provide plenty of time to
97: stabilize Kyua in NetBSD-current.
98:
99: * The existence of the deprecated `atf-run` and `atf-report` tools hinders
100: the development of new features that require changes to the inteface of
101: ATF test programs. As an example: a highly requested feature is the
102: ability to change the timeout settings of the test programs (for the
103: benefit of old, slow platforms); doing this in the ATF codebase is
104: tricky.
105:
106: * Some developers continue to add old-style tests to the tree. While this
107: is suboptimal and against the policy dictated by core@, Kyua brings in a
108: mechanism to allow running these tests unmodified (i.e. without adding
109: any ATF calls in their code). This, in turn, will let such developers
110: add their tests more easily, and thus increase the tests coverage.
111:
112: # The plan
113:
114: This section details what the transition plan looks like and an estimate
115: timeline. As all things that depend on free time (opensource software
116: hacking, in this case), take the estimates with a grain of salt.
117:
118: ## Discuss this plan in tech-userlevel@
119:
120: If you are here it's possibly because a review request for this plan has
1.4 jmmv 121: already been published and thus the plan has already begun.
1.1 jmmv 122:
123: This plan will be sent to the tech-userlevel@ mailing list asking for
124: comments. **Two weeks shall be allowed for initial discussion.** Depending
125: on the outcome, the plan and/or the software will need to be adjusted
126: (which in turn may require significant amounts of time not yet accounted
127: for). I'll be optimistic for now.
128:
129: ## Get core@ approval
130:
131: By policy, the import of any new software component into src requires core@
132: approval. Even though this will have been discussed at length in
133: tech-userlevel@ (as per the previous step), the final decider on this issue
134: will be core@. The corollary of this is that, if no consensus can be
135: reached in tech-userlevel@ regarding this plan, core@ will be asked to come
136: up with a decision.
137:
138: If core@ approves the plan, the next steps shall start immediately. If
139: core@ disagrees, core@ will be asked to provide advice on the corrections
140: that should be made before the plan can be approved.
141:
1.4 jmmv 142: It is hard to tell how long this step will last, but possibly account for 2
143: to 4 weeks.
144:
1.5 jmmv 145: ## Address feedback as a new release
146:
147: Publish a new Kyua release that collects all the feedback from the reviews
148: above.
149:
150: The list of issues to be addressed can be found by querying the
151: [bug tracker](http://code.google.com/p/kyua/issues/list) for the
152: [Milestone=Release0.6](http://code.google.com/p/kyua/issues/list?can=2&q=Milestone%3DRelease0.6&colspec=ID+Type+Status+Priority+Milestone+Owner+Summary&cells=tiles)
153: keyword. In particular, the following are the issues that have arisen from
154: the review:
155:
1.9 jmmv 156: * [Issue 36](http://code.google.com/p/kyua/issues/detail?id=36): Fix the
1.10 jmmv 157: `help` command to not fail if the configuration file is bogus. **DONE**
1.9 jmmv 158:
159: * [Issue 37](http://code.google.com/p/kyua/issues/detail?id=37): Simplify
160: the syntax definition of configuration and `Kyuafile` files by removing
1.10 jmmv 161: the format name. **DONE**
1.9 jmmv 162:
1.5 jmmv 163: * [Issue 40](http://code.google.com/p/kyua/issues/detail?id=40): Provide
1.6 jmmv 164: manpages instead of an info document. **DONE**
1.5 jmmv 165:
1.8 jmmv 166: * [Issue 47](http://code.google.com/p/kyua/issues/detail?id=47): Implement
167: independent testers, which reduces the amount of C++ code and avoids the
168: need of modifying `bsd.dep.mk`. **DONE**
169:
1.9 jmmv 170: * [Issue 57](http://code.google.com/p/kyua/issues/detail?id=57): Generalized
171: support for all metadata properties to plain test programs. This is to
172: make plain test programs more versatile by bringing them closer to feature
1.10 jmmv 173: parity with ATF test programs. **DONE**
1.9 jmmv 174:
1.1 jmmv 175: ## Import Kyua into src
176:
1.2 jmmv 177: As the [[introductory page to Kyua|/kyua]] describes, Kyua has been
178: available in pkgsrc for a while and can be readily installed and used to
179: run the tests from `/usr/tests`.
1.1 jmmv 180:
181: However, because ATF lives in src, and because NetBSD aims to provide the
182: best environment for testing "out of the box", Kyua should be imported into
183: src just like ATF was. The major reasons for this, as have been explained
184: in the past, are: first, to allow any new deployment of NetBSD to be
185: validated right after installation and continuously afterwards; and,
186: second, to permit the execution of tests during development without having
187: to install any additional software.
188:
189: The specific steps to perform this import are:
190:
191: 1. Import Lutok into `external/bsd/lutok/`. This is a shared library that
192: wraps the native Lua C interface in C++. Lutok was originally part of
193: Kyua, and was split into its own package per the request of some users
194: that found this component useful on its own.
195:
1.8 jmmv 196: 1. Import the Kyua testers into `external/bsd/kyua-testers/`. This yields two
197: new binaries in `/usr/libexec` (`kyua-atf-tester` and
198: `kyua-plain-tester`) and a bunch of tests in `/usr/tests/kyua-testers`.
199:
200: 1. Import the Kyua frontend into `external/bsd/kyua-cli/`. This yields a
201: new kyua binary in `/usr/bin`, a lot of test programs in
202: `/usr/tests/kyua-cli` (around 100) and some auxiliary files in
203: `/usr/share/kyua`.
1.1 jmmv 204:
1.6 jmmv 205: 1. Protect all products of Lutok and Kyua with a new `MKKYUA` knob. **Set
206: `MKKYUA=no` by default.** Once the ATF tools are removed, the existence
207: of both the `MKATF` and `MKKYUA` knows will probably be confusing. When
208: that happens, we can revisit this decision by possibly replacing both
209: with an `MKTESTS`.
1.1 jmmv 210:
211: 1. Update `bsd.test.mk` to generate `Kyuafile`s *in addition to*
212: `Atffile`s.
213:
214: There is no real need to do this import in a branch given that this import
1.6 jmmv 215: only adds new functionality without touching existing stuff, and the new
216: code is disabled by default.
1.1 jmmv 217:
218: All the preparatory work for the import can be done offline (in about two
219: weeks at most, given that I have mot of this ready). Aside from the code
220: changes, this will involve the validation of NetBSD/amd64, NetBSD/i386 and
221: NetBSD/macppc builds (which are the ports I have access to). If you
222: consider that some other tricky architecture should be build-tested
223: (sparc64?), let me know and I'll include it in the list.
224:
1.8 jmmv 225: The submission to CVS will be prepared locally and performed on a package
226: basis (i.e. `lutok`, `kyua-testers` and `kyua-cli`, in this order). These
227: are to be imported separately to simplify the review of the changes and to
228: allow me to better test every individual change locally. There may be an
229: arbitrary amount of time between the submission of each package: this
230: should not be a problem because these modules are still disabled due to
231: `MKKYUA` being set to `no` by default.
1.1 jmmv 232:
233: ## Adjust continuous testing systems to use Kyua
234:
235: With `kyua` being part of the release sets, it is possible to adjust the
236: continuous test systems to make use of this tool in the test environments
237: without having to take any additional step.
238:
239: I'll work with gson@ and pgoyette@ to adapt their continuous testing
240: machines to use the new built-in `kyua` binary instead of `atf-run` and
241: `atf-report`. I'm planning to do the necessary work to change `anita`
242: myself, and I expect to help them deploy the changes to their own systems.
243: Because Kyua and ATF will cohexist in the base system at this point,
244: migrating the continuous testing systems to Kyua can happen at its own
245: peace.
246:
247: It might happen that Kyua misses some little detail needed by these
248: systems. In that case, this may require a new release of Kyua and a
249: reimport into the tree. Incremental imports with new features are much
250: easier than the original import described in here. Also, it will be
251: possible to cherry-pick any external changes into the tree without a
252: reimport (as has often been done in ATF).
253:
1.4 jmmv 254: This step can take a few weeks of time, mostly due to the back and forth
255: between different people in different timezones.
256:
1.13 ! jmmv 257: ## Flip MKKYUA to yes
1.6 jmmv 258:
259: The previous steps imported Kyua but didn't enable its build by default so
260: that proper testing can be performed by the only people that care. Once
261: basic testing (particularly build testing on a variety of platforms) is
262: performed, flip `MKKYUA=yes`.
263:
264: ## Update documentation
265:
266: The [[Kyua: An introduction for NetBSD users|/kyua]] wiki page will be
267: updated to explain how Kyua is bundled in NetBSD and how to use the bundled
268: version.
269:
270: The [[Creating atf-based tests for NetBSD src|/tutorials/atf]] wiki page
271: will be updated to account for the differences in test programs execution
272: with Kyua instead of ATF.
273:
274: The afterboot(8) and tests(7) manpages will be adjusted to mention `kyua`
275: instead of the ATF tools.
276:
1.1 jmmv 277: ## User validation period
278:
279: At this stage, **at least one month shall be given to the community** to
280: test the new tools and the new test results dashboards. Collect feedback
1.5 jmmv 281: and address requests as appropriate (possibly by releasing and importing a
282: new version of Kyua).
283:
284: One important thing to validate is that the results reported by `atf-run`
285: and `kyua test` match with each other. I have already been validating this
286: with every public release of Kyua, but it has been a manual process. To
287: make this more reliable, I will set up a continuous testing machine of my
288: own in which I will execute `atf-run` and `kyua test` in sequence (possibly
289: within anita) and will add an automatic comparison of the exit status of
290: each other.
1.1 jmmv 291:
1.4 jmmv 292: *Because the old `atf-run` and `atf-report` tools have not yet been dropped
293: at this point, we can spend as much time as necessary on this phase to get
294: things right.*
295:
1.1 jmmv 296: ## Replace atf-run and atf-report with kyua-atf-compat
297:
298: Import the Kyua-based `atf-run` and `atf-report` compatibility tools and
299: stop building the deprecated versions of these. The compatibility versions
300: are shipped in a separate `kyua-atf-compat` package, and thus this will be
301: imported into `external/bsd/kyua-atf-compat/`.
302:
303: Once this is done, change `bsd.test.mk` to not generate `Atffile`s any
304: more, as the compatibility tools do not need them.
305:
306: ## Drop atf-run and atf-report's code
307:
308: Delete the native `atf-run` and `atf-report` utilities from the source tree
309: (and from the upstream repository). This also gets rid of a lot of
310: supporting helper code, which makes the various ATF libraries leaner (and
311: therefore benefits all test programs!).
312:
313: ## Get rid of some ATF wrappers as a proof of concept
314:
315: There are some test programs (e.g. the ones in the ipf test suite) that are
316: not ATF-based. To plug them into the NetBSD test suite, we have
317: implemented ATF-based wrappers that invoke the original plain tests. This
318: is certainly suboptimal, as this level of indirection hinders readability
319: and makes development harder. (In particular, it prevents some key
320: developers from contributing tests in the first place.)
321:
322: Because Kyua has the ability to run these "plain" (non-ATF) test programs
323: directly, and because a bunch of developers are really looking forward to
324: this feature, I will convert a few tests to not include ATF wrappers as a
325: proof of concept of this feature. The ipf tests are probably a good choice
326: for this.
327:
328: ## Remove atf-run and atf-report compatibility tools
329:
330: This is still really far away in the future (needs to happen after NetBSD
331: 7.0 is branched), but I'm listing it for completeness of the plan. The
332: idea is to get rid of any ATF compatibility scripts by NetBSD 8.0. This
333: gives end users a full release cycle (that of 7) to adapt to Kyua while at
334: the same time being able to use the old-style tools.
335:
336: # Possible concerns
337:
338: This section attempts to collect the list of possible concerns and/or
339: objections that may come up during the review, together with an attempt of
340: rebuttal from my side.
341:
342: ## Increased compilation time
343:
344: The addition of any new component to src increases the build time of the
345: whole system. In the case of Kyua, this increase might be noticeable
346: *because of the large amount of test programs provided (roughly 100)*, all
347: of which are in C++.
348:
349: Note that ATF also had a relatively large codebase and a bunch of tests
350: (although not as many). The "trick" was that the majority of these tests
351: were written in shell, and as such they did not increase the build time by
352: much. However, they significantly increased the run time of the test
353: suite, and they were less detailed (mostly integration tests, few unit
354: tests) than the Kyua tests.
355:
356: In order to mitigate this issue, the build of all pieces of Kyua will be
1.5 jmmv 357: protected by `MKKYUA` so that people allergic to C++ can avoid the whole
1.1 jmmv 358: thing. Even more, there are some other additional provisions described
359: below.
360:
361: ## Introduction of more C++ code in base
362:
363: First of all, why did I use C++? To make the implementation simpler and
364: safer (the RAII programming pattern is really useful in avoiding memory and
365: resource leaks with minimal effort). And C++ is part of the base system
366: and a supported language, so there was no reason not to do so. But that's
1.4 jmmv 367: not the point of this item: if you dislike like C++, this is not going to
368: make you think I did right.
1.1 jmmv 369:
370: It's true that, if we count the number of lines, Kyua brings in more C++
371: code than what will eventually be dropped by the removal of the ATF tools.
372: However, because ATF was also C++, the import of Kyua itself does not make
373: the situation significantly worse.
374:
375: Additionally, the two compilers we can really use (GCC and LLVM) already
376: use, or will soon use, C++ in their code base. It is unlikely that we will
377: be able to remove all C++ support from base anytime soon due to this, while
378: at the same time keeping support for all the ports that NetBSD has.
379:
1.8 jmmv 380: Thanks to the
381: [testers project](http://code.google.com/p/kyua/wiki/TestersDesign), and
382: starting with Kyua 0.6, a lot of the tricky OS-specific code in Kyua has
383: been rewritten in plain C. This paves the way to rewriting parts of the
384: now-simpler frontend in C or Lua, if the use of C++ proves to be a serious
385: problem in the future.
1.7 jmmv 386:
387: In the short term, the replacement of ATF with Kyua does not make things
388: worse: this project just changes one chunk of code with another.
1.1 jmmv 389:
390: ## No need for Lutok as a public library
391:
392: Kyua depends on Lutok, which is a C++ interface to Lua. The code of this
393: library was originally part of Kyua, but I split it into its own project
394: because some users asked for it.
395:
396: If there is no desire to ship Lutok as a shared library in `/usr/lib`, we
397: can build Lutok as a static private library and link Kyua against it.
398: There is no need to install this as a shared library.