As of August 20th, 2007, ATF 0.1 was published to coincide with the end of the Gogle Summer of Code 2007 project. This page will not be updated any more, but will be kept for historical purposes. Please see ATF's official web site for more details and also my blog.
Note that ATF is not yet merged into NetBSD. However, the official page contains pointers to up-to-date patches to make this happen. Those will be applied to the tree once ATF is stabilized from an API/ABI point of view.
This main goal of this project is to develop an automated testing framework for the NetBSD operating system which allows any developer or end user to easily see if the system behaves to its specifications.
Some of its features will be:
Support for different kinds of tests, including unit tests, regression tests and integration tests.
Generation of a report, at the end of the whole testing process, that clearly exposes which tests succeeded, which failed and which were skipped. This report will be generated in a presentation-independent format (XML seems to be a good candidate) and later transformed to user-friendly formats such as plain text or HTML.
Interface to define tests either in C or in POSIX shell scripts. Many tests are more easily written in POSIX shell, but others require the interaction with internal system interfaces.
Independence from the source tree. The tests should be executable on a production machine without the source tree (nor the compiler suite) available.
Isolated execution: some tests are likely to crash the system instead of reporting a harmless error code. Such tests should be executed in an isolated environment such as an emulator (QEmu for i386, for example) so that these crashes do not prevent the test suite to reach completion.
Status reports will be published on my blog, The Julipedia; you can subscribe to it through either the Atom feed or the RSS feed. I will attempt to post weekly reports, but posts may appear more frequently. Note that I will read any answer you make to the posts and will try to reply to them as soon as possible.
For the record, here are direct links to the relevant posts:
May 28th, 2007: Students begin coding for their GSoC projects; Google begins issuing initial student payments.
July 9th, 2007: Students upload code to code.google.com/hosting; mentors begin mid-term evaluations.
July 16th, 2007: Mid-term evaluation deadline; Google begins issuing mid-term student payments.
August 20th, 2007: Students upload code to code.google.com/hosting; mentors begin final evaluations; students begin final program evaluations.
August 31st, 2007: Final evaluation deadline; Google begins issuing student and mentoring organization payments.
atf is currently hosted on the mtn-host.prjek.net public monotone server. This means that you need to install the monotone version control system in order to fetch the atf source code. You can do this by simply installing the devel/monotone package if you are using pkgsrc. You can also browse the source code online and download tarballs from specific revisions.
The project is currently composed of two branches:
org.NetBSD.atf.src: The atf's source code. This is completely independent from NetBSD and will be released separately.
org.NetBSD.src.atf-merge: A set of patches and a script to easily integrate the above branch into an existing NetBSD source tree. Among other things, this adds reachover Makefiles and converts (will convert) old tests to the new framework.
To get started, create a database to store the above two branches. Note that you must use a single database; otherwise the atf-merge.sh script will not work. Simply do:
$ mtn --db=atf.mtn db init
Then, pull the above two branches from the server. This is the only step that requires connection to the network:
$ mtn --db=atf.mtn pull atf.mtn-host.prjek.net "org.NetBSD.*atf*"
You can now proceed to extract (checkout) a fresh source tree from any of the two branches:
$ mtn --db=atf.mtn checkout --branch=org.NetBSD.atf.src atf $ mtn --db=atf.mtn checkout --branch=org.NetBSD.src.atf-merge atf-merge
At last, whenever you want to synchronize your local trees with what is in the server, pull the new branches and then run an update on each tree. Note that, as both trees share the same database, pulling from within one of them will effectively pull the two branches:
$ cd atf $ mtn pull $ mtn update $ cd ../atf-merge $ mtn update
You do not have to trust the public server to trust the code you received. You only have to trust my monotone public key, which signs all revisions in the database:
Key identifier: jmmv@NetBSD.org
Key fingerprint: 9c8ed88a0605d2f8da2251d24df4829ec2a75bba
[pubkey jmmv@NetBSD.org] MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDaJCQlBMjgG2uUPIrtT39OF+44BAVU4QZe DWWJEHsstsA+XRYD0wRLLJE1Xi0pugiovijMNTlW2n+MbrO0VusscMpSwi2UzGdhrDUibxEP VBs3CK5LcNpteAmqrMnSArn0GKQwCrQzR7stP0A0BEsgGBkdrZz1J6DlEOF//AnDjQIDAQAB [end]
To verify the above information with what you have on your database, use:
$ mtn --db=atf.mtn list keys $ mtn --db=atf.mtn pubkey jmmv@NetBSD.org
You can then trust all revisions signed by me, something that appears in the logs (run mtn log) as the Author field.
Mandatory (must-have) components:
A utility to run a set of tests, collect their results and generate the final report in an internal format.
A utility to convert the report generated by the previous tool and convert it to some user-friendly format.
A C library and a set of POSIX shell subroutines that simplify the creation of tests. These libraries provide the framework to define the test's metadata (purpose, requirements, keywords, etc.) and features to consistently report the test's results.
All the above will likely be provided as a new tgz set as part of the base system.
Optional (would-be-nice) components:
Conversion of the existing regression tests to the new framework. Many of them will be converted to ensure the framework works, but maybe not all of them will be converted on a first run.
Addition of new tests based on open Problem Reports (PRs).
Check this document for a lot more details on this project.