diff --git a/tests/openpgp/README b/tests/openpgp/README index 9b384beb1..84faf1cdd 100644 --- a/tests/openpgp/README +++ b/tests/openpgp/README @@ -1,161 +1,175 @@ # Emacs, this is an -*- org -*- file. * How to run the test suite -** using the legacy driver -On POSIX you can just use +From your build directory, run - $ make -C tests/openpgp check + obj $ make -C tests/openpgp check -or +to run all tests or - $ make -C tests/openpgp check XTESTS="setup.scm your-test.scm finish.scm" + obj $ make -C tests/openpgp check XTESTS=your-test.scm -as before. -** using the Scheme driver +to run a specific test (or any number of tests separated by spaces). + +If you want to debug a test, add verbose=1 to see messages printed by +spawned programs to their standard error stream, verbose=2 to see what +programs are executed, or verbose=3 to see even more program output +and exit codes. + +** Passing options to the test driver + +You can set TESTFLAGS to pass flags to 'run-tests.scm'. For example, +to speed up the test suite when bisecting, do + + obj $ make -C tests/openpgp check TESTFLAGS=--parallel + +See below for the arguments supported by the driver. + +** Calling the test driver directly This is a bit tricky because one needs to manually set some environment variables. We should make that easier. See discussion below. From your build directory, do: - obj $ srcdir=/tests/openpgp \ + obj $ TMP=/tmp srcdir=/tests/openpgp \ GPGSCM_PATH=/tests/gpgscm:/tests/openpgp \ $(pwd)/tests/gpgscm/gpgscm [gpgscm args] \ run-tests.scm [test suite runner args] *** Arguments supported by the test suite runner The test suite runner supports four modes of operation, {sequential,parallel}x{isolated,shared}. You can select the mode of operation using a combination of the flags --parallel, --sequential, --shared, and --isolated. By default the tests are run in sequential order, each one in a clean environment. You can specify the tests to run as positional arguments relative to srcdir (e.g. just 'version.scm'). By default all tests listed in run-tests.scm are executed. Note that you do not have to specify setup.scm and finish.scm, they are executed implicitly. The test suite runner can be executed in any location that the current user can write to. It will create temporary files and directories, but will in general clean up all of them. *** Discussion of the various environment variables **** srcdir Must be set to the source of the openpgp test suite. Used to locate data files. **** GPGSCM_PATH Used to locate the Scheme library as well as code used by the test suite. **** BIN_PREFIX The test suite does not hardcode any paths to tools. If set it is used to locate the tools to test, otherwise the test suite assumes to be run from the build directory. **** MKTDATA and GPG_PRESET_PASSPHRASE These two tools are not installed by 'make install', hence we need to explicitly override their position. In fact, the location of any tool used by the test suite can be overridden this way. See defs.scm. **** argv[0] run-tests.scm depends on being able to re-exec gpgscm. It uses argv[0] for that. Therefore you must use an absolute path to invoke gpgscm. * How to write tests gpgscm provides a number of functions to aid you in writing tests, as well as bindings to process management abstractions provided by GnuPG. For the Scheme environment provided by TinySCHEME, see the TinySCHEME manual that is included in tests/gpgscm/Manual.txt. For a quick start, please have a look at various tests that are already implemented, e.g. 'encrypt.scm'. ** The test framework The functions info, error, and skip display their first argument and flush the output buffers. error and skip will also terminate the process, signaling that the test failed or should be skipped. (for-each-p msg proc list) will display msg, and call proc with each element of list while displaying the progress appropriately. for-each-p' is similar, but accepts another callback before the 'list' argument to format each item. for-each-p can be safely nested, and the inner progress indicator will be abbreviated using '.'. ** Temporary files (lettmp ) will create and delete temporary files that you can use in . (with-temporary-working-directory ) will create a temporary director, change to that, and clean it up after executing ). make-temporary-file will create a temporary file. You can optionally provide an argument to that function that will serve as tag so you can distinguish the files for debugging. remove-temporary-file will delete a file created using make-temporary-file. ** Monadic transformer and pipe support Tests often perform sequential transformations on files, or connect processes using pipes. To aid you in this, the test framework provides two monadic data structures. (Currently, the implementation mashes the 'bind' operation together with the application of the monad. Also, there is no 'return' operation. I guess all of that could be implemented on top of call/cc, but it isn't at the moment.) *** pipe The pipe monad constructs pipe lines. It consists of a function pipe:do that binds the functions together and manages the execution of the child processes, a family of functions that act as sources, a function to spawn processes, and a family of functions acting as sinks. Sources are pipe:open, pipe:defer, pipe:echo. To spawn a process use pipe:spawn, or the convenience function pipe:gpg. To sink the data use pipe:splice, or pipe:write-to. Example: (pipe:do (pipe:echo "3\n1\n2\n") (pipe:spawn '("/usr/bin/sort")) (pipe:write-to "sorted" (logior O_WRONLY O_CREAT) #o600)) Caveats: Due to the single-threaded nature of gpgscm you cannot use both a source and sink that is implemented in Scheme. pipe:defer and pipe:echo are executing in gpgscm, and so does pipe:splice. *** tr The transformer monad describes sequential file transformations. There is one source function, tr:open. To describe a transformation using some process, use tr:spawn, tr:gpg, or tr:pipe-do. There are several sinks, although sink is not quite the right term, because the data is not consumed, and hence one can use them at any position. The "sinks" are tr:write-to, tr:call-with-content, tr:assert-identity, and tr:assert-weak-identity. A somewhat contrived example demonstrating many functions is: (tr:do (tr:pipe-do (pipe:echo "3\n1\n2\n") (pipe:spawn '("/usr/bin/sort"))) (tr:write-to "reference") (tr:call-with-content (lambda (c) (echo "currently, c contains" (string-length c) "bytes"))) (tr:spawn "" '("/usr/bin/gcc" -x c "-E" -o **out** **in**)) (tr:pipe-do (pipe:spawn '("/bin/grep" -v "#"))) (tr:assert-identity "reference")) Caveats: As a convenience, gpgscm allows one to specify command line arguments as Scheme symbols. Scheme symbols, however, are case-insensitive, and get converted to lower case. Therefore, the -E argument must be given as a string in the example above. Similarly, you need to quote numerical values. ** Process management If you just need to execute a single command, there is (call-with-fds cmdline infd outfd errfd) which executes cmdline with the given file descriptors bound to it, and waits for its completion returning the status code. There is (call cmdline) which is similar, but calls the command with a closed stdin, connecting stdout and stderr to stderr if gpgscm is executed with --verbose. (call-check cmdline) raises an exception if the command does not return 0. (call-popen cmdline input) calls a command, writes input to its stdin, and returns any output from stdout, or raises an exception containing stderr on failure. * Sample messages