PageRenderTime 59ms CodeModel.GetById 50ms app.highlight 3ms RepoModel.GetById 2ms app.codeStats 0ms

/contrib/cvs/TESTS

https://bitbucket.org/freebsd/freebsd-head/
#! | 240 lines | 191 code | 49 blank | 0 comment | 0 complexity | bec13f94acb4083d57d835ba72e173a6 MD5 | raw file
  1To run the tests:
  2
  3	$ make check
  4
  5Note that if your /bin/sh doesn't support shell functions, you'll
  6have to try something like this, where "/bin/sh5" is replaced by the
  7pathname of a shell which handles normal shell functions:
  8
  9	$ make SHELL=/bin/sh5 check
 10
 11Also note that you must be logged in as a regular user, not root.
 12
 13WARNING:  This test can take quite a while to run, esp. if your
 14disks are slow or over-loaded.
 15
 16The tests work in /tmp/cvs-sanity (which the tests create) by default.
 17If for some reason you want them to work in a different directory, you
 18can set the TESTDIR environment variable to the desired location
 19before running them.
 20
 21The tests use a number of tools (awk, expr, id, tr, etc.) that are not
 22required for running CVS itself.  In most cases, the standard vendor-
 23supplied versions of these tools work just fine, but there are some
 24exceptions -- expr in particular is heavily used and many vendor
 25versions are deficient in one way or another.  Note that some vendors
 26provide multiple versions of tools (typically an ancient, traditional
 27version and a new, standards-conforming version), so you may already
 28have a usable version even if the default version isn't.  If you don't
 29have a suitable tool, you can probably get one from the GNU Project (see
 30http://www.gnu.org).  At this writting, expr and id are both part of the
 31GNU shellutils package, tr is part of the GNU textutils package, and awk
 32is part of the GNU gawk package.  The test script tries to verify that
 33the tools exist and are usable; if not, it tries to find the GNU
 34versions and use them instead.  If it can't find the GNU versions
 35either, it will print an error message and, depending on the severity of
 36the deficiency, it may exit.  There are environment variables you can
 37set to use a particular version of a tool -- see the test script
 38(src/sanity.sh) for details.
 39
 40Some of the tests use fairly long command lines -- this usually isn't a
 41problem, but if you have a very short command line length limit (or a
 42lot of environment variables), you may run into trouble.  Also, some of
 43the tests expect your local timezone to be an integral number of hours
 44from UTC -- if you usually use a fractional timezone, use a different
 45(integral) timezone when running the tests to avoid spurious failures.
 46
 47If running the tests produces the output "FAIL:" followed by the name
 48of the test that failed, then the details on the failure are in the
 49file check.log.  If it says "exit status is " followed by a number,
 50then the exit status of the command under test was not what the test
 51expected.  If it says "** expected:" followed by a regular expression
 52followed by "** got:" followed by some text, then the regular
 53expression is the output which the test expected, and the text is the
 54output which the command under test actually produced.  In some cases
 55you'll have to look closely to see how they differ.
 56
 57If output from "make remotecheck" is out of order compared to what is
 58expected (for example,
 59
 60   a
 61   b
 62   cvs foo: this is a demo
 63
 64is expected and
 65
 66   a
 67   cvs foo: this is a demo
 68   b
 69
 70is output), this is probably a well-known bug in the CVS server
 71(search for "out-of-order" in src/server.c for a comment explaining
 72the cause).  It is a real pain in running the testsuite, but if you
 73are lucky and/or your machine is fast and/or lightly loaded, you won't
 74run into it.  Running the tests again might succeed if the first run
 75failed in this manner.
 76
 77For more information on what goes in check.log, and how the tests are
 78run in general, you'll have to read sanity.sh.  Depending on just what
 79you are looking for, and how familiar you are with the Bourne shell
 80and regular expressions, it will range from relatively straightforward
 81to obscure.
 82
 83If you choose to submit a bug report based on tests failing, be
 84aware that, as with all bug reports, you may or may not get a
 85response, and your odds might be better if you include enough
 86information to reproduce the bug, an analysis of what is going
 87wrong (if you have the time to provide one), etc.  The check.log
 88file is the first place to look.
 89
 90ABOUT STDOUT AND STDERR
 91***********************
 92
 93The sanity.sh test framework combines stdout and stderr and for tests
 94to pass requires that output appear in the given order.  Some people
 95suggest that ordering between stdout and stderr should not be
 96required, or to put it another way, that the out-of-order bug referred
 97to above, and similar behaviors, should be considered features, or at
 98least tolerable.  The reasoning behind the current behavior is that
 99having the output appear in a certain order is the correct behavior
100for users using CVS interactively--that users get confused if the
101order is unpredictable.
102
103ABOUT TEST FRAMEWORKS
104*********************
105
106People periodically suggest using dejagnu or some other test
107framework.  A quick look at sanity.sh should make it clear that there
108are indeed reasons to be dissatisfied with the status quo.  Ideally a
109replacement framework would achieve the following:
110
1111.  Widely portable, including to a wide variety of unices, NT, Win95,
112OS/2, VMS, probably DOS and Win3, etc.
113
1142.  Nicely match extended regular expressions of unlimited length.
115
1163.  Be freely redistributable, and if possible already the kind of
117thing people might have already installed.  The harder it is to get
118and install the framework, the less people will run the tests.
119
120The various contenders are:
121
122* Bourne shell and GNU expr (the status quo).  Falls short on #1
123(we've only tried unix and NT, although MKS might help with other DOS
124mutants).  #3 is pretty good (the main dependency is GNU expr which is
125fairly widely available).
126
127* Bourne shell with a new regexp matcher we would distribute with
128CVS.  This means maintaining a regexp matcher and the makefiles which
129go with it.  Not clearly a win over Bourne shell and GNU expr.
130
131* Bourne shell, and use sed to remove variable portions of output, and
132thus produce a form that can be compared with cmp or diff (this
133sidesteps the need for a full regular expression matcher as mentioned
134in #2 above).  The C News tests are said to work this way.  This would
135appear to rely on variable portions of output having a certain syntax
136and might spuriously recognize them out of context (this issue needs
137more investigation; it isn't clear how big a problem it is in
138practice).  Same portability issues as the other choices based on the
139Bourne shell.
140
141* Dejagnu.  This is overkill; most of dejagnu is either unnecessary
142(e.g. libraries for communicating with target boards) or undesirable
143(e.g. the code which stats every file in sight to find the tests).  On
144the plus side, dejagnu is probably closer than any of the other
145choices to having everything which is needed already there.
146
147* Write our own small framework directly in tcl and distribute with
148CVS.  The tests would look much like dejagnu tests, but we'd avoid the
149unnecessary baggage.  The only dependency would be on tcl (that is,
150wish).
151
152* perl or python or <any other serious contenders here?>
153
154It is worth thinking about how to:
155
156a.  include spaces in arguments which we pass to the program under
157test (sanity.sh dotest cannot do this; see test rcs-9 for a
158workaround).
159
160b.  pass stdin to the program under test (sanity.sh, again, handles
161this by bypassing dotest).
162
163c.  have a send-expect type dialog with the program under test
164    (e.g. see server-7 or pserver-4 which want to talk the CVS
165    protocol, or the many tests which need to answer the prompt of "cvs
166    release", e.g. deep-5).
167
168ABOUT ADDING YOUR OWN TESTS
169***************************
170
171As stated in the HACKING file, patches are not accepted without documentation
172and tests.  Many people seem to be scared off by the large size of the
173sanity.sh script, but it is not really very complicated.
174
175You can probably ignore most of the begining of the script.  This section
176just sets some environment variables and finds the tools the script needs to
177run.
178
179There is one main loop you can find by grepping for "The big loop".  This loop
180repeatedly calls a case statement where the individual cases are of the form:
181
182    testname)
183            ...
184            ;;
185
186If you add a complete new test be sure to add it into the default list of tests
187(grep for 'tests=' near the begining of the script) as well as the case
188statement.  During debugging, be aware that the sanity.sh usage allows for a '-f
189testname' option to continue through the default list "from" a particular test
190as well as interpreting everything in argv past the required options as test
191names to run individual tests.
192
193Within each major test section, individual tests usually look like:
194
195    dotest testname-subtestname "shell command" "optionally multiline regexp"
196
197Tests should always start in $testdir and create a subdirectory to operate in
198and remove their cruft and end back in $testdir.  The dotest functions output
199failure messages and exit if the shell command exits with the wrong exit code or
200its stdin/stderr output doesn't match the regexp.  There are a few dotest
201variations, most notably dotest_fail for expected non-zero exit codes.
202
203Other than that the script is mostly vanilla Bourne shell.  There are a few
204constructs used for versatility and portability.  You can grep for the ones I
205miss, but here are a few important ones.  I'm leaving off long explanations
206after the first few since it probably gives you the idea and the data is in
207sanity.sh.
208
209Note that the boolean variables contain shell commands which return true or
210false when executed and are intended to be used like,
211"if $remote; then ... ; else ... ; fi"
212
213
214   * $testdir     = the directory this test is taking place in
215                    (CVSROOT=$testdir/cvsroot or
216                    CVSROOT=:fork:$testdir/cvsroot)
217   * $testcvs     = full path to the cvs executable we are testing
218   * $PLUS        = expr dependant uninterpreted '+' since this can vary
219   * $DOTSTAR     = expr dependant _interpreted_ .* since some exprs don't
220                    match EOL
221   * $username    = the username of the user running the tests
222   * $username8   = the first 8 characters of $username, output by some
223                    system and CVS commands
224   * $anyusername = regexp to match any valid system or CVS username
225   * $hostname    = regexp to match a hostname
226   * $PROG        = regexp to match progname in CVS error messages
227   * $remote      = ':' (true) or 'false', depending on whether the script is
228                    running with a remote CVSROOT
229   * $keep        = ':' (true) or 'false'.  When set, the first test run will
230                    leave any files and directories it created in $testdir and
231                    exit when complete.
232
233And, of course, some characters like '.' in regexps need to be '\' escaped when
234you mean them literally.  Some characters may be interpreted by the shell,
235e.g. backquotes and '$', are usually either escaped or replaced with '.'.
236dotest adds the final '$' anchor to the regexp itself and all the expr
237implementations I know of implicitly supply the start anchor ('^').
238
239If you only make a few mistakes, the work is, of course, still usable, though we
240may send the patch back to you for repair.  :)