PageRenderTime 20ms CodeModel.GetById 10ms app.highlight 5ms RepoModel.GetById 1ms app.codeStats 0ms

/security/nss/tests/doc/qa_wrapper.html

http://github.com/zpao/v8monkey
HTML | 269 lines | 249 code | 20 blank | 0 comment | 0 complexity | 3e4e9ebf8f575767909e455fd334c808 MD5 | raw file
  1<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
  2<html>
  3<head>
  4   <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
  5   <meta name="GENERATOR" content="Mozilla/4.7 [en] (X11; U; SunOS 5.8 sun4u) [Netscape]">
  6</head>
  7<body text="#000000" bgcolor="#FFFFFF" link="#0000EE" vlink="#551A8B" alink="#FF0000">
  8
  9<h3>
 10<b><font face="Times New Roman,Times">Author Sonja Mirtitsch</font></b></h3>
 11
 12<h3>
 13<b><font face="Times New Roman,Times">Last updated: 4/4/2001</font></b></h3>
 14
 15<h1>
 16<b><font face="Times New Roman,Times">NSS 3.2.QA Wrapper</font></b></h1>
 17
 18<p><br>The QA&nbsp; wrapper tests the nightly builds of NSS. The actual
 19tests are being run are called from the QA script all.sh. I will add documentation
 20for the actual QA soon. The main purpose of the wrapper is: find out which
 21build (NSS version, date, Build Platform) to test on which machine (OS,
 22OS version) and construct a summary report, which is then mailed to the
 23nss developers (aka mailing list nss-qa-report@netscape.com). Please see
 24also the <a href="#advertisement">feature</a> section.
 25<p><a href="#nssqa">nssqa</a>&nbsp; - the script that calls the actual
 26qa script all.sh
 27<br><a href="#qa_stat">qa_stat</a> - sends out status reports
 28<br><a href="#qaclean">qaclean</a>&nbsp; - if everything else fails
 29<p>Sample <a href="/u/sonmi/doc/publish/glob_result.html">global result</a>,
 30<a href="/u/sonmi/doc/publish/results.html">individual result </a>and <a href="/u/sonmi/doc/publish/output.log">log
 31files</a>
 32<p>The QA wrapper consistst mainly of scripts, most located in security/nss/tests
 33and subdirectories, but run from /u/sonmi/bin
 34<p>nssqa and qa_stat, the main scripts both include a common header (<a href="../header">header</a>)
 35and a common environment (<a href="../set_environment">set_environment</a>).
 36<br>Also used is <a href="../mksymlinks">mksymlinks</a> and <a href="../path_uniq">path_uniq</a>
 37and <a href="#qaclean">qaclean</a>.
 38<p>The scripts that are used on a daily basis are located in /u/sonmi/bin
 39and checked into security/nss/tests
 40<p>Parameters and Options are the same for most scripts.
 41<p><a NAME="Parameters"></a><b><u><font size=+1>Parameters</font></u></b>
 42<br>&nbsp;&nbsp;&nbsp; nssversion (supported: 30b, 31, tip, default tip)
 43<br>&nbsp;&nbsp;&nbsp; builddate (default - today, format mmdd)
 44<p><a NAME="Options"></a><b><u><font size=+1>Options</font></u></b>
 45<br>&nbsp;&nbsp;&nbsp; -y answer all questions with y - use at your own
 46risk... ignores warnings
 47<br>&nbsp;&nbsp;&nbsp; -s silent (only usefull with -y)
 48<br>&nbsp;&nbsp;&nbsp; -h, -? -help you guessed right - displays the usage
 49<br>&nbsp;&nbsp;&nbsp; -d debug
 50<br>&nbsp;&nbsp;&nbsp; -f &lt;filename> - write the (error)output to filename
 51<br>&nbsp;&nbsp;&nbsp; -fcron writes resultfile in the same location as
 52would the -cron
 53<br>&nbsp;&nbsp;&nbsp; -m &lt;mailinglist> - send filename to mailinglist
 54(csl) only useful
 55<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; with -f on nssqa
 56<br>&nbsp;&nbsp;&nbsp; -l &lt;mozroot> run on a local build - does not
 57work at this time
 58<br>&nbsp;&nbsp;&nbsp; -cron equivalient to -y -s -d -f $RESULTDIR/$HOST.&lt;scriptname>
 59<br>&nbsp;
 60<p>Please be aware that some iPlanet specific environments and features
 61are being used.
 62<p>-d Debug option might be removed from cron in a few weeks - or maybe
 63not
 64<br>-l QA on local build is not fully implemented yet - will not be implemented,
 65all.sh can be called directly instead
 66<p>Please do not use on Windows 95 and 98, ME platforms yet.
 67<p>use -d if script behaves strange or exits unexpectedly
 68<p><b><font size=+1>How to use the QA-wrapper</font></b>
 69<br>To test a build, first run nssqa on the required QA platforms (some
 70buildplatforms require QA to be run on additional platforms - for example
 71Solaris 2.6 has to be tested on 2.8 32 and 64bit) If QA has been run on
 72multiple or all required platforms it makes sense to run qa_stat on the
 73output of nssqa as well.
 74<br>Before used on a new system (even if the same platform has been tested
 75before) please use completely interactive, to see what the variables are
 76being initialized to, and read the warnings. Same is true if being run
 77from a different user account than svbld.
 78<p>In any case, if you are using it, please let me know the results.
 79<p><a NAME="nssqa"></a><b><u><font size=+1>nssqa:</font></u></b>
 80<p>the script that calls the actual qa script all.sh
 81<p>nssqa <a href="#Parameters">parameters</a> and&nbsp; <a href="#Options">options</a>
 82<p><a href="../nssqa">view the script</a>
 83<p><b><u><font size=+1>Pseudocode Description of nssqa</font></u></b>
 84<br>not quite up to date
 85<p>&nbsp;&nbsp;&nbsp; header:init (global)
 86<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set flags and variables
 87to default values
 88<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; signal trap (for interupts
 89and kills)
 90<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set HOST and DOMSUF variables
 91if running from cron
 92<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; parse parameters and options
 93<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; determine os and set up
 94the environment (espec. PATH)
 95<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set the directories to run
 96in (influenced by parameters and -l option)<br>
 97&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set the directories for backward
 98compatibility testing
 99<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set and initialize the tmp
100/ debugging / output files
101<p>&nbsp;&nbsp;&nbsp; nssqa:init (local)
102<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; locking: if nssqa is already
103running on this systems (yes-exit,
104<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
105no-lockfile)
106<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set HOST and DOMSUF variables
107if running interavtively
108<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; set flag to kill remaining
109selfserv processes during cleanup
110<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if QA platform different
111from build platform create neccessary
112<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
113symbolic links
114<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; wait for the build to finish
115(max of 5h)
116<p>&nbsp;&nbsp;&nbsp; main:
117<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; repeated per test (optimized,
118debug, 32, 64 bit)
119<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
120set flags for this run of all.sh (optimized, debug, 32, 64 bit)
121<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
122set the DIST directory (where the binaries reside)
123<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
124kill running selfservers (sorry - just don't use the svbld
125<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
126account if you need to do your own testing... I will fix
127<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
128selfserv as soon as I can - but it hangs too often and
129<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
130disturbs all following QA)
131<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
132run all.sh
133<p>&nbsp;&nbsp;&nbsp; header:exit (global)
134<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; remove temporary files
135<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; kill remaining selfservers
136<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; send email to the list
137<br>&nbsp;
138<p>&nbsp;&nbsp;&nbsp; errorhandling
139<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Option / Parameter errors:
140Exit with usage information
141<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Severe errors: Exit wit errormessage
142<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
143example: directory in which all.sh resides does not exist
144<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
145can't create files or directories
146<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
147build not done after 5 hours
148<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
149is already running
150<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Other errors: User is prompted
151with the "errormessage - continue (y/n)?"
152<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
153example: local DIST dir does not exist (continues with next all.sh)
154<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
155outputdirectory does not exist (user can specify other)
156<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Signals 2, 3, 15 are treated
157as severe errors
158<br>&nbsp;
159<br>&nbsp;
160<br>&nbsp;
161<p><img SRC="clean.gif" height=129 width=92 align=LEFT><a NAME="qaclean"></a><b><u><font size=+2>qaclean:</font></u></b>/u/sonmi/bin/qaclean
162<br>&nbsp;
163<p>Use qaclean as user "svbld" to get the propper permissions. It is supposed
164to clean up after a "hanging" QA and will also brutally kill, interupt
165and disturb any other nss related test or performance meassurement on the
166named machine. NT and 2000 might require an additional reboot, since the
167ps is not so good about telling us the actual programmname - so we can't
168kill them... Please note that this is a brute force script, it should not
169be used on a regular basis, file a bug whenever you have to use it, since
170hanging QA is nothing that should occur frequently
171<p>&nbsp;<a href="../qaclean">view the script</a>
172<p>What it does:
173<ol>
174<li>
175see if there is a lockfile (/tmp/nssqa.$$ or $TMP/nssqa.$$)</li>
176
177<br>if yes:
178<ol>kill the process of the lockfile <font color="#666666">(future expansion
179and if possible it's children )</font>
180<br>rm the lockfile</ol>
181
182<li>
183kill selfservers</li>
184
185<li>
186kill whatever other qa related processes might be hanging</li>
187
188<li>
189clean up tmp files</li>
190</ol>
191<b>QAClean Parameters:</b>
192<br>&nbsp;&nbsp;&nbsp; machinename.
193<br>&nbsp;&nbsp;&nbsp; for example
194<br>&nbsp;&nbsp;&nbsp; qaclean kentuckyderby
195<br>&nbsp;&nbsp;&nbsp; started on any machine, will clean up on kentuckyderby
196<p><a NAME="qa_stat"></a><b><u><font size=+2>qa_stat</font></u></b>
197<p>qa_stat is the script that is being started from the svbld cron on kentuckyderby
198every morning at 10:00 and runs some (very primitive) analysis on the qa
199results.
200<br>I'd like to rewrite the whole thing in perl, and in a few weeks I might
201just do this...
202<p>&nbsp;<a href="../qa_stat">view the script</a>
203<p>qa_stat <a href="#Parameters">parameters</a> and&nbsp; <a href="#Options">options</a>
204<p><a NAME="advertisement"></a><b><u><font size=+1>Why we need the QA wrapper</font></u></b>
205<p>We need the new QA wrapper, because we have to test on so many platforms,
206that running the tests and evaluating the results for the nightly builds
207took about an average workday.
208<p><b><font size=+1>New Features:</font></b>
209<ul>
210<li>
211runs from <b>cron</b> / rsh or <b>interactive</b> if desired</li>
212
213<li>
214generates <b>summary</b> (no need to look through 60-90 directories)</li>
215
216<li>
217sends <b>email</b> about results</li>
218
219<li>
220automatically <b>recognizes common errors</b> and problems and conflicts
221and corrects them</li>
222
223<br>(or attempts to correct them :-)
224<li>
225automatically determines <b>which build </b>to test (waits if build in
226progress, exits if no build)</li>
227
228<li>
229runs on <b>all required platforms</b> (Windows 98 and before not functional
230yet)</li>
231
232<li>
233Windows version runs on <b>free Cygnus</b> as well as on MKS</li>
234
235<li>
236debug mode, normal mode and silent mode</li>
237
238<li>
239<b>locking</b> mechanism so it won't run twice</li>
240
241<li>
242<b>cleanup</b> after being killed and most errors (no remaining selfservers,
243tmpfiles, lock files)</li>
244</ul>
245The 1st script is started via cron between 5:00 and 8:00 am on different
246systems, and starts QA on the nightly build. At 10:00 the next script is
247started, and sends a QA summary to the nss developers.
248<p><b>Cygnus Advantages</b>:
249<ul>
250<li>
251<b>free</b></li>
252
253<li>
254better handling of <b>processes</b> (background, processIDs, Signals)</li>
255
256<li>
257Unix / Linux <b>compatible</b> sh / bash</li>
258</ul>
259<b>Disadvantages</b>
260<ul>
261<li>
262MKS functionality needs to be preserved (makes <b>8 Windows platforms</b>
263instead of 4 for the QA suites - makes 32 testruns on Windows alone)</li>
264
265<br>In certain functionality's <b>slow</b>
266<br><b></b>&nbsp;</ul>
267<b>Porting the windows QA&nbsp;to Uwin as well is also being considered</b>
268</body>
269</html>