PageRenderTime 33ms CodeModel.GetById 16ms app.highlight 9ms RepoModel.GetById 1ms app.codeStats 0ms

/IronPython_Main/Languages/IronPython/Tests/debugging/__init__.py

#
Python | 774 lines | 751 code | 1 blank | 22 comment | 1 complexity | 23367d8ddb5878a6ebd0840e175c36f0 MD5 | raw file
  1#####################################################################################
  2#
  3#  Copyright (c) Microsoft Corporation. All rights reserved.
  4#
  5# This source code is subject to terms and conditions of the Apache License, Version 2.0. A 
  6# copy of the license can be found in the License.html file at the root of this distribution. If 
  7# you cannot locate the  Apache License, Version 2.0, please send an email to 
  8# ironpy@microsoft.com. By using this source code in any fashion, you are agreeing to be bound 
  9# by the terms of the Apache License, Version 2.0.
 10#
 11# You must not remove this notice, or any other, from this software.
 12#
 13#
 14#####################################################################################
 15
 16'''
 17-------------------------------------------------------------------------------
 18INTRODUCTION
 19This test plan addresses debugging test coverage for the 2.x series of 
 20IronPython interpreter releases.
 21
 22EDITORS NOTES
 23Throughout this document you will find references to documentation in
 24other packages similar to "See documentation for debugging.pdb_module".  This simply
 25means that relative to this package (i.e., debugging), you should follow
 26the 'pdb_module (package)' link at the bottom of this page.
 27
 28-------------------------------------------------------------------------------
 29FEATURE GOALS
 30The goals of IP's debugging feature is to provide IronPython users with 
 31complete use of CPython's existing debugger functionality, the "pdb" and
 32"bdb" modules respectively, while also providing a great interactive debugging 
 33experience using Microsoft tools such as mdbg and the Visual Studio IDE.
 34
 35-------------------------------------------------------------------------------
 36PRIMARY TESTING CONCERNS
 37- new builtin module attributes added to IP to support CPython's "bdb" and "pdb"
 38  modules
 39- compatibility with CPython's use of the "bdb" and "pdb" modules
 40- what happens breakpoint-wise when IP steps into .NET code from "pdb"?
 41- .NET applications hosting IP should be able to debug Python code
 42- the experience of debugging IP under Microsoft's mdbg and Visual 
 43  Studio tools is what existing CSharp and VB.Net would expect; but still
 44  Pythonic
 45- "-D" command-line flag
 46
 47-------------------------------------------------------------------------------
 48PRIMARY TESTING FOCUS
 49Our principal testing focus will be on testing that IP's support of the
 50"pdb" module is completely compatible with that of CPython by running existing
 51CPython tests, and also by creating a new generic side-by-side testing framework.
 52
 53The secondardy focus will be testing that debugging IP under mdbg is a
 54pleasant experience.  This will initially be acheived by using the CLR Team's
 55existing debugging test framework, but in the long run we will move to a
 56new generic, side-by-side testing framework which can be reused for other 
 57side-by-side testing areas.
 58
 59-------------------------------------------------------------------------------
 60REFERENCES
 61* IronPython Test Plan              Internal IP SharePoint site
 62* Feature Specifications            None...
 63* Development Documents             None...
 64* Schedule Documents                None...
 65* CPython pdb module                http://docs.python.org/library/pdb.html
 66* CPython bdb module                http://docs.python.org/library/bdb.html
 67* pdb tutorial                      http://onlamp.com/pub/a/python/2005/09/01/debugger.html
 68* MDbg.exe                          http://msdn.microsoft.com/en-us/library/ms229861.aspx
 69* Debugging IP in Visual Studio     http://devhawk.net/2008/05/08/Debugging+IronPython+Code+In+Visual+Studio.aspx
 70* Bug queries                       http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx
 71                                    and set the component to "Debugging"
 72* Build drop location               Updated at http://ironpython.codeplex.com/SourceControl/ListDownloadableCommits.aspx
 73                                    on weekdays
 74* source file tree                  $/.../dlr/Debugging/ and also $/.../dlr/.../IronPython/*
 75
 76-------------------------------------------------------------------------------
 77PERSONNEL
 78Program Manager:    hpierson
 79Developer:          dinov
 80Tester:             dfugate
 81
 82-------------------------------------------------------------------------------
 83TESTING SCHEDULE
 84    
 85    Phase I - Test Plan and Enabling Existing Tests
 86        - Write test plan for IronPython Debugging feature
 87        - Enable any existing CPython test cases for "pdb" and "bdb" in SNAP
 88        - Enable any existing CPython test cases for Python features supporting 
 89          "pdb" and "bdb" in SNAP
 90        
 91    Phase II - Examine Coverage of Existing Cases and Write High-priority Cases
 92        - Verify that developer tests for new additions to IP 
 93          supporting "pdb" and "bdb" modules are sufficient. Write more test
 94          cases if not
 95        - Write additional test cases for the "pdb" module if CPython's test cases 
 96          are insufficient
 97        - Examine our existing mdbg test (cases) looking for holes. Write new test
 98          cases if needed
 99        - Author a few manual sanity test cases for verifying IP can 
100          be debugged under the Visual Studio IDE. These should be folded into
101          the CodePlex test pass signoff procedure for major releases
102          
103    Phase III - Design and Implement Side-by-Side Testing Framework
104        - Based on the needs of test cases added in the previous phase either:
105            1. Design a side-by-side testing infrastructure to be used with testing
106               IP's CPython debugging compatibility OR
107            2. Look for existing side-by-side testing frameworks we could reuse.
108               The pexpect package might be useful here
109        - Implement (or add existing) side-by-side testing framework
110    
111    Phase IV - Implement High-priority Cases
112        - Implement test cases for additions to IP which support the
113          "pdb" module IF NEEDED
114        - Implement test cases for the "pdb" module IF NEEDED
115        - Implement test cases for mdbg IF NEEDED
116        
117    Phase IV - Write Lower Priority Cases
118        - Write test cases around stepping into (hosted) Python code from a 
119          .NET application
120        - Write test cases around stepping into .NET code from an ipy.exe
121          process
122        - Write performance test cases
123                    
124    Phase V - Implement Lower Priority Cases
125        - Implement test cases around stepping into (hosted) Python code from a 
126          .NET application
127        - Implement test cases around stepping into .NET code from an ipy.exe
128          process
129        - Implement performance test cases
130    
131    Phase VI - Future
132        - Develop test cases for the "bdb" module if CPython's test cases are
133          insufficient
134        - Automate manual sanity Visual Studio debugging tests
135        - Migrate existing mdbg tests to the new side-by-side testing framework
136        - Low-resource test cases and implementation
137        - Reliability test cases and implementation
138        - Scalability test cases and implementation
139        - Stress test cases and implementation
140        - Internationalization test cases and implementation
141
142    Phase P - All Public Releases
143        - Verify all user documentation with respect to debugging
144        - Setup testing - verify pdb.py and all supporting modules are installed by the MSI
145        - Generate code coverage report and verify block coverage as greater
146          than 80%
147
148-------------------------------------------------------------------------------
149FEATURE HISTORY
1502.0:     Tests implemented for IronPython under mdbg. Scenarios which still 
151         need to be covered are:
152         * Attach scenario with and without the -D option
153         * Step into/out when there is no pdb file for IP 
154           binaries
155         * Multiple statements in one line (e.g., "x = 1;y=2"
156         * if x and \\n y: print 1
157         * dict/tuple spanning multiple lines
158         * break/continue, explicit raise
159         * other multi-line statement scenarios such as:
160           if True: print 1
161           if True:\\n    print 1
162         * MdbgInterop
163           able to break into the C# code from Python
164           able to break back into Python code from C#
165
1662.6B2:   First release of IronPython to include support of CPython's pdb module.
167         Only works if -X:Frames or -X:FullFrames option supplied to ipy.exe.
168         No pdb-related tests running in SNAP at this point, but we are running
169         all of CPython's tests WRT supporting functions in the sys module.
170         Debugging test plan completely rewritten in the form of pydoc strings,
171         and made public.
172
173-------------------------------------------------------------------------------
174FEATURES:
175- sys.settrace, sys.gettrace, and sys.call_tracing.  NOTE: while each of these
176  three builtin module members needs to be tested within unit tests on their
177  own, the real use case here is through the "bdb" and "pdb" modules
178- ipy.exe's generation of PDBs
179- ipy.exe's PDB visualizers
180
181-------------------------------------------------------------------------------
182FILES AND MODULES:
183
184- FILES LIST:
185    * everything under Microsoft.Scripting.Debugging, although the DLR 
186      officially owns this
187    * sys.cs
188    * IronPython.Runtime.PythonTracebackListener
189- REGISTRY, INI SETTINGS: None
190- SETUP PROCEDURES: 
191    CPython standard library must be installed and present in sys.path to be 
192    usable
193- DE-INSTALLATION PROCEDURES
194    N/A
195- DATABASE SETUP AND PROCEDURES
196    N/A
197- NETWORK DOMAIN/TOPOLOGIES CONFIGURATION PROCEDURES
198    N/A
199- PERFORMANCE MONITORING COUNTERS SETUP AND CONFIGURATIONS
200    N/A
201
202-------------------------------------------------------------------------------
203OPERATIONAL ISSUES
204N/A.  This feature of IP is not being monitored/maintained by operational 
205staff, and IP is provided on an as-is basis.
206
207-------------------------------------------------------------------------------
208SCOPE OF TEST CASES
209We'll be getting most of our test coverage on the debugging feature of 
210IP by unit testing the CPython 'pdb' module.  Doing this thoroughly
211should guarantee we hit nearly all blocks of code supporting debugging in 
212IP DLLs. This will be validated and verified through monthly code 
213coverage runs.  As an aside, this should in theory hit most of the DLR's
214Microsoft.Scripting.Debugging.dll as well.
215
216As IP currently has no tie-in into the Visual Studio IDE, this is 
217arguably far less important to test than our support of the 'pdb' module or 
218even mdbg.  As a result, the Microsoft toolset testing emphasis will involve
219exahaustively covering all aspects of mdbg. That said, some of our users do in
220fact have VS Pro installed implying that we should perform at least minimal,
221manual sanity tests before every major public release of IP.  Should we
222ever provide visualizers for our debug symbols we'll need much more 
223comprehensive and automated tests for VS of course.
224 
225-------------------------------------------------------------------------------
226ACCEPTANCE CRITERIA
227- no debugging related feature should be checked into the internal source 
228  repository without some form of unit test hooked into SNAP checked in as
229  well
230- we should not publically advertise the existance of any new debugging features
231  in an IP release unless:
232    - if the feature is CPython-based; the majority of it's corresponding 
233      CPython unit test cases have been enabled in SNAP and pass
234    - if the feature is novel to IP; block coverage should be greater
235      than 70%
236    - the feature needs to be documented in some form other than a blog
237    - performance of the feature needs to be "within reason"
238    - if the feature is not CPython-based; it needs a Pythonic feel to it
239    - at least half of all bugs opened on the feature since its inception are
240      fixed
241
242-------------------------------------------------------------------------------
243KEY FEATURE ISSUES
244- no spec on debugging IP from mdbg, VS, or what debugging symbols 
245  should be generated for various Python constructs
246- insufficient test resources to adequately test this feature
247- insufficient resources to implement all debugging features called out in this
248  test plan
249- existing mdbg tests are based on CLR infrastructure and require a Perl 
250  installation. We cannot redistribute this publically
251- existing mdbg tests in SNAP are flakey
252
253-------------------------------------------------------------------------------
254TEST APPROACH
255- DESIGN VALIDATION
256    We have no part in the design of the "pdb" and "bdb" standard modules. 
257    
258    As for debugging under mdbg...there is no current design document on 
259    debugging Python sessions.  Generally speaking test will simply ensure that
260    whatever mdbg functionality exists with regard to IP is "Pythonic",
261    but at the same time familiar to CSharp and VB developers.
262
263- DATA VALIDATION
264    pdb/bdb: at a very low-level, all we need to validate is that parameters passed
265    to a trace function we set via sys.settrace are as expected.  In particular,
266    the first parameter passed in is the current stack frame which we have a 
267    very solid chance of getting wrong.
268
269    mdbg/VS: need to validate that generation of PDB symbols "make sense"
270    with respect to whatever Python source is being compiled. There are at 
271    least two ways to accomplish this:
272    1. Create a regression test in which we compare the generated PDB of a 
273       known Python script to an expected PDB file.  The Python script should
274       exhaustively cover everything that's possible in the Python grammar.
275       Testing in this manner is not really maintainable in the long run as we
276       expect the PDB generated for a given Python script to change over time
277       reflecting optimizations and new features added to IP
278    2. Test that the behavior of mdbg, given a known Python script, does not 
279       regress.  Again, this Python script should exhaustively cover everything
280       that's possible in Python's grammar, but this time around we'll also
281       need to cover everything that's possible under mdbg
282
283- API TESTING
284    We'll exhaustively cover the entire "pdb" standard module without ever 
285    touching the CSharp APIs directly.  Prior coverage runs have shown that 
286    (generally speaking) any block of code that is not directly hittable through 
287    the Python API is likely dead code.  Also, testing "bdb" module is lower 
288    priority as "pdb" uses this for its own implementation.  The final 
289    justification for this is that there is actually very little in terms of 
290    debugging APIs in IP DLLs.  Most of the debugging support is built
291    directly into the DLR's Microsoft.Scripting.Debugging.dll.
292    
293    For mdbg/VS, we'll exhaustively cover all mdbg commands.
294    
295- CONTENT TESTING
296  All debugging documentation should be thoroughly reviewed by Test before any
297  public release of IronPython.  Careful attention will be given to duplicating
298  exactly any command fed into ipy.exe or mdbg.exe, and this should be 
299  automated entirely if feasible.
300  
301- LOW-RESOURCE TESTING
302  Use of the debugging feature has a few side effects which might be interesting
303  from a limited resources perspective.  First and foremost, use of the -X:Frames
304  and -X:FullFrames IP console flags imply that IP will consume more 
305  memory than under normal circumstances.  We should check that IP does not 
306  consume "too much" extra memory.  Next, use of the debugging flag, '-D', in
307  conjunction with the '-X:SaveAssemblies' flag will generate 
308  Snippets.debug.scripting.pdb in the %TMP% directory. What occurs if the drive
309  containing %TMP% is full?
310
311  NOTE: while this is an interesting test area to explore, there are no current
312  plans to test it.  We should perform general low-resource testing of 
313  IP before tackling it for this specific feature.
314  
315- SETUP TESTING
316  The only option of IP's MSI installer capable of affecting
317  the debugging feature is the ability to selectively install CPython's standard
318  library.  As we generate the list of CPython modules to include in the MSI
319  dynamically, it would be worthwhile to always check that pdb.py is included.
320  
321  NOTE: IronPython currently has no automated tests for the MSI. Once we
322  do, we need to add a check to ensure pdb.py is always included.  For the time
323  being this will have to be done manually.
324  
325- MODES AND RUNTIME OPTIONS
326  We should focus our efforts on testing IP with the following modes:
327  * -D -X:Frames
328  * -D -X:FullFrames
329  * -D -X:Frames -X:SaveAssemblies
330  * -D
331  * -X:Frames
332  * 
333  using both debug and release IP assemblies.  Also note that this
334  needs to be performed for both Python interactive sessions and Python scripts
335  passed into ipy.exe (e.g., "ipy.exe test_str.py").  Last but
336  not least, there's the %PYTHONDEBUG% environment variable to consider.
337  
338- INTEROPERABILITY
339  Of primary concern is that IP produces an identical experience as
340  CPython when given input which utilizes the pdb module. This will be 
341  accomplished via side-by-side testing and verification that the output of 
342  IP is the same as CPython given an identical input.
343  
344  We will also verify that IP can be debugged from mdbg and the Visual
345  Studio IDE.
346  
347- INTEGRATION TESTING
348  * Silverlight support of debugging?
349  * breaking into .NET code from pdb
350  * breaking into COM from pdb
351  * breaking into IronRuby from pdb
352  * breaking into Python code from DLR hosting APIs
353  * breaking into Python code from IronRuby
354
355- BETA TESTING
356  The second beta release of IronPython 2.6 will support the debugging feature
357  in the form of very limited pdb module support with the -X:Frames flag 
358  passed to ipy.exe. As with all beta releases of IP, this will be 
359  released on CodePlex to the general public. We will then use feedback
360  from the IronPython Community to determine the amount of effort that goes 
361  into fully testing this feature.
362  
363- ENVIRONMENT/SYSTEM - GENERAL
364  The %PYTHONDEBUG% environment variable may have some impact on this feature.
365  Other than this, the VS IDE will of course need to be installed for testing
366  the VS debugging experience.
367  
368- CONFIGURATION
369  Need .NET 2.0 Service Pack 1 installed to run IP, and .NET 3.5
370  Service Pack 1 to build the feature.
371  
372- USER INTERFACE
373  IP provides no user interfaces.
374  
375- PERFORMANCE & CAPACITY TESTING
376  Minimally, we need to run one or more tests in the perf lab under the -X:Frames
377  and -X:FullFrames test modes.  We should also measure the end-to-end run time
378  of a complete, yet minimal, debugging sample utilizing the pdb module in the 
379  lab.  The focus here will simply be on ensuring there are no perf regressions
380  which are greater than 15% for any given checkin, and also that overall 
381  performance of the debugging feature remains palatable to our userbase.
382  
383  In terms of capacity testing we should look at:
384  - if perf gets affected when breaking into deeply nested functions (e.g., 
385    recursive functions)
386  - anything else?
387  
388- PRIVACY
389  Does %TMP%\Snippets.debug.scripting.pdb contain sensitive customer data? If 
390  so, does IP disclose the existence of this file?
391  Is remote debugging supported? If so, what safeguards are in place?
392  
393- RELIABILITY
394  Most end-users are expected to enter into debugging sessions for short 
395  periods of time to diagnose issues with Python code.  From this perspective
396  the so-called "up time" of debugging sessions is less important than the 
397  overall sporadic failure frequency. We can get a good sense of how often 
398  sporadic failures occur by automatically running debugging tests in SNAP 
399  for every IP checkin.
400  
401- SCALABILITY
402  Does debugging still work correctly when stepping into deeply nested (e.g., 1000)
403  functions?
404  
405- STRESS TESTING
406  - gcstress
407  - MDA
408  - assemblies in the GAC
409  - call pdb functions multiple times and look for memory leaks
410  - what happens when stepping into unreasonably deeply nested functions?
411  
412- VOLUME TESTING
413  App building exercise or try utilizing pdb from some of our internal Python 
414  tools. It would be interesting to use this from gopackage when exceptions 
415  are encountered. 
416  
417- INTERNATIONAL ISSUES
418  As Python is not a localized language and available only in English, we 
419  simply need to confirm that nothing gets localized.  This can be done by
420  running a simple debugbing test against non-English OSes such as Deutsche
421  Vista.
422  
423- ROBUSTNESS
424  We'll need a dedicated test or two in the stress lab to ensure there are no
425  memory leaks.  Also, we'll need to keep an eye out for sporadic failures of
426  debugging tests in SNAP.
427  
428- ERROR TESTING
429  What happens when invalid commands are fed to the pdb debugger? Is this 
430  handled identically to CPython?
431  
432  Is there any conceivable way to fully break debugging without actually 
433  breaking ipy.exe in general?  What happens under the mode 
434  "-D -X:SaveAssemblies" when the current user doesn't have write 
435  permissions on %TMP%?  What happens if MS.Scripting.Debugging.dll is 
436  removed outright?
437  
438- USABILITY
439  There are three major usability issues with debugging:
440  * IP offers no visualizer for the VS IDE
441  * we're largely incompatible with CPython's pdb module
442  * debugging Python code from .NET apps isn't currently possible
443  
444  Test will depend on feedback from the IronPython Mailing List and blogs to
445  determine more usability issues. Resource permitting, we may also do 
446  app building exercises.
447  
448  The usability goal of this feature is that there are no complaints about
449  the IP debugging experience from our users.
450  
451- ACCESSIBILITY
452  As this feature provides no new user interfaces to IronPython. and simply 
453  emits output to the stdout stream of command prompts, there should be no 
454  need to invest into accessibility testing.
455  
456- USER SCENARIOS
457  It's anticipated that users of this feature will be limited almost exclusively to
458  developers. Expert Python users will likely prefer using the pdb
459  module or debugging their applications directly via the "-i" ipy.exe 
460  option.  Existing Microsoft customers with little Python background will 
461  probably be more comfortable with the VS IDE debugging experience. Due to 
462  this we must make sure both experiences are great!
463  
464  The types of applications being debugged will likely:
465  * be non-trivial
466  * consist of many different modules/packages
467  * make heavy use of the CPython standard library and/or third party Python
468    packages
469  * be debugged often during the application development phase
470  * be debugged infrequently in a production environment when something goes 
471    awry
472  
473  With these constraints in mind, we should try to debug existing major third
474  party Python applications and/or add debugging support to our own internal
475  Python tools.
476  
477- BOUNDARIES AND LIMITS
478  What happens when stepping into a recursive function at the maximum recursion
479  level?
480  
481- OPERATIONAL ISSUES
482  None
483  
484- SPECIAL CODE PROFILING AND OTHER METRICS
485  Overall block coverage of all assemblies with "ipy" or "IronPython" in their
486  names should be above 80% and file coverage should stay above 97%. Visual 
487  Studio's mstest will be used to measure code coverage.
488
489-------------------------------------------------------------------------------
490TEST ENVIRONMENT
491
492- OPERATING SYSTEMS
493  32-bit Windows XP
494  32-bit Windows 2003
495  64-bit Windows 2003
496  32-bit Windows Vista
497  64-bit Windows Vista
498  32-bit Windows Vista (Deutsche)
499- NETWORKS
500  General Intranet network connection required for SNAP.
501  May be other special network needs if remote debugging is/becomes a supported scenario.
502- HARDWARE
503  - MACHINES
504    At least two machines of every OS variety called out above with at least
505    10 gigs of free hard disk space, 2 gigs of RAM, and a modern CPU
506- SOFTWARE
507  * Visual Studio 2008 Team System Service Pack 1 installed to run the 
508    test suite
509  * PowerShell 1.0 to run any supporting test scripts
510  * CPython for side-by-side tests
511  * mdbg
512
513-------------------------------------------------------------------------------
514UNIQUE TESTING CONCERNS FOR SPECIFIC FEATURES
515It should be relatively easy to drive pdb or mdbg via the command line in an
516automated fashion, but the same cannot be said about the VS IDE debugging 
517experience.  We'll need to reuse the VS team's automation infrastructure and 
518ensure any machines running these tests in SNAP are never locked at the 
519username/password screen which would cause the test to fail.  Also, should 
520visualization ever be a supported scenario, we may have no other option than to 
521test this manually (note - this needs to be investigated).
522
523-------------------------------------------------------------------------------
524AREA BREAKDOWN
525- CPython's "pdb" module
526  See debugging.pdb_mod.
527
528- mdbg tool support
529  See debugging.mdbg_tool.
530      
531- VS IDE
532  See debugging.vs.
533  
534- sys module extensions
535  - sys.settrace
536    settrace(function) 
537    Set the system's trace function, which allows you to implement a Python 
538    source code debugger in Python. The function is thread-specific; for a 
539    debugger to support multiple threads, it must be registered using 
540    settrace() for each thread being debugged.
541
542    Trace functions should have three arguments: frame, event, and arg. frame 
543    is the current stack frame. event is a string: 'call', 'line', 'return', 
544    'exception', 'c_call', 'c_return', or 'c_exception'. arg depends on the 
545    event type.
546
547    The trace function is invoked (with event set to 'call') whenever a new 
548    local scope is entered; it should return a reference to a local trace 
549    function to be used that scope, or None if the scope shouldn't be traced.
550
551    The local trace function should return a reference to itself (or to another 
552    function for further tracing in that scope), or None to turn off tracing 
553    in that scope.
554
555    The events have the following meaning:
556    'call'
557        A function is called (or some other code block entered). The global 
558        trace function is called; arg is None; the return value specifies the 
559        local trace function.
560    'line'
561        The interpreter is about to execute a new line of code (sometimes 
562        multiple line events on one line exist). The local trace function is 
563        called; arg is None; the return value specifies the new local trace 
564        function.
565    'return'
566        A function (or other code block) is about to return. The local trace 
567        function is called; arg is the value that will be returned. The trace 
568        function's return value is ignored.
569    'exception'
570        An exception has occurred. The local trace function is called; arg is 
571        a tuple (exception, value, traceback); the return value specifies the 
572        new local trace function.
573    'c_call'
574        A C function is about to be called. This may be an extension function 
575        or a builtin. arg is the C function object.
576    'c_return'
577        A C function has returned. arg is None.
578    'c_exception'
579        A C function has thrown an exception. arg is None.
580    
581    Note that as an exception is propagated down the chain of callers, an 
582    'exception' event is generated at each level.
583  
584  - sys.gettrace
585    Get the trace function as set by settrace().
586    
587  - sys.call_tracing
588    call_tracing(func, args) -> object
589
590    Call func(*args), while tracing is enabled.  The tracing state is
591    saved, and restored afterwards.  This is intended to be called from
592    a debugger from a checkpoint, to recursively debug some other code.
593
594-------------------------------------------------------------------------------
595TEST CASE STRUCTURE
596Test cases will be stored directly in the pydoc strings of the functions 
597implementing the tests.  In turn, the modules containing these test case 
598functions can be found under the "debugging" test package. Generating pydoc for
599the "debugging" test package and all sub-modules/sub-packages will generate 
600this test plan.
601
602-------------------------------------------------------------------------------
603SPEC REVIEW ISSUES
604There is no IronPython spec for this feature.  Any issues observed in the pdb 
605module, which acts as a spec, and reproducible under CPython should be reported 
606to http://bugs.python.org.
607
608-------------------------------------------------------------------------------
609TEST TOOLS
610One of the following:
611- reuse CLR's debugging test infrastructure.  NOTE: this is only suitable for 
612  mdbg tests
613- write our own new side-by-side testing infrastructure entirely in Python
614  so that we can eventually contribute pdb tests back to CPython
615- write our own new side-by-side testing infrastructure in any
616  language using any framework. http://sourceforge.net/projects/pexpect/ could
617  be useful as would be PowerShell
618- re-use some internal tool that might not be redistributable on CodePlex
619
620which will minimally need to support the following features:
621- ability to ignore lines (matching some regular expression) entirely
622- ability to substitute strings in lines
623- ability to run arbitrary commands (e.g., not limited to "ipy.exe ..." or "mdbg ...")
624- ability to match the current output against a reference output stored on disk
625- ability to run two or more arbitrary commands and compare their output against each other
626
627As per usual, the plan is to automate all tests using the SNAP checkin system
628and VS's mstest tool.
629
630-------------------------------------------------------------------------------
631SMOKE TEST (ACCEPTANCE TEST, BUILD VERIFICATION, ETC.)
632
633pdb acceptance test
634    ipy.exe -c "import pdb"
635
636mdbg
637    TODO (July 30, 2009)
638    
639Visual Studio IDE
640    TODO (July 30, 2009)
641
642
643-------------------------------------------------------------------------------
644AUTOMATED TESTS
645pdb and mdbg tests will be 100% automated in SNAP.
646
647
648-------------------------------------------------------------------------------
649MANUAL TESTS
650Setup testing will be manual for now, and we'll eventually move to automated 
651testing in SNAP using PowerShell support scripts.
652
653Visual Studio IDE testing is also manual for now.  Sometime in the 
654future we'll migrate to using the VS team's automation technologies.
655
656Manual testing will only occur for major IP releases and the 
657instructions will be documented on the internal IP website under the 
658"Release Process" wiki.
659
660
661-------------------------------------------------------------------------------
662REGRESSION TESTS
663Regression tests will be added to existing test modules under the "debugging" 
664package before any CodePlex or internal bug dealing with debugging is closed.
665As such, the majority of these regressions will be automated and run on every
666developer or test checkin.  Developers should preferably add a regression test
667for every bug fix, and Test will verify the sufficiency of the test, extending
668it if necessary.
669
670-------------------------------------------------------------------------------
671BUG BASHES
672While considered to be extremely useful, we have no current plans for internal 
673bug bashes on this feature.  Quite simply put, we need more IronPython headcount 
674and/or interest from the rest of Visual Studio Languages to accomplish this.
675
676-------------------------------------------------------------------------------
677BUG REPORTING
678All bug reports on this feature which do not include information consided to be
679Microsoft confidential are to be filed at 
680http://ironpython.codeplex.com/WorkItem/AdvancedList.aspx under the "Debugging"
681component using the bug template found at 
682http://ironpython.codeplex.com/Wiki/View.aspx?title=IronPython%20Bug%20Template.
683Bugs will be triaged on a weekly basis by the entire IronPython Team.
684
685-------------------------------------------------------------------------------
686PLAN CONTINGENCIES
687Without at least one additional SDET or other test tasks getting dropped, only 
688phases I and II of this plan are implementable prior to the release of 
689IP 2.6 RTM.
690
691-------------------------------------------------------------------------------
692EXTERNAL DEPENDENCIES
693We depend on the DLR for their implementation of Microsoft.Scripting.Debugging.dll 
694which forms the basis of our support of the CPython pdb module.  This plan 
695assumes that this DLL has been adequately tested.
696
697There are no teams/projects we know about which are taking a dependency on 
698IP's debugging functionality.
699
700-------------------------------------------------------------------------------
701HEADCOUNT REQUIREMENTS
702At least two FTE IronPython SDETs will be needed to implement this test plan 
703fully: one to handle general, day-to-day IP test operations
704(i.e., passes/investigations/issues/regressions, etc), and another to work exclusively on the 
705debugging feature.  A loose approximation is that the SDET working full-time
706on debugging would need anywhere from one to three months to completely 
707implement all aspects of this plan.  This estimate varies considerably 
708depending upon how much emphasis is to be given to pdb versus mdbg versus the 
709VS IDE.  Having a second full-time SDET on the debugging feature would have a
710not positive effect as the division of labor between testing command-line IP
711debugging support and VS IDE support is quite clear.
712
713-------------------------------------------------------------------------------
714PRODUCT SUPPORT
715IP is not a supported product in the conventional Microsoft sense. 
716Support for IP is freely given directly by the IP Team via the 
717IronPython Mailing List.
718
719-------------------------------------------------------------------------------
720DROP PROCEDURES
721All developer checkins must go through the SNAP checkin system => IronPython
722sources are always in a "good" state.  Test will build IP from sources
723using developer instructions.
724
725-------------------------------------------------------------------------------
726RELEASE PROCEDURES
727See the IP "Release Process" wiki on the internal IP website.
728
729-------------------------------------------------------------------------------
730ALIAS/NEWSGROUPS AND COMMUNICATION CHANNELS
731Major changes around IP debugging support should be announced on the 
732IronPython Mailing List, users@lists.ironpython.com.
733
734-------------------------------------------------------------------------------
735REGULAR MEETINGS
736- FEATURE TEAM MEETINGS
737  There is no feature team for debugging.
738- PROJECT TEST TEAM MEETINGS
739  IronPython weekly team meeting. Day and time are subject to change.
740  We'll triage debugging work items during this meeting and discuss IP 
741  debugging issues as necessary.
742- FEATURE TEAM TEST MEETINGS
743  There is no feature test team for debugging.
744
745-------------------------------------------------------------------------------
746DECISION MAKING PROCEDURES
747Decisions on this feature will be driven by the following criteria:
748- Microsoft business needs
749- IronPython Community feature requests
750- Previous decisions made by CPython. E.g., the pdb module
751- Existing debugging support in Microsoft products such as VS
752In the event a conscensus cannot be reached by the team on something, Jim
753Hugunin's input should be sought.
754
755Miscellaneous Procedures:
756- the gopackage tool is responsible for validating all source pushes to 
757  CodePlex
758- at least two IP team members need to sanity check public builds
759  before they can be released
760- full test pass signoff on this feature will be needed for every public, signed
761  release
762- bug triages will be performed with all three software disciplines in attendance
763- development design should be reviewed both by the PM and Test disciplines
764
765-------------------------------------------------------------------------------
766NOTES
767
768MDBG TEST CASE WRITING TIPS
769- use different language structures as the last line in the Python file
770- when only one line lives inside a block of code, but the code should not be
771  stepped through
772- correct step-through: no extra step-throughs or lack thereof
773- 
774'''