PageRenderTime 69ms CodeModel.GetById 62ms app.highlight 3ms RepoModel.GetById 1ms app.codeStats 0ms

/README.txt

http://testability-explorer.googlecode.com/
Plain Text | 457 lines | 366 code | 91 blank | 0 comment | 0 complexity | 4e84529dd4b1399f4bc1cd66ad24cb74 MD5 | raw file
  1#summary Readme first
  2=Testability Metric Explained=
  3
  4==Basic Idea==
  5The testability score represents the difficulty of testing a particular
  6piece of code. The difficulty is a measure of:
  7  # how hard it will be to construct this object
  8  # how hard will it be to execute the method in order to be able to assert something in a test.
  9
 10==How to run it==
 11{{{
 12$ testability.sh 
 13Argument "classes and packages" is required
 14
 15 classes and packages                  : Classes or packages to analyze. Matches any class starting with these.
 16                                         Ex. com.example.analyze.these com.google.and.these.packages com.google.AClass
 17 -cp VAL                               : colon delimited classpath to analyze (jars or directories)
 18                                         Ex. lib/one.jar:lib/two.jar
 19 -maxAcceptableCost N                  : Maximum Total Class cost to be classify it as 'acceptable'.
 20 -maxExcellentCost N                   : Maximum Total Class cost to be classify it as 'excellent'.
 21 -minCost N                            : Minimum Total Class cost required to print that class' metrics.
 22 -print VAL                            : summary: (default) print package summary information.
 23                                         detail: print detail drill down information for each method call.
 24 -printDepth N                         : Maximum depth to recurse and print costs of classes/methods that the classes un
 25                                         der analysis depend on. Defaults to 0.
 26 -whitelist VAL                        : colon delimited whitelisted packages that will not count against you. Matches p
 27                                         ackages/classes starting with given values. (Always whitelists java.*. RegExp O
 28                                         K.)
 29 -worstOffenderCount N                 : Print N number of worst offending classes.
 30 cyclomatic cyclomatic cost multiplier : When computing the overall cost of the method the individual costs are added us
 31                                         ing weighted average. This represents the weight of the cyclomatic cost.
 32 global global state cost multiplier   : When computing the overall cost of the method the individual costs are added us
 33                                         ing weighted average. This represents the weight of the global state cost.
 34}}}
 35
 36
 37==Output Format==
 38
 39There are two kinds of output format: summary and detail
 40
 41=== Output Format: summary ===
 42
 43Use the summary mode to get a quick overview of the project. This example shows what running the testability metric
 44on itself produces. The top shows statistical breakdown of "Excellent", "Good" and "Needs work" classes.
 45It is followed by a breakdown chart showing the breakdown visually. A more detailed breakdown is shown in the
 46histogram. Finally the list of 20 highest offending classes is shown sorted by test difficulty.
 47
 48{{{
 49$ testability.sh -print summary com.google.test.metric -whitelist com.google.test.org.objectweb.asm.
 50      Analyzed classes:   125
 51 Excellent classes (.):   123  98.4%
 52      Good classes (=):     0   0.0%
 53Needs work classes (@):     2   1.6%
 54             Breakdown: [..................................................]
 55       0                                                                     98
 56     3 |......................................................................:    98
 57     9 |.........                                                             :    12
 58    15 |......                                                                :     8
 59    21 |..                                                                    :     2
 60    27 |...                                                                   :     3
 61    33 |                                                                      :     0
 62    39 |                                                                      :     0
 63    45 |                                                                      :     0
 64    51 |                                                                      :     0
 65    57 |                                                                      :     0
 66    63 |                                                                      :     0
 67    69 |                                                                      :     0
 68    75 |                                                                      :     0
 69    81 |                                                                      :     0
 70    87 |                                                                      :     0
 71    93 |                                                                      :     0
 72    99 |                                                                      :     0
 73   105 |                                                                      :     0
 74   111 |@                                                                     :     1
 75   117 |                                                                      :     0
 76   123 |                                                                      :     0
 77   129 |                                                                      :     0
 78   135 |                                                                      :     0
 79   141 |                                                                      :     0
 80   147 |@                                                                     :     1
 81
 82Highest Cost
 83============
 84com.google.test.metric.Testability 148
 85com.google.test.metric.asm.MethodVisitorBuilder 113
 86com.google.test.metric.method.BlockDecomposer 29
 87com.google.test.metric.asm.MethodVisitorBuilder$GetFieldRunnable 27
 88com.google.test.metric.asm.MethodVisitorBuilder$PutFieldRunnable 27
 89com.google.test.metric.asm.MethodVisitorBuilder$2 23
 90com.google.test.metric.Type 22
 91com.google.test.metric.asm.ClassInfoBuilderVisitor 18
 92com.google.test.metric.asm.FieldVisitorBuilder 18
 93com.google.test.metric.MetricComputer 17
 94com.google.test.metric.asm.MethodVisitorBuilder$30 17
 95com.google.test.metric.asm.MethodVisitorBuilder$12 16
 96com.google.test.metric.collection.KeyedMultiStack 15
 97com.google.test.metric.method.BlockDecomposer$1 14
 98com.google.test.metric.method.op.turing.MethodInvocation 14
 99com.google.test.metric.asm.MethodVisitorBuilder$28 12
100com.google.test.metric.asm.SignatureParser 12
101com.google.test.metric.asm.SignatureParser$TypeVisitor 12
102com.google.test.metric.report.TextReport 12
103com.google.test.metric.asm.MethodVisitorBuilder$7 10
104}}}
105
106=== Output Format: detail ===
107
108To get a more in depth view of view a particular class has a high cost use the detail output format.
109
110{{{
111$ testability.sh -print detail com.google.test.metric.example.Lessons.Primeness
112-----------------------------------------
113Packages/Classes To Enter: 
114  com.google.test.metric.example.Lessons.Primeness*
115Max Method Print Depth: 1
116Min Class Cost: 1
117-----------------------------------------
118
119Testability cost for com.google.test.metric.example.Lessons.Primeness [ cost = 2 ] [ 2 TCC, 0 TGC ]
120  com.google.test.metric.example.Lessons.Primeness.isPrime(I)Z [2, 0 / 2, 0]
121}}}
122
123An example score for a method is:
124  `package.Class.methodName()V[1, 2 / 3, 4]`
125
126The four numbers, in order, represent this method's:
127  # _Testability Complexity_:
128  # _Global State Complexity_:
129  # _Total Testability Complexity_:
130  # _Total Global State Complexity_: 
131
132==Simplest Example==
133Let's start with a simple example of analyzing a simple class.
134
135*SOURCE:*
136{{{
137public class Primeness {
138  public boolean isPrime(int number) {
139    for (int i = 2; i < number / 2; i++) {
140      if (number % i == 0) {
141        return false;
142      }
143    }
144    return true;
145  }
146}
147}}}
148*TRY IT:
149   `testability.sh -printDepth 10 com.google.test.metric.example.Lessons.Primeness`
150
151*OUTPUT:*
152{{{
153-----------------------------------------
154Packages/Classes To Enter: 
155 com.google.test.metric.example.Lessons.Primeness*
156-----------------------------------------
157
158
159Testability cost for com.google.test.metric.example.Lessons.Primeness [ 2 TCC, 0 TGC ]
160  com.google.test.metric.example.Lessons.Primeness.isPrime(I)Z [2, 0 / 2, 0]
161
162-----------------------------------------
163Summary Statistics:
164 TCC for all classes entered: 2
165 TGC for all classes entered: 0
166 Average TCC for all classes entered: 2.00
167 Average TGC for all classes entered: 0.00
168
169Key:
170 TCC: Total Compexity Cost
171 TGC: Total Global Cost
172
173Analyzed 1 classes (plus non-whitelisted external dependencies)
174}}}
175
176*EXPLANATION:*
177
178In the above example the test complexity is 2. This is because the
179method `isPrime` has a loop and an `if` statement. Therefore there are 2
180additional paths of execution for a total of 3.
181 # Loop does not execute
182 # Loop executes but if does not evaluate to true
183 # Loop executes and if evaluates to true.
184
185*Note:* Test cost is the method's cyclomatic complexity minus one. Subtract one
186to not penalize the method for decomposing the method into 2 smaller methods. 
187(If the lowest cost would be 1, splitting the method into two would result in 
188the cost of 2.) The simplest method's score is 0 such that the method can be
189split into smaller methods with no penalty.
190
191==Example: Injectability Scoring==
192This example shows the differences in scores based on how injectable a class is.
193`SumOfPrimes1` directly instantiates a `new Primeness()`. The `new` operator
194prevents you from being able to inject in a different subclass of `Primeness`
195for testing. Thus the scores differ:
196  * `sum(I)I[2, 0 / 4, 0]` <- total test complexity of 4 for `SumOfPrimes1`
197  * `sum(I)I[2, 0 / 2, 0]` <- total test complexity of 2 for `SumOfPrimes2`
198
199*SOURCE:*
200{{{
201public class SumOfPrimes1 {
202
203  private Primeness primeness = new Primeness();
204  
205  public int sum(int max) {
206    int sum = 0;
207    for (int i = 0; i < max; i++) {
208      if (primeness.isPrime(i)) {
209        sum += i;
210      }
211    }
212    return sum;
213  }
214  
215}
216}}}
217
218*TRY IT:*
219   `testability.sh -printDepth 10 com.google.test.metric.example.Lessons.SumOfPrimes1`
220
221*OUTPUT:*
222{{{
223-----------------------------------------
224Packages/Classes To Enter: 
225 com.google.test.metric.example.Lessons.SumOfPrimes1*
226-----------------------------------------
227
228
229Testability cost for com.google.test.metric.example.Lessons.SumOfPrimes1 [ 4 TCC, 0 TGC ]
230  com.google.test.metric.example.Lessons.SumOfPrimes1.sum(I)I [2, 0 / 4, 0]
231    line 25: com.google.test.metric.example.Lessons.Primeness.isPrime(I)Z [2, 0 / 2, 0]
232
233-----------------------------------------
234Summary Statistics:
235 TCC for all classes entered: 4
236 TGC for all classes entered: 0
237 Average TCC for all classes entered: 4.00
238 Average TGC for all classes entered: 0.00
239
240Key:
241 TCC: Total Compexity Cost
242 TGC: Total Global Cost
243
244Analyzed 1 classes (plus non-whitelisted external dependencies)
245}}}
246
247The testability and global state costs for the constructor (indicated by <init>)
248is zero. 
249
250The testability complexity of `sum` is 2, and the total testability complexity
251is 4. The 2 comes from `sum` directly. The total of 4 is `sum`'s 2 plus 2 from 
252`isPrime`.  The reason for this is that there is no way for a test to execute
253the `sum` method and intercept the call to `isPrime` method. In other
254words there is no way to write a true unit test which will test only the class
255`SumOfPrimes1`. In this case it may not be a problem as the `isPrime` method is
256not very expensive (in terms of complexity), however if the `isPrime` method
257represented something expensive, like an external system, this would possess a
258testing problem as there would be no way to test `sum` method without incurring
259the cost of `isPrime`.
260
261*SOURCE:*
262{{{
263public class SumOfPrimes2 {
264
265  private final Primeness primeness;
266  
267  public SumOfPrimes2(Primeness primeness) {
268    this.primeness = primeness;
269  }
270
271  public int sum(int max) {
272    int sum = 0;
273    for (int i = 0; i < max; i++) {
274      if (primeness.isPrime(i)) {
275        sum += i;
276      }
277    }
278    return sum;
279  }
280
281}
282}}}
283
284*TRY IT:*
285   `testability.sh -printDepth 10 com.google.test.metric.example.Lessons.SumOfPrimes2`
286
287*OUTPUT:*
288{{{ 
289-----------------------------------------
290Packages/Classes To Enter: 
291 com.google.test.metric.example.Lessons.SumOfPrimes2*
292-----------------------------------------
293
294
295Testability cost for com.google.test.metric.example.Lessons.SumOfPrimes2 [ 2 TCC, 0 TGC ]
296  com.google.test.metric.example.Lessons.SumOfPrimes2.sum(I)I [2, 0 / 2, 0]
297
298-----------------------------------------
299Summary Statistics:
300 TCC for all classes entered: 2
301 TGC for all classes entered: 0
302 Average TCC for all classes entered: 2.00
303 Average TGC for all classes entered: 0.00
304
305Key:
306 TCC: Total Compexity Cost
307 TGC: Total Global Cost
308
309Analyzed 1 classes (plus non-whitelisted external dependencies)
310}}}
311
312In this case `Primeness` is injected the into the constructor of the 
313`SumOfPrimes2`. As a result the cost of the `sum` method remains at 2, but the
314cost of `isPrime` is now 0. (A mock of `Primeness` could be injected to test
315`SumOfPrimes2`).
316
317The cost of construction is added to the cost of testing the class. (You can 
318only test a class which you can construct). In order to compute the cost we go 
319through several phases:
320  # Compute the cost of the constructor, giving zero cost to injectable variables 
321  # Look for setter methods and use those to mark more fields as injectable.
322  # Compute the cost of the method while respecting the injectability of fields, and method parameters.
323
324
325==Injectability==
326
327A variable (local variable, field or a parameter) is considered injectable if it
328can be set from the outside (i.e. in the test). Any variable assigned from an
329injectable variable is also considered injectable. In other words injectability 
330is transitive. An Injectable variable can be replaced by a mock in test. Any 
331method dispatched on an injectable variable has no cost. (It can be intercepted). 
332
333_(Caveat: The method can not be static, private, or final, as those methods can not be 
334overridden)._ 
335
336==Example: Global State==
337
338Global state makes it hard to tests ones code as it allows cross talk between
339test. This makes it so that tests can pass by themselves but fail when run in 
340a suite. It is also possible to make tests which will run in only a specific
341order.
342
343*SOURCE:*
344{{{
345package com.google.test.metric.example;
346
347public class GlobalExample {
348
349  public static class Gadget {
350    private final String id;
351    private int count;
352
353    public Gadget(String id, int count) {
354      this.id = id;
355      this.count = count;
356    }
357
358    public String getId() {
359      return id;
360    }
361
362    public int getCount() {
363      return count;
364    }
365
366    public int increment() {
367      return ++count;
368    }
369  }
370
371  public static class Globals {
372    public static final Gadget instance = new Gadget("Global", 1);
373  }
374
375  public Gadget getInstance() {
376    return Globals.instance;
377  }
378
379  public String getGlobalId() {
380    return Globals.instance.getId();
381  }
382
383  public int getGlobalCount() {
384    return Globals.instance.getCount();
385  }
386
387  public int globalIncrement() {
388    return Globals.instance.increment();
389  }
390}
391}}}
392
393*TRY IT:*
394   `testability.sh -printDepth 10 com.google.test.metric.example.GlobalExample com.google.test.metric.example.GlobalExample`
395
396*OUTPUT:*
397
398{{{
399-----------------------------------------
400Packages/Classes To Enter: 
401 com.google.test.metric.example.GlobalExample*
402-----------------------------------------
403
404
405Testability cost for com.google.test.metric.example.GlobalExample [ 0 TCC, 2 TGC ]
406  com.google.test.metric.example.GlobalExample.getGlobalCount()I [0, 0 / 0, 1]
407    line 55: com.google.test.metric.example.GlobalExample$Gadget.getCount()I [0, 1 / 0, 1]
408  com.google.test.metric.example.GlobalExample.globalIncrement()I [0, 0 / 0, 1]
409    line 59: com.google.test.metric.example.GlobalExample$Gadget.increment()I [0, 1 / 0, 1]
410
411Testability cost for com.google.test.metric.example.GlobalExample$Globals [ 0 TCC, 3 TGC ]
412  com.google.test.metric.example.GlobalExample$Globals.<init>()V [0, 0 / 0, 1]
413    line 43: com.google.test.metric.example.GlobalExample$Globals.<clinit>()V [0, 1 / 0, 1]
414  com.google.test.metric.example.GlobalExample$Globals.<clinit>()V [0, 2 / 0, 2]
415
416-----------------------------------------
417Summary Statistics:
418 TCC for all classes entered: 0
419 TGC for all classes entered: 5
420 Average TCC for all classes entered: 0.00
421 Average TGC for all classes entered: 1.67
422
423Key:
424 TCC: Total Compexity Cost
425 TGC: Total Global Cost
426
427Analyzed 3 classes (plus non-whitelisted external dependencies)
428}}}
429
430===`getGlobalId()`===
431
432Method `getGlobalId()` has no global cost. This may be surprising given that
433it accesses static variables. However, upon closer examinations, these
434variables are not mutable (they are declared `final`). For this reason there
435is no global mutable state associated with this method.
436
437===`getInstance()`===
438
439Similarly `getInstance()` has no global cost either as it only accesses
440variables which are final.
441
442===`getGlobalCount()`===
443
444Method `getGlobalCount()` accesses the `Globals.instance` which is a global
445constant. Accessing global constants is not a problem and hence does not 
446incur a cost. It then calls the `Gadget.getCount()` which accesses the `count`
447field. Because the `Gadget`'s `this` is a global constant, all of its
448fields are global as well. The global property is transitive. Because  `count`
449is mutable (no `final` keyword) reading `count` incurs a cost.
450
451===`globalIncrement()`===
452
453Method `getGlobalIncrement()` follows the same logic as `getGlobalCount()`.
454
455==Future Enhancements / Requests==
456Please talk about what you want on the mailing list:
457http://groups.google.com/group/testability-metrics