/README.txt
Plain Text | 457 lines | 366 code | 91 blank | 0 comment | 0 complexity | 4e84529dd4b1399f4bc1cd66ad24cb74 MD5 | raw file
1#summary Readme first 2=Testability Metric Explained= 3 4==Basic Idea== 5The testability score represents the difficulty of testing a particular 6piece of code. The difficulty is a measure of: 7 # how hard it will be to construct this object 8 # how hard will it be to execute the method in order to be able to assert something in a test. 9 10==How to run it== 11{{{ 12$ testability.sh 13Argument "classes and packages" is required 14 15 classes and packages : Classes or packages to analyze. Matches any class starting with these. 16 Ex. com.example.analyze.these com.google.and.these.packages com.google.AClass 17 -cp VAL : colon delimited classpath to analyze (jars or directories) 18 Ex. lib/one.jar:lib/two.jar 19 -maxAcceptableCost N : Maximum Total Class cost to be classify it as 'acceptable'. 20 -maxExcellentCost N : Maximum Total Class cost to be classify it as 'excellent'. 21 -minCost N : Minimum Total Class cost required to print that class' metrics. 22 -print VAL : summary: (default) print package summary information. 23 detail: print detail drill down information for each method call. 24 -printDepth N : Maximum depth to recurse and print costs of classes/methods that the classes un 25 der analysis depend on. Defaults to 0. 26 -whitelist VAL : colon delimited whitelisted packages that will not count against you. Matches p 27 ackages/classes starting with given values. (Always whitelists java.*. RegExp O 28 K.) 29 -worstOffenderCount N : Print N number of worst offending classes. 30 cyclomatic cyclomatic cost multiplier : When computing the overall cost of the method the individual costs are added us 31 ing weighted average. This represents the weight of the cyclomatic cost. 32 global global state cost multiplier : When computing the overall cost of the method the individual costs are added us 33 ing weighted average. This represents the weight of the global state cost. 34}}} 35 36 37==Output Format== 38 39There are two kinds of output format: summary and detail 40 41=== Output Format: summary === 42 43Use the summary mode to get a quick overview of the project. This example shows what running the testability metric 44on itself produces. The top shows statistical breakdown of "Excellent", "Good" and "Needs work" classes. 45It is followed by a breakdown chart showing the breakdown visually. A more detailed breakdown is shown in the 46histogram. Finally the list of 20 highest offending classes is shown sorted by test difficulty. 47 48{{{ 49$ testability.sh -print summary com.google.test.metric -whitelist com.google.test.org.objectweb.asm. 50 Analyzed classes: 125 51 Excellent classes (.): 123 98.4% 52 Good classes (=): 0 0.0% 53Needs work classes (@): 2 1.6% 54 Breakdown: [..................................................] 55 0 98 56 3 |......................................................................: 98 57 9 |......... : 12 58 15 |...... : 8 59 21 |.. : 2 60 27 |... : 3 61 33 | : 0 62 39 | : 0 63 45 | : 0 64 51 | : 0 65 57 | : 0 66 63 | : 0 67 69 | : 0 68 75 | : 0 69 81 | : 0 70 87 | : 0 71 93 | : 0 72 99 | : 0 73 105 | : 0 74 111 |@ : 1 75 117 | : 0 76 123 | : 0 77 129 | : 0 78 135 | : 0 79 141 | : 0 80 147 |@ : 1 81 82Highest Cost 83============ 84com.google.test.metric.Testability 148 85com.google.test.metric.asm.MethodVisitorBuilder 113 86com.google.test.metric.method.BlockDecomposer 29 87com.google.test.metric.asm.MethodVisitorBuilder$GetFieldRunnable 27 88com.google.test.metric.asm.MethodVisitorBuilder$PutFieldRunnable 27 89com.google.test.metric.asm.MethodVisitorBuilder$2 23 90com.google.test.metric.Type 22 91com.google.test.metric.asm.ClassInfoBuilderVisitor 18 92com.google.test.metric.asm.FieldVisitorBuilder 18 93com.google.test.metric.MetricComputer 17 94com.google.test.metric.asm.MethodVisitorBuilder$30 17 95com.google.test.metric.asm.MethodVisitorBuilder$12 16 96com.google.test.metric.collection.KeyedMultiStack 15 97com.google.test.metric.method.BlockDecomposer$1 14 98com.google.test.metric.method.op.turing.MethodInvocation 14 99com.google.test.metric.asm.MethodVisitorBuilder$28 12 100com.google.test.metric.asm.SignatureParser 12 101com.google.test.metric.asm.SignatureParser$TypeVisitor 12 102com.google.test.metric.report.TextReport 12 103com.google.test.metric.asm.MethodVisitorBuilder$7 10 104}}} 105 106=== Output Format: detail === 107 108To get a more in depth view of view a particular class has a high cost use the detail output format. 109 110{{{ 111$ testability.sh -print detail com.google.test.metric.example.Lessons.Primeness 112----------------------------------------- 113Packages/Classes To Enter: 114 com.google.test.metric.example.Lessons.Primeness* 115Max Method Print Depth: 1 116Min Class Cost: 1 117----------------------------------------- 118 119Testability cost for com.google.test.metric.example.Lessons.Primeness [ cost = 2 ] [ 2 TCC, 0 TGC ] 120 com.google.test.metric.example.Lessons.Primeness.isPrime(I)Z [2, 0 / 2, 0] 121}}} 122 123An example score for a method is: 124 `package.Class.methodName()V[1, 2 / 3, 4]` 125 126The four numbers, in order, represent this method's: 127 # _Testability Complexity_: 128 # _Global State Complexity_: 129 # _Total Testability Complexity_: 130 # _Total Global State Complexity_: 131 132==Simplest Example== 133Let's start with a simple example of analyzing a simple class. 134 135*SOURCE:* 136{{{ 137public class Primeness { 138 public boolean isPrime(int number) { 139 for (int i = 2; i < number / 2; i++) { 140 if (number % i == 0) { 141 return false; 142 } 143 } 144 return true; 145 } 146} 147}}} 148*TRY IT: 149 `testability.sh -printDepth 10 com.google.test.metric.example.Lessons.Primeness` 150 151*OUTPUT:* 152{{{ 153----------------------------------------- 154Packages/Classes To Enter: 155 com.google.test.metric.example.Lessons.Primeness* 156----------------------------------------- 157 158 159Testability cost for com.google.test.metric.example.Lessons.Primeness [ 2 TCC, 0 TGC ] 160 com.google.test.metric.example.Lessons.Primeness.isPrime(I)Z [2, 0 / 2, 0] 161 162----------------------------------------- 163Summary Statistics: 164 TCC for all classes entered: 2 165 TGC for all classes entered: 0 166 Average TCC for all classes entered: 2.00 167 Average TGC for all classes entered: 0.00 168 169Key: 170 TCC: Total Compexity Cost 171 TGC: Total Global Cost 172 173Analyzed 1 classes (plus non-whitelisted external dependencies) 174}}} 175 176*EXPLANATION:* 177 178In the above example the test complexity is 2. This is because the 179method `isPrime` has a loop and an `if` statement. Therefore there are 2 180additional paths of execution for a total of 3. 181 # Loop does not execute 182 # Loop executes but if does not evaluate to true 183 # Loop executes and if evaluates to true. 184 185*Note:* Test cost is the method's cyclomatic complexity minus one. Subtract one 186to not penalize the method for decomposing the method into 2 smaller methods. 187(If the lowest cost would be 1, splitting the method into two would result in 188the cost of 2.) The simplest method's score is 0 such that the method can be 189split into smaller methods with no penalty. 190 191==Example: Injectability Scoring== 192This example shows the differences in scores based on how injectable a class is. 193`SumOfPrimes1` directly instantiates a `new Primeness()`. The `new` operator 194prevents you from being able to inject in a different subclass of `Primeness` 195for testing. Thus the scores differ: 196 * `sum(I)I[2, 0 / 4, 0]` <- total test complexity of 4 for `SumOfPrimes1` 197 * `sum(I)I[2, 0 / 2, 0]` <- total test complexity of 2 for `SumOfPrimes2` 198 199*SOURCE:* 200{{{ 201public class SumOfPrimes1 { 202 203 private Primeness primeness = new Primeness(); 204 205 public int sum(int max) { 206 int sum = 0; 207 for (int i = 0; i < max; i++) { 208 if (primeness.isPrime(i)) { 209 sum += i; 210 } 211 } 212 return sum; 213 } 214 215} 216}}} 217 218*TRY IT:* 219 `testability.sh -printDepth 10 com.google.test.metric.example.Lessons.SumOfPrimes1` 220 221*OUTPUT:* 222{{{ 223----------------------------------------- 224Packages/Classes To Enter: 225 com.google.test.metric.example.Lessons.SumOfPrimes1* 226----------------------------------------- 227 228 229Testability cost for com.google.test.metric.example.Lessons.SumOfPrimes1 [ 4 TCC, 0 TGC ] 230 com.google.test.metric.example.Lessons.SumOfPrimes1.sum(I)I [2, 0 / 4, 0] 231 line 25: com.google.test.metric.example.Lessons.Primeness.isPrime(I)Z [2, 0 / 2, 0] 232 233----------------------------------------- 234Summary Statistics: 235 TCC for all classes entered: 4 236 TGC for all classes entered: 0 237 Average TCC for all classes entered: 4.00 238 Average TGC for all classes entered: 0.00 239 240Key: 241 TCC: Total Compexity Cost 242 TGC: Total Global Cost 243 244Analyzed 1 classes (plus non-whitelisted external dependencies) 245}}} 246 247The testability and global state costs for the constructor (indicated by <init>) 248is zero. 249 250The testability complexity of `sum` is 2, and the total testability complexity 251is 4. The 2 comes from `sum` directly. The total of 4 is `sum`'s 2 plus 2 from 252`isPrime`. The reason for this is that there is no way for a test to execute 253the `sum` method and intercept the call to `isPrime` method. In other 254words there is no way to write a true unit test which will test only the class 255`SumOfPrimes1`. In this case it may not be a problem as the `isPrime` method is 256not very expensive (in terms of complexity), however if the `isPrime` method 257represented something expensive, like an external system, this would possess a 258testing problem as there would be no way to test `sum` method without incurring 259the cost of `isPrime`. 260 261*SOURCE:* 262{{{ 263public class SumOfPrimes2 { 264 265 private final Primeness primeness; 266 267 public SumOfPrimes2(Primeness primeness) { 268 this.primeness = primeness; 269 } 270 271 public int sum(int max) { 272 int sum = 0; 273 for (int i = 0; i < max; i++) { 274 if (primeness.isPrime(i)) { 275 sum += i; 276 } 277 } 278 return sum; 279 } 280 281} 282}}} 283 284*TRY IT:* 285 `testability.sh -printDepth 10 com.google.test.metric.example.Lessons.SumOfPrimes2` 286 287*OUTPUT:* 288{{{ 289----------------------------------------- 290Packages/Classes To Enter: 291 com.google.test.metric.example.Lessons.SumOfPrimes2* 292----------------------------------------- 293 294 295Testability cost for com.google.test.metric.example.Lessons.SumOfPrimes2 [ 2 TCC, 0 TGC ] 296 com.google.test.metric.example.Lessons.SumOfPrimes2.sum(I)I [2, 0 / 2, 0] 297 298----------------------------------------- 299Summary Statistics: 300 TCC for all classes entered: 2 301 TGC for all classes entered: 0 302 Average TCC for all classes entered: 2.00 303 Average TGC for all classes entered: 0.00 304 305Key: 306 TCC: Total Compexity Cost 307 TGC: Total Global Cost 308 309Analyzed 1 classes (plus non-whitelisted external dependencies) 310}}} 311 312In this case `Primeness` is injected the into the constructor of the 313`SumOfPrimes2`. As a result the cost of the `sum` method remains at 2, but the 314cost of `isPrime` is now 0. (A mock of `Primeness` could be injected to test 315`SumOfPrimes2`). 316 317The cost of construction is added to the cost of testing the class. (You can 318only test a class which you can construct). In order to compute the cost we go 319through several phases: 320 # Compute the cost of the constructor, giving zero cost to injectable variables 321 # Look for setter methods and use those to mark more fields as injectable. 322 # Compute the cost of the method while respecting the injectability of fields, and method parameters. 323 324 325==Injectability== 326 327A variable (local variable, field or a parameter) is considered injectable if it 328can be set from the outside (i.e. in the test). Any variable assigned from an 329injectable variable is also considered injectable. In other words injectability 330is transitive. An Injectable variable can be replaced by a mock in test. Any 331method dispatched on an injectable variable has no cost. (It can be intercepted). 332 333_(Caveat: The method can not be static, private, or final, as those methods can not be 334overridden)._ 335 336==Example: Global State== 337 338Global state makes it hard to tests ones code as it allows cross talk between 339test. This makes it so that tests can pass by themselves but fail when run in 340a suite. It is also possible to make tests which will run in only a specific 341order. 342 343*SOURCE:* 344{{{ 345package com.google.test.metric.example; 346 347public class GlobalExample { 348 349 public static class Gadget { 350 private final String id; 351 private int count; 352 353 public Gadget(String id, int count) { 354 this.id = id; 355 this.count = count; 356 } 357 358 public String getId() { 359 return id; 360 } 361 362 public int getCount() { 363 return count; 364 } 365 366 public int increment() { 367 return ++count; 368 } 369 } 370 371 public static class Globals { 372 public static final Gadget instance = new Gadget("Global", 1); 373 } 374 375 public Gadget getInstance() { 376 return Globals.instance; 377 } 378 379 public String getGlobalId() { 380 return Globals.instance.getId(); 381 } 382 383 public int getGlobalCount() { 384 return Globals.instance.getCount(); 385 } 386 387 public int globalIncrement() { 388 return Globals.instance.increment(); 389 } 390} 391}}} 392 393*TRY IT:* 394 `testability.sh -printDepth 10 com.google.test.metric.example.GlobalExample com.google.test.metric.example.GlobalExample` 395 396*OUTPUT:* 397 398{{{ 399----------------------------------------- 400Packages/Classes To Enter: 401 com.google.test.metric.example.GlobalExample* 402----------------------------------------- 403 404 405Testability cost for com.google.test.metric.example.GlobalExample [ 0 TCC, 2 TGC ] 406 com.google.test.metric.example.GlobalExample.getGlobalCount()I [0, 0 / 0, 1] 407 line 55: com.google.test.metric.example.GlobalExample$Gadget.getCount()I [0, 1 / 0, 1] 408 com.google.test.metric.example.GlobalExample.globalIncrement()I [0, 0 / 0, 1] 409 line 59: com.google.test.metric.example.GlobalExample$Gadget.increment()I [0, 1 / 0, 1] 410 411Testability cost for com.google.test.metric.example.GlobalExample$Globals [ 0 TCC, 3 TGC ] 412 com.google.test.metric.example.GlobalExample$Globals.<init>()V [0, 0 / 0, 1] 413 line 43: com.google.test.metric.example.GlobalExample$Globals.<clinit>()V [0, 1 / 0, 1] 414 com.google.test.metric.example.GlobalExample$Globals.<clinit>()V [0, 2 / 0, 2] 415 416----------------------------------------- 417Summary Statistics: 418 TCC for all classes entered: 0 419 TGC for all classes entered: 5 420 Average TCC for all classes entered: 0.00 421 Average TGC for all classes entered: 1.67 422 423Key: 424 TCC: Total Compexity Cost 425 TGC: Total Global Cost 426 427Analyzed 3 classes (plus non-whitelisted external dependencies) 428}}} 429 430===`getGlobalId()`=== 431 432Method `getGlobalId()` has no global cost. This may be surprising given that 433it accesses static variables. However, upon closer examinations, these 434variables are not mutable (they are declared `final`). For this reason there 435is no global mutable state associated with this method. 436 437===`getInstance()`=== 438 439Similarly `getInstance()` has no global cost either as it only accesses 440variables which are final. 441 442===`getGlobalCount()`=== 443 444Method `getGlobalCount()` accesses the `Globals.instance` which is a global 445constant. Accessing global constants is not a problem and hence does not 446incur a cost. It then calls the `Gadget.getCount()` which accesses the `count` 447field. Because the `Gadget`'s `this` is a global constant, all of its 448fields are global as well. The global property is transitive. Because `count` 449is mutable (no `final` keyword) reading `count` incurs a cost. 450 451===`globalIncrement()`=== 452 453Method `getGlobalIncrement()` follows the same logic as `getGlobalCount()`. 454 455==Future Enhancements / Requests== 456Please talk about what you want on the mailing list: 457http://groups.google.com/group/testability-metrics