PageRenderTime 478ms CodeModel.GetById 19ms RepoModel.GetById 0ms app.codeStats 0ms

/blog/2012/06/18/finding-oldest-firefox-code/index.html

https://github.com/indygreg/indygreg.github.com
HTML | 256 lines | 215 code | 30 blank | 11 comment | 0 complexity | 7545e3e5fa27d376c0e139f6d25e88ac MD5 | raw file
    

  1. 

  2. 

  3. 

  4. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
    5. 

  5.     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    6. 

  6. <!--
    7. 

  7. Design by Free CSS Templates
    8. 

  8. http://www.freecsstemplates.org
    9. 

  9. Released for free under a Creative Commons Attribution 2.5 License
    10. 

  10. 

  11. Name       : Pollinating  
    12. 

  12. Description: A two-column, fixed-width design with dark color scheme.
    13. 

  13. Version    : 1.0
    14. 

  14. Released   : 20101114
    15. 

  15. 

  16. -->
    17. 

  17. <html xmlns="http://www.w3.org/1999/xhtml">
    18. 

  18.   <head>
    19. 

  19.     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    20. 

  20.     
    21. 

  21.   <title>Gregory Szorc's Digital Home
    22. 

  22.   | Finding Oldest Firefox Code
    23. 

  23. </title>
    24. 

  24. <link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="/blog/feed" />
    25. 

  25. <link rel="alternate" type="application/atom+xml" title="Atom 1.0"
    26. 

  26. href="/blog/feed/atom" />
    27. 

  27. <link rel="stylesheet" href="/style/style.css" type="text/css" />
    28. 

  28. <link rel="stylesheet" href="/css/pygments_murphy.css" type="text/css" />
    29. 

  29. 

  30. 

  31.   </head>
    32. 

  32.   <body>
    33. 

  33.     <div id="wrapper">
    34. 

  34.       
    35. 

  35.   <div id="menu">
    36. 

  36.   <ul>
    37. 

  37.     <li><a href="/">Home</a></li>
    38. 

  38.     <li><a href="/blog/">Blog</a></li>
    39. 

  39.     <li><a href="/notes">Notes</a></li>
    40. 

  40.     <li><a href="/work.html">Work</a></li>
    41. 

  41.     <li><a href="/skills.html">Skills</a></li>
    42. 

  42.     <li><a href="/thoughts.html">Thoughts</a></li>
    43. 

  43.     <li><a href="/resume.pdf">Resume</a></li>
    44. 

  44.   </ul>
    45. 

  45. </div>
    46. 

  46. 

  47. 

  48.       <div id="page">
    49. 

  49.         <div id="page-bgtop">
    50. 

  50.           <div id="page-bgbtm">
    51. 

  51.               <div id="content">
    52. 

  52.                 
    53. 

  53. 

  54. <div class="blog_post">
    55. 

  55.   <a name="finding-oldest-firefox-code"></a>
    56. 

  56.   <h2 class="blog_post_title"><a href="/blog/2012/06/18/finding-oldest-firefox-code" rel="bookmark" title="Permanent Link to Finding Oldest Firefox Code">Finding Oldest Firefox Code</a></h2>
    57. 

  57.   <small>June 18, 2012 at 10:10 AM | categories: 
    58. 

  58. 

  59. <a href='/blog/category/mozilla'>Mozilla</a>
    60. 

  60. </small><p/>
    61. 

  61.   <div class="post_prose">
    62. 

  62.     
    63. 

  63.   <p>On Twitter the other night, Justin Dolske
    64. 

  64. <a href="https://twitter.com/dolske/status/214241105179451393">posed a question</a>:</p>
    65. 

  65. <pre><code>Weekend challenge: what is the oldest line of code still shipping
    66. 

  66. in Firefox (tiebreaker: largest contiguous chunk)?
    67. 

  67. </code></pre>
    68. 

  68. <p>Good question and good challenge!</p>
    69. 

  69. <h2>Technical Approach</h2>
    70. 

  70. <p>To solve this problem, I decided my first task would be to produce a
    71. 

  71. database holding line-by-line metadata for files <em>currently</em> in the
    72. 

  72. Firefox repository. This sounded like a difficult problem at first,
    73. 

  73. especially considering the Mercurial repository doesn't contain CVS
    74. 

  74. history and this would be needed to identify code older than Mercurial
    75. 

  75. that is <em>still</em> active.</p>
    76. 

  76. <p>Fortunately, there exists a
    77. 

  77. <a href="https://github.com/mozilla/mozilla-central/">Git repository</a> with full
    78. 

  78. history of mozilla-central, including CVS and Mercurial! Armed with a
    79. 

  79. clone of this repository, I wrote a quick shell <em>one-liner</em> to ascertain
    80. 

  80. the history of every line in the repository:</p>
    81. 

  81. <pre><code>for f in `git ls-tree -r --name-only HEAD`; do \
    82. 

  82.   echo "BEGIN_RECORD $f"; \
    83. 

  83.   git blame -l -t -M -C -n -w -p $f; \
    84. 

  84.   echo "END_RECORD $f"; \
    85. 

  85. done
    86. 

  86. </code></pre>
    87. 

  87. <p>The <em>git ls-tree</em> command prints the names of every file in
    88. 

  88. the current tree. This is basically doing <em>find . -type f</em> except for
    89. 

  89. files under version control by Git. <em>git blame</em> attempts to ascertain
    90. 

  90. the history of each line in a file. It is worth pointing out arguments
    91. 

  91. <em>-M</em> and <em>-C</em>. These attempt to find moves/copies of the line from
    92. 

  92. within the same commit. If these are omitted, simple refactoring such as
    93. 

  93. renaming a file or reordering code within a file would result in a
    94. 

  94. improper commit attribution. Basically, Git would associate the line
    95. 

  95. with the commit that changed it. With these flags, Git attempts to
    96. 

  96. complete the chain and find the true origin of the line (to some
    97. 

  97. degree).</p>
    98. 

  98. <p>Now, something I thought was really cool is <em>git blame</em>'s porcelain
    99. 

  99. output format (<em>-p</em>). Not only does it allow for relatively simple machine
    100. 

  100. readability of the output (yay), but it also compacts the output
    101. 

  101. so adjacent lines sharing the same history metadata share the
    102. 

  102. same metadata/header block. In other words, it solves Dolske's <em>largest
    103. 

  103. contiguous chunk</em> tiebreaker for free! Thanks, Git!</p>
    104. 

  104. <p>I should also say that <em>git blame</em> isn't perfect for attributing code.
    105. 

  105. But I think it is good enough to solve a Twitter challenge.</p>
    106. 

  106. <p>I piped the output of the above command into a file so I could have the
    107. 

  107. original data available to process. After all, this data is idempotent,
    108. 

  108. so it makes no sense to <em>not</em> save it. After running for a while, I noticed
    109. 

  109. things were running slower than I'd like. I think it took about 2 hours
    110. 

  110. to obtain info for ~5000 files. No good. I played around a bit and
    111. 

  111. realized the <em>-M</em> and <em>-C</em> flags were slowing things down. This is
    112. 

  112. expected. But, I really wanted this data for a comprehensive data
    113. 

  113. analysis.</p>
    114. 

  114. <p>I re-discovered <a href="https://www.gnu.org/software/parallel/">GNU Parallel</a>
    115. 

  115. and modified my one-liner to use <strong>all the cores</strong>:</p>
    116. 

  116. <pre><code>git ls-tree -r --name-only HEAD | \
    117. 

  117. parallel 'echo "BEGIN_RECORD {}"; git blame -l -t -M -C -n -w -p {}; echo "END_RECORD {}"'
    118. 

  118. </code></pre>
    119. 

  119. <p>This made things run <em>substantially</em> faster since I was now running on
    120. 

  120. all 8 cores, not just 1. With GNU Parallel, this simple kind of
    121. 

  121. parallelism is almost too easy. Now, I say <em>substantially</em> faster, but
    122. 

  122. overall execution is still slow. How slow? Well, on my Sandy Bridge Macbook
    123. 

  123. Pro:</p>
    124. 

  124. <pre><code>real  525m49.149s
    125. 

  125. user  3592m15.862s
    126. 

  126. sys   201m4.482s
    127. 

  127. </code></pre>
    128. 

  128. <p>8:45 wall time and nearly 60 hours of CPU time. Yeah, I'm surprised
    129. 

  129. my laptop didn't explode too! The output file was 1,099,071,448 bytes
    130. 

  130. uncompressed and 155,354,423 bzipped.</p>
    131. 

  131. <p>While Git was obtaining data, I set about writing a consumer and data
    132. 

  132. processor. I'm sure the wheel of parsing this <em>porcelain</em> output format
    133. 

  133. has been invented before. But, I hadn't coded any Perl in a while and
    134. 

  134. figured this was as good of an excuse as any!</p>
    135. 

  135. <p>The Perl script to do the parsing and data analysis is available at
    136. 

  136. <a href="https://gist.github.com/2945604">https://gist.github.com/2945604</a>.
    137. 

  137. The core parsing function simply calls a supplied callback whenever a
    138. 

  138. new <em>block</em> of code from the same commit/file is encountered.</p>
    139. 

  139. <p>I implemented a function that records a mapping of commit times to
    140. 

  140. blocks. Finally, I wrote a simple function to write the results.</p>
    141. 

  141. <h2>Results</h2>
    142. 

  142. <p>What did 60 hours of CPU time tell us? Well, the oldest recorded line
    143. 

  143. dates from 1998-03-28. This is actually the <em>Free the Lizard</em> commit -
    144. 

  144. the first commit of open source Gecko to CVS. From this commit (Git
    145. 

  145. commit 781c48087175615674 for those playing at home), a number of lines
    146. 

  146. linger, including code for <em>mkdepend</em> and <em>nsinstall</em>.</p>
    147. 

  147. <p>But, Dolske's question was about <em>shipping</em> code. Well, as far as I can
    148. 

  148. tell, the oldest shipping code in the tree honor is shared by the
    149. 

  149. following:</p>
    150. 

  150. <ul>
    151. 

  151. <li>js/jsd/jsd1640.rc (pretty much the whole file)</li>
    152. 

  152. <li>js/jsd/jsd3240.rc (ditto)</li>
    153. 

  153. <li>js/jsd/jsd_atom.c:47-73</li>
    154. 

  154. <li>js/src/jsfriendapi.cpp:388-400</li>
    155. 

  155. <li>js/src/yarr/YarrParser.h:295-523</li>
    156. 

  156. <li>media/libjpeg/jdarith.c:21-336</li>
    157. 

  157. <li>media/libjpeg/jctrans.c:20-178</li>
    158. 

  158. <li>media/libjpeg (misc files all have large blocks)</li>
    159. 

  159. <li>xpcom/components/nsComponentManager.cpp:897-901</li>
    160. 

  160. <li>gfx/src/nsTransform2D.h (a couple 3-4 line chunks)</li>
    161. 

  161. <li>toolkit/library/nsDllMain.cpp:22-31</li>
    162. 

  162. <li>widget/windows/nsUXThemeData.cpp:213-257</li>
    163. 

  163. <li>widget/windows/nsWindowGfx.cpp:756-815</li>
    164. 

  164. <li>xpcom/ds/nsUnicharBuffer.cpp:14-33</li>
    165. 

  165. <li>xpcom/glue/nsCRTGlue.cpp:128-174</li>
    166. 

  166. </ul>
    167. 

  167. <p>There are also a few small chunks of 1 or 2 lines in a couple dozen
    168. 

  168. other files from that initial commit.</p>
    169. 

  169. <h2>Further Analysis</h2>
    170. 

  170. <p>If anyone is interested in performing additional analysis, you can just
    171. 

  171. take my Gist and install your own <em>onBlock</em> and output formatting
    172. 

  172. functions! Of course, you'll need a data set. My code is written so it
    173. 

  173. will work with any Git repository. If you want to analyze Firefox, it
    174. 

  174. will take hours of CPU time to extract the data from Git. To save some
    175. 

  175. time, you can obtain a copy of the raw data from commit
    176. 

  176. <a href="http://people.mozilla.org/~gszorc/mc-blame-0b961fb702a9576cb456.bz2">0b961fb702a9576cb456809410209adbbb956bc8</a>.</p>
    177. 

  177. <p>There is certainly no shortage of interesting analysis that
    178. 

  178. can be performed over this data set. Some that come to mind are a
    179. 

  179. <em>scoreboard</em> for most lines of code in the current tree (friendly
    180. 

  180. competition, anyone?) and a breakdown of active lines by the period they
    181. 

  181. were introduced.</p>
    182. 

  182. <p>I'll leave more advanced analysis as an exercise for the reader.</p>
    183. 

    184. 

  184.   </div>
    185. 

  185. </div>
    186. 

  186. 

  187. 

  188. 

  189. 

  190.               </div>
    191. 

  191.               
    192. 

  192.           <div id="sidebar">
    193. 

  193.           <ul>
    194. 

  194.             <li>
    195. 

  195.               <h2>Categories</h2>
    196. 

  196.               <ul>
    197. 

  197.                 <li><a href="/blog/category/apple">Apple</a></li>
    198. 

  198.                 <li><a href="/blog/category/bugzilla">Bugzilla</a></li>
    199. 

  199.                 <li><a href="/blog/category/ci">CI</a></li>
    200. 

  200.                 <li><a href="/blog/category/clang">Clang</a></li>
    201. 

  201.                 <li><a href="/blog/category/docker">Docker</a></li>
    202. 

  202.                 <li><a href="/blog/category/firefox">Firefox</a></li>
    203. 

  203.                 <li><a href="/blog/category/git">Git</a></li>
    204. 

  204.                 <li><a href="/blog/category/javascript">JavaScript</a></li>
    205. 

  205.                 <li><a href="/blog/category/mercurial">Mercurial</a></li>
    206. 

  206.                 <li><a href="/blog/category/mozreview">MozReview</a></li>
    207. 

  207.                 <li><a href="/blog/category/mozilla">Mozilla</a></li>
    208. 

  208.                 <li><a href="/blog/category/personal">Personal</a></li>
    209. 

  209.                 <li><a href="/blog/category/programming">Programming</a></li>
    210. 

  210.                 <li><a href="/blog/category/puppet">Puppet</a></li>
    211. 

  211.                 <li><a href="/blog/category/pyoxidizer">PyOxidizer</a></li>
    212. 

  212.                 <li><a href="/blog/category/python">Python</a></li>
    213. 

  213.                 <li><a href="/blog/category/review-board">Review Board</a></li>
    214. 

  214.                 <li><a href="/blog/category/rust">Rust</a></li>
    215. 

  215.                 <li><a href="/blog/category/sync">Sync</a></li>
    216. 

  216.                 <li><a href="/blog/category/browsers">browsers</a></li>
    217. 

  217.                 <li><a href="/blog/category/build-system">build system</a></li>
    218. 

  218.                 <li><a href="/blog/category/code-review">code review</a></li>
    219. 

  219.                 <li><a href="/blog/category/compilers">compilers</a></li>
    220. 

  220.                 <li><a href="/blog/category/internet">internet</a></li>
    221. 

  221.                 <li><a href="/blog/category/logging">logging</a></li>
    222. 

  222.                 <li><a href="/blog/category/mach">mach</a></li>
    223. 

  223.                 <li><a href="/blog/category/make">make</a></li>
    224. 

  224.                 <li><a href="/blog/category/misc">misc</a></li>
    225. 

  225.                 <li><a href="/blog/category/movies">movies</a></li>
    226. 

  226.                 <li><a href="/blog/category/packaging">packaging</a></li>
    227. 

  227.                 <li><a href="/blog/category/pymake">pymake</a></li>
    228. 

  228.                 <li><a href="/blog/category/security">security</a></li>
    229. 

  229.                 <li><a href="/blog/category/sysadmin">sysadmin</a></li>
    230. 

  230.                 <li><a href="/blog/category/testing">testing</a></li>
    231. 

  231.               </ul>
    232. 

  232.             </li>
    233. 

  233.           </ul>
    234. 

  234.         </div>
    235. 

  235. 

  236. 

  237. 

  238.               <div style="clear: both;">&nbsp;</div>
    239. 

  239.           </div>
    240. 

  240.         </div>
    241. 

  241.       </div>
    242. 

  242.       <div id="footer">
    243. 

  243.         
    244. 

  244.   <hr/>
    245. 

  245.   <p>Copyright (c) 2012- Gregory Szorc. All rights reserved. Design by <a href="http://www.freecsstemplates.org/"> CSS Templates</a>.</p>
    246. 

  246. 

  247. 

  248.       </div>
    249. 

  249.     </div>
    250. 

  250.   </body>
    251. 

  251. </html>
    252. 

  252. 

  253. 

  254. 

  255. 

  256. 

  257.