/documentation/docs/lexer.html

http://github.com/jashkenas/coffee-script · HTML · 2024 lines · 1768 code · 256 blank · 0 comment · 0 complexity · 9c4487192ff7778f37ad5ad37144c273 MD5 · raw file

Large files are truncated click here to view the full file

    

  1. <!DOCTYPE html>
    2. 

  2. 

  3. <html>
    4. 

  4. <head>
    5. 

  5.   <title>lexer.coffee</title>
    6. 

  6.   <meta http-equiv="content-type" content="text/html; charset=UTF-8">
    7. 

  7.   <meta name="viewport" content="width=device-width, target-densitydpi=160dpi, initial-scale=1.0; maximum-scale=1.0; user-scalable=0;">
    8. 

  8.   <link rel="stylesheet" media="all" href="docco.css" />
    9. 

  9. </head>
    10. 

  10. <body>
    11. 

  11.   <div id="container">
    12. 

  12.     <div id="background"></div>
    13. 

  13.     
    14. 

  14.       <ul id="jump_to">
    15. 

  15.         <li>
    16. 

  16.           <a class="large" href="javascript:void(0);">Jump To &hellip;</a>
    17. 

  17.           <a class="small" href="javascript:void(0);">+</a>
    18. 

  18.           <div id="jump_wrapper">
    19. 

  19.           <div id="jump_page_wrapper">
    20. 

  20.             <div id="jump_page">
    21. 

  21.               
    22. 

  22.                 
    23. 

  23.                 <a class="source" href="browser.html">
    24. 

  24.                   browser.coffee
    25. 

  25.                 </a>
    26. 

  26.               
    27. 

  27.                 
    28. 

  28.                 <a class="source" href="cake.html">
    29. 

  29.                   cake.coffee
    30. 

  30.                 </a>
    31. 

  31.               
    32. 

  32.                 
    33. 

  33.                 <a class="source" href="coffee-script.html">
    34. 

  34.                   coffee-script.coffee
    35. 

  35.                 </a>
    36. 

  36.               
    37. 

  37.                 
    38. 

  38.                 <a class="source" href="command.html">
    39. 

  39.                   command.coffee
    40. 

  40.                 </a>
    41. 

  41.               
    42. 

  42.                 
    43. 

  43.                 <a class="source" href="grammar.html">
    44. 

  44.                   grammar.coffee
    45. 

  45.                 </a>
    46. 

  46.               
    47. 

  47.                 
    48. 

  48.                 <a class="source" href="helpers.html">
    49. 

  49.                   helpers.coffee
    50. 

  50.                 </a>
    51. 

  51.               
    52. 

  52.                 
    53. 

  53.                 <a class="source" href="index.html">
    54. 

  54.                   index.coffee
    55. 

  55.                 </a>
    56. 

  56.               
    57. 

  57.                 
    58. 

  58.                 <a class="source" href="lexer.html">
    59. 

  59.                   lexer.coffee
    60. 

  60.                 </a>
    61. 

  61.               
    62. 

  62.                 
    63. 

  63.                 <a class="source" href="nodes.html">
    64. 

  64.                   nodes.coffee
    65. 

  65.                 </a>
    66. 

  66.               
    67. 

  67.                 
    68. 

  68.                 <a class="source" href="optparse.html">
    69. 

  69.                   optparse.coffee
    70. 

  70.                 </a>
    71. 

  71.               
    72. 

  72.                 
    73. 

  73.                 <a class="source" href="register.html">
    74. 

  74.                   register.coffee
    75. 

  75.                 </a>
    76. 

  76.               
    77. 

  77.                 
    78. 

  78.                 <a class="source" href="repl.html">
    79. 

  79.                   repl.coffee
    80. 

  80.                 </a>
    81. 

  81.               
    82. 

  82.                 
    83. 

  83.                 <a class="source" href="rewriter.html">
    84. 

  84.                   rewriter.coffee
    85. 

  85.                 </a>
    86. 

  86.               
    87. 

  87.                 
    88. 

  88.                 <a class="source" href="scope.html">
    89. 

  89.                   scope.litcoffee
    90. 

  90.                 </a>
    91. 

  91.               
    92. 

  92.                 
    93. 

  93.                 <a class="source" href="sourcemap.html">
    94. 

  94.                   sourcemap.litcoffee
    95. 

  95.                 </a>
    96. 

  96.               
    97. 

  97.             </div>
    98. 

  98.           </div>
    99. 

  99.         </li>
    100. 

  100.       </ul>
    101. 

  101.     
    102. 

  102.     <ul class="sections">
    103. 

  103.         
    104. 

  104.           <li id="title">
    105. 

  105.               <div class="annotation">
    106. 

  106.                   <h1>lexer.coffee</h1>
    107. 

  107.               </div>
    108. 

  108.           </li>
    109. 

  109.         
    110. 

  110.         
    111. 

  111.         
    112. 

  112.         <li id="section-1">
    113. 

  113.             <div class="annotation">
    114. 

  114.               
    115. 

  115.               <div class="pilwrap ">
    116. 

  116.                 <a class="pilcrow" href="#section-1">&#182;</a>
    117. 

  117.               </div>
    118. 

  118.               <p>The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt
    119. 

  119. matches against the beginning of the source code. When a match is found,
    120. 

  120. a token is produced, we consume the match, and start again. Tokens are in the
    121. 

  121. form:</p>
    122. 

  122. <pre><code>[tag, value, locationData]
    123. 

  123. </code></pre><p>where locationData is {first_line, first_column, last_line, last_column}, which is a
    124. 

  124. format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>.  These
    125. 

  125. are read by jison in the <code>parser.lexer</code> function defined in coffee-script.coffee.</p>
    126. 

  126. 

  127.             </div>
    128. 

  128.             
    129. 

  129.             <div class="content"><div class='highlight'><pre>
    130. 

  130. {Rewriter, INVERSES} = <span class="hljs-built_in">require</span> <span class="hljs-string">'./rewriter'</span></pre></div></div>
    131. 

  131.             
    132. 

  132.         </li>
    133. 

  133.         
    134. 

  134.         
    135. 

  135.         <li id="section-2">
    136. 

  136.             <div class="annotation">
    137. 

  137.               
    138. 

  138.               <div class="pilwrap ">
    139. 

  139.                 <a class="pilcrow" href="#section-2">&#182;</a>
    140. 

  140.               </div>
    141. 

  141.               <p>Import the helpers we need.</p>
    142. 

  142. 

  143.             </div>
    144. 

  144.             
    145. 

  145.             <div class="content"><div class='highlight'><pre>{count, starts, compact, repeat, invertLiterate,
    146. 

  146. locationDataToString,  throwSyntaxError} = <span class="hljs-built_in">require</span> <span class="hljs-string">'./helpers'</span></pre></div></div>
    147. 

  147.             
    148. 

  148.         </li>
    149. 

  149.         
    150. 

  150.         
    151. 

  151.         <li id="section-3">
    152. 

  152.             <div class="annotation">
    153. 

  153.               
    154. 

  154.               <div class="pilwrap ">
    155. 

  155.                 <a class="pilcrow" href="#section-3">&#182;</a>
    156. 

  156.               </div>
    157. 

  157.               <h2 id="the-lexer-class">The Lexer Class</h2>
    158. 

  158. 

  159.             </div>
    160. 

  160.             
    161. 

  161.         </li>
    162. 

  162.         
    163. 

  163.         
    164. 

  164.         <li id="section-4">
    165. 

  165.             <div class="annotation">
    166. 

  166.               
    167. 

  167.               <div class="pilwrap ">
    168. 

  168.                 <a class="pilcrow" href="#section-4">&#182;</a>
    169. 

  169.               </div>
    170. 

  170.               
    171. 

  171.             </div>
    172. 

  172.             
    173. 

  173.         </li>
    174. 

  174.         
    175. 

  175.         
    176. 

  176.         <li id="section-5">
    177. 

  177.             <div class="annotation">
    178. 

  178.               
    179. 

  179.               <div class="pilwrap ">
    180. 

  180.                 <a class="pilcrow" href="#section-5">&#182;</a>
    181. 

  181.               </div>
    182. 

  182.               <p>The Lexer class reads a stream of CoffeeScript and divvies it up into tagged
    183. 

  183. tokens. Some potential ambiguity in the grammar has been avoided by
    184. 

  184. pushing some extra smarts into the Lexer.</p>
    185. 

  185. 

  186.             </div>
    187. 

  187.             
    188. 

  188.             <div class="content"><div class='highlight'><pre><span class="hljs-built_in">exports</span>.Lexer = <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Lexer</span></span></pre></div></div>
    189. 

  189.             
    190. 

  190.         </li>
    191. 

  191.         
    192. 

  192.         
    193. 

  193.         <li id="section-6">
    194. 

  194.             <div class="annotation">
    195. 

  195.               
    196. 

  196.               <div class="pilwrap ">
    197. 

  197.                 <a class="pilcrow" href="#section-6">&#182;</a>
    198. 

  198.               </div>
    199. 

  199.               <p><strong>tokenize</strong> is the Lexer’s main method. Scan by attempting to match tokens
    200. 

  200. one at a time, using a regular expression anchored at the start of the
    201. 

  201. remaining code, or a custom recursive token-matching method
    202. 

  202. (for interpolations). When the next token has been recorded, we move forward
    203. 

  203. within the code past the token, and begin again.</p>
    204. 

  204. <p>Each tokenizing method is responsible for returning the number of characters
    205. 

  205. it has consumed.</p>
    206. 

  206. <p>Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a>.</p>
    207. 

  207. 

  208.             </div>
    209. 

  209.             
    210. 

  210.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">tokenize</span>: <span class="hljs-function"><span class="hljs-params">(code, opts = {})</span> -&gt;</span>
    211. 

  211.     <span class="hljs-property">@literate</span>   = opts.literate  <span class="hljs-comment"># Are we lexing literate CoffeeScript?</span>
    212. 

  212.     <span class="hljs-property">@indent</span>     = <span class="hljs-number">0</span>              <span class="hljs-comment"># The current indentation level.</span>
    213. 

  213.     <span class="hljs-property">@baseIndent</span> = <span class="hljs-number">0</span>              <span class="hljs-comment"># The overall minimum indentation level</span>
    214. 

  214.     <span class="hljs-property">@indebt</span>     = <span class="hljs-number">0</span>              <span class="hljs-comment"># The over-indentation at the current level.</span>
    215. 

  215.     <span class="hljs-property">@outdebt</span>    = <span class="hljs-number">0</span>              <span class="hljs-comment"># The under-outdentation at the current level.</span>
    216. 

  216.     <span class="hljs-property">@indents</span>    = []             <span class="hljs-comment"># The stack of all current indentation levels.</span>
    217. 

  217.     <span class="hljs-property">@ends</span>       = []             <span class="hljs-comment"># The stack for pairing up tokens.</span>
    218. 

  218.     <span class="hljs-property">@tokens</span>     = []             <span class="hljs-comment"># Stream of parsed tokens in the form `['TYPE', value, location data]`.</span>
    219. 

  219.     <span class="hljs-property">@seenFor</span>    = <span class="hljs-literal">no</span>             <span class="hljs-comment"># Used to recognize FORIN and FOROF tokens.</span>
    220. 

  220. 

  221.     <span class="hljs-property">@chunkLine</span> =
    222. 

  222.       opts.line <span class="hljs-keyword">or</span> <span class="hljs-number">0</span>         <span class="hljs-comment"># The start line for the current @chunk.</span>
    223. 

  223.     <span class="hljs-property">@chunkColumn</span> =
    224. 

  224.       opts.column <span class="hljs-keyword">or</span> <span class="hljs-number">0</span>       <span class="hljs-comment"># The start column of the current @chunk.</span>
    225. 

  225.     code = <span class="hljs-property">@clean</span> code         <span class="hljs-comment"># The stripped, cleaned original source code.</span></pre></div></div>
    226. 

  226.             
    227. 

  227.         </li>
    228. 

  228.         
    229. 

  229.         
    230. 

  230.         <li id="section-7">
    231. 

  231.             <div class="annotation">
    232. 

  232.               
    233. 

  233.               <div class="pilwrap ">
    234. 

  234.                 <a class="pilcrow" href="#section-7">&#182;</a>
    235. 

  235.               </div>
    236. 

  236.               <p>At every position, run through this list of attempted matches,
    237. 

  237. short-circuiting if any of them succeed. Their order determines precedence:
    238. 

  238. <code>@literalToken</code> is the fallback catch-all.</p>
    239. 

  239. 

  240.             </div>
    241. 

  241.             
    242. 

  242.             <div class="content"><div class='highlight'><pre>    i = <span class="hljs-number">0</span>
    243. 

  243.     <span class="hljs-keyword">while</span> <span class="hljs-property">@chunk</span> = code[i..]
    244. 

  244.       consumed = \
    245. 

  245.            <span class="hljs-property">@identifierToken</span>() <span class="hljs-keyword">or</span>
    246. 

  246.            <span class="hljs-property">@commentToken</span>()    <span class="hljs-keyword">or</span>
    247. 

  247.            <span class="hljs-property">@whitespaceToken</span>() <span class="hljs-keyword">or</span>
    248. 

  248.            <span class="hljs-property">@lineToken</span>()       <span class="hljs-keyword">or</span>
    249. 

  249.            <span class="hljs-property">@stringToken</span>()     <span class="hljs-keyword">or</span>
    250. 

  250.            <span class="hljs-property">@numberToken</span>()     <span class="hljs-keyword">or</span>
    251. 

  251.            <span class="hljs-property">@regexToken</span>()      <span class="hljs-keyword">or</span>
    252. 

  252.            <span class="hljs-property">@jsToken</span>()         <span class="hljs-keyword">or</span>
    253. 

  253.            <span class="hljs-property">@literalToken</span>()</pre></div></div>
    254. 

  254.             
    255. 

  255.         </li>
    256. 

  256.         
    257. 

  257.         
    258. 

  258.         <li id="section-8">
    259. 

  259.             <div class="annotation">
    260. 

  260.               
    261. 

  261.               <div class="pilwrap ">
    262. 

  262.                 <a class="pilcrow" href="#section-8">&#182;</a>
    263. 

  263.               </div>
    264. 

  264.               <p>Update position</p>
    265. 

  265. 

  266.             </div>
    267. 

  267.             
    268. 

  268.             <div class="content"><div class='highlight'><pre>      [<span class="hljs-property">@chunkLine</span>, <span class="hljs-property">@chunkColumn</span>] = <span class="hljs-property">@getLineAndColumnFromChunk</span> consumed
    269. 

  269. 

  270.       i += consumed
    271. 

  271. 

  272.       <span class="hljs-keyword">return</span> {<span class="hljs-property">@tokens</span>, <span class="hljs-attribute">index</span>: i} <span class="hljs-keyword">if</span> opts.untilBalanced <span class="hljs-keyword">and</span> <span class="hljs-property">@ends</span>.length <span class="hljs-keyword">is</span> <span class="hljs-number">0</span>
    273. 

  273. 

  274.     <span class="hljs-property">@closeIndentation</span>()
    275. 

  275.     <span class="hljs-property">@error</span> <span class="hljs-string">"missing <span class="hljs-subst">#{end.tag}</span>"</span>, end.origin[<span class="hljs-number">2</span>] <span class="hljs-keyword">if</span> end = <span class="hljs-property">@ends</span>.pop()
    276. 

  276.     <span class="hljs-keyword">return</span> <span class="hljs-property">@tokens</span> <span class="hljs-keyword">if</span> opts.rewrite <span class="hljs-keyword">is</span> <span class="hljs-literal">off</span>
    277. 

  277.     (<span class="hljs-keyword">new</span> Rewriter).rewrite <span class="hljs-property">@tokens</span></pre></div></div>
    278. 

  278.             
    279. 

  279.         </li>
    280. 

  280.         
    281. 

  281.         
    282. 

  282.         <li id="section-9">
    283. 

  283.             <div class="annotation">
    284. 

  284.               
    285. 

  285.               <div class="pilwrap ">
    286. 

  286.                 <a class="pilcrow" href="#section-9">&#182;</a>
    287. 

  287.               </div>
    288. 

  288.               <p>Preprocess the code to remove leading and trailing whitespace, carriage
    289. 

  289. returns, etc. If we’re lexing literate CoffeeScript, strip external Markdown
    290. 

  290. by removing all lines that aren’t indented by at least four spaces or a tab.</p>
    291. 

  291. 

  292.             </div>
    293. 

  293.             
    294. 

  294.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">clean</span>: <span class="hljs-function"><span class="hljs-params">(code)</span> -&gt;</span>
    295. 

  295.     code = code.slice(<span class="hljs-number">1</span>) <span class="hljs-keyword">if</span> code.charCodeAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> BOM
    296. 

  296.     code = code.replace(<span class="hljs-regexp">/\r/g</span>, <span class="hljs-string">''</span>).replace TRAILING_SPACES, <span class="hljs-string">''</span>
    297. 

  297.     <span class="hljs-keyword">if</span> WHITESPACE.test code
    298. 

  298.       code = <span class="hljs-string">"\n<span class="hljs-subst">#{code}</span>"</span>
    299. 

  299.       <span class="hljs-property">@chunkLine</span>--
    300. 

  300.     code = invertLiterate code <span class="hljs-keyword">if</span> <span class="hljs-property">@literate</span>
    301. 

  301.     code</pre></div></div>
    302. 

  302.             
    303. 

  303.         </li>
    304. 

  304.         
    305. 

  305.         
    306. 

  306.         <li id="section-10">
    307. 

  307.             <div class="annotation">
    308. 

  308.               
    309. 

  309.               <div class="pilwrap ">
    310. 

  310.                 <a class="pilcrow" href="#section-10">&#182;</a>
    311. 

  311.               </div>
    312. 

  312.               <h2 id="tokenizers">Tokenizers</h2>
    313. 

  313. 

  314.             </div>
    315. 

  315.             
    316. 

  316.         </li>
    317. 

  317.         
    318. 

  318.         
    319. 

  319.         <li id="section-11">
    320. 

  320.             <div class="annotation">
    321. 

  321.               
    322. 

  322.               <div class="pilwrap ">
    323. 

  323.                 <a class="pilcrow" href="#section-11">&#182;</a>
    324. 

  324.               </div>
    325. 

  325.               
    326. 

  326.             </div>
    327. 

  327.             
    328. 

  328.         </li>
    329. 

  329.         
    330. 

  330.         
    331. 

  331.         <li id="section-12">
    332. 

  332.             <div class="annotation">
    333. 

  333.               
    334. 

  334.               <div class="pilwrap ">
    335. 

  335.                 <a class="pilcrow" href="#section-12">&#182;</a>
    336. 

  336.               </div>
    337. 

  337.               <p>Matches identifying literals: variables, keywords, method names, etc.
    338. 

  338. Check to ensure that JavaScript reserved words aren’t being used as
    339. 

  339. identifiers. Because CoffeeScript reserves a handful of keywords that are
    340. 

  340. allowed in JavaScript, we’re careful not to tag them as keywords when
    341. 

  341. referenced as property names here, so you can still do <code>jQuery.is()</code> even
    342. 

  342. though <code>is</code> means <code>===</code> otherwise.</p>
    343. 

  343. 

  344.             </div>
    345. 

  345.             
    346. 

  346.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">identifierToken</span>:<span class="hljs-function"> -&gt;</span>
    347. 

  347.     <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = IDENTIFIER.exec <span class="hljs-property">@chunk</span>
    348. 

  348.     [input, id, colon] = match</pre></div></div>
    349. 

  349.             
    350. 

  350.         </li>
    351. 

  351.         
    352. 

  352.         
    353. 

  353.         <li id="section-13">
    354. 

  354.             <div class="annotation">
    355. 

  355.               
    356. 

  356.               <div class="pilwrap ">
    357. 

  357.                 <a class="pilcrow" href="#section-13">&#182;</a>
    358. 

  358.               </div>
    359. 

  359.               <p>Preserve length of id for location data</p>
    360. 

  360. 

  361.             </div>
    362. 

  362.             
    363. 

  363.             <div class="content"><div class='highlight'><pre>    idLength = id.length
    364. 

  364.     poppedToken = <span class="hljs-literal">undefined</span>
    365. 

  365. 

  366.     <span class="hljs-keyword">if</span> id <span class="hljs-keyword">is</span> <span class="hljs-string">'own'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'FOR'</span>
    367. 

  367.       <span class="hljs-property">@token</span> <span class="hljs-string">'OWN'</span>, id
    368. 

  368.       <span class="hljs-keyword">return</span> id.length
    369. 

  369.     <span class="hljs-keyword">if</span> id <span class="hljs-keyword">is</span> <span class="hljs-string">'from'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'YIELD'</span>
    370. 

  370.       <span class="hljs-property">@token</span> <span class="hljs-string">'FROM'</span>, id
    371. 

  371.       <span class="hljs-keyword">return</span> id.length
    372. 

  372.     [..., prev] = <span class="hljs-property">@tokens</span>
    373. 

  373.     forcedIdentifier = colon <span class="hljs-keyword">or</span> prev? <span class="hljs-keyword">and</span>
    374. 

  374.       (prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> [<span class="hljs-string">'.'</span>, <span class="hljs-string">'?.'</span>, <span class="hljs-string">'::'</span>, <span class="hljs-string">'?::'</span>] <span class="hljs-keyword">or</span>
    375. 

  375.       <span class="hljs-keyword">not</span> prev.spaced <span class="hljs-keyword">and</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">is</span> <span class="hljs-string">'@'</span>)
    376. 

  376.     tag = <span class="hljs-string">'IDENTIFIER'</span>
    377. 

  377. 

  378.     <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> forcedIdentifier <span class="hljs-keyword">and</span> (id <span class="hljs-keyword">in</span> JS_KEYWORDS <span class="hljs-keyword">or</span> id <span class="hljs-keyword">in</span> COFFEE_KEYWORDS)
    379. 

  379.       tag = id.toUpperCase()
    380. 

  380.       <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'WHEN'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">in</span> LINE_BREAK
    381. 

  381.         tag = <span class="hljs-string">'LEADING_WHEN'</span>
    382. 

  382.       <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'FOR'</span>
    383. 

  383.         <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">yes</span>
    384. 

  384.       <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'UNLESS'</span>
    385. 

  385.         tag = <span class="hljs-string">'IF'</span>
    386. 

  386.       <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">in</span> UNARY
    387. 

  387.         tag = <span class="hljs-string">'UNARY'</span>
    388. 

  388.       <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">in</span> RELATION
    389. 

  389.         <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">isnt</span> <span class="hljs-string">'INSTANCEOF'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@seenFor</span>
    390. 

  390.           tag = <span class="hljs-string">'FOR'</span> + tag
    391. 

  391.           <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">no</span>
    392. 

  392.         <span class="hljs-keyword">else</span>
    393. 

  393.           tag = <span class="hljs-string">'RELATION'</span>
    394. 

  394.           <span class="hljs-keyword">if</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'!'</span>
    395. 

  395.             poppedToken = <span class="hljs-property">@tokens</span>.pop()
    396. 

  396.             id = <span class="hljs-string">'!'</span> + id
    397. 

  397. 

  398.     <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> JS_FORBIDDEN
    399. 

  399.       <span class="hljs-keyword">if</span> forcedIdentifier
    400. 

  400.         tag = <span class="hljs-string">'IDENTIFIER'</span>
    401. 

  401.         id  = <span class="hljs-keyword">new</span> String id
    402. 

  402.         id.reserved = <span class="hljs-literal">yes</span>
    403. 

  403.       <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> RESERVED
    404. 

  404.         <span class="hljs-property">@error</span> <span class="hljs-string">"reserved word '<span class="hljs-subst">#{id}</span>'"</span>, <span class="hljs-attribute">length</span>: id.length
    405. 

  405. 

  406.     <span class="hljs-keyword">unless</span> forcedIdentifier
    407. 

  407.       <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> COFFEE_ALIASES
    408. 

  408.         alias = id
    409. 

  409.         id = COFFEE_ALIAS_MAP[id]
    410. 

  410.       tag = <span class="hljs-keyword">switch</span> id
    411. 

  411.         <span class="hljs-keyword">when</span> <span class="hljs-string">'!'</span>                 <span class="hljs-keyword">then</span> <span class="hljs-string">'UNARY'</span>
    412. 

  412.         <span class="hljs-keyword">when</span> <span class="hljs-string">'=='</span>, <span class="hljs-string">'!='</span>          <span class="hljs-keyword">then</span> <span class="hljs-string">'COMPARE'</span>
    413. 

  413.         <span class="hljs-keyword">when</span> <span class="hljs-string">'&amp;&amp;'</span>, <span class="hljs-string">'||'</span>          <span class="hljs-keyword">then</span> <span class="hljs-string">'LOGIC'</span>
    414. 

  414.         <span class="hljs-keyword">when</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">'false'</span>     <span class="hljs-keyword">then</span> <span class="hljs-string">'BOOL'</span>
    415. 

  415.         <span class="hljs-keyword">when</span> <span class="hljs-string">'break'</span>, <span class="hljs-string">'continue'</span> <span class="hljs-keyword">then</span> <span class="hljs-string">'STATEMENT'</span>
    416. 

  416.         <span class="hljs-keyword">else</span>  tag
    417. 

  417. 

  418.     tagToken = <span class="hljs-property">@token</span> tag, id, <span class="hljs-number">0</span>, idLength
    419. 

  419.     tagToken.origin = [tag, alias, tagToken[<span class="hljs-number">2</span>]] <span class="hljs-keyword">if</span> alias
    420. 

  420.     tagToken.variable = <span class="hljs-keyword">not</span> forcedIdentifier
    421. 

  421.     <span class="hljs-keyword">if</span> poppedToken
    422. 

  422.       [tagToken[<span class="hljs-number">2</span>].first_line, tagToken[<span class="hljs-number">2</span>].first_column] =
    423. 

  423.         [poppedToken[<span class="hljs-number">2</span>].first_line, poppedToken[<span class="hljs-number">2</span>].first_column]
    424. 

  424.     <span class="hljs-keyword">if</span> colon
    425. 

  425.       colonOffset = input.lastIndexOf <span class="hljs-string">':'</span>
    426. 

  426.       <span class="hljs-property">@token</span> <span class="hljs-string">':'</span>, <span class="hljs-string">':'</span>, colonOffset, colon.length
    427. 

  427. 

  428.     input.length</pre></div></div>
    429. 

  429.             
    430. 

  430.         </li>
    431. 

  431.         
    432. 

  432.         
    433. 

  433.         <li id="section-14">
    434. 

  434.             <div class="annotation">
    435. 

  435.               
    436. 

  436.               <div class="pilwrap ">
    437. 

  437.                 <a class="pilcrow" href="#section-14">&#182;</a>
    438. 

  438.               </div>
    439. 

  439.               <p>Matches numbers, including decimals, hex, and exponential notation.
    440. 

  440. Be careful not to interfere with ranges-in-progress.</p>
    441. 

  441. 

  442.             </div>
    443. 

  443.             
    444. 

  444.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">numberToken</span>:<span class="hljs-function"> -&gt;</span>
    445. 

  445.     <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = NUMBER.exec <span class="hljs-property">@chunk</span>
    446. 

  446.     number = match[<span class="hljs-number">0</span>]
    447. 

  447.     lexedLength = number.length
    448. 

  448.     <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0[BOX]/</span>.test number
    449. 

  449.       <span class="hljs-property">@error</span> <span class="hljs-string">"radix prefix in '<span class="hljs-subst">#{number}</span>' must be lowercase"</span>, <span class="hljs-attribute">offset</span>: <span class="hljs-number">1</span>
    450. 

  450.     <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/E/</span>.test(number) <span class="hljs-keyword">and</span> <span class="hljs-keyword">not</span> <span class="hljs-regexp">/^0x/</span>.test number
    451. 

  451.       <span class="hljs-property">@error</span> <span class="hljs-string">"exponential notation in '<span class="hljs-subst">#{number}</span>' must be indicated with a lowercase 'e'"</span>,
    452. 

  452.         <span class="hljs-attribute">offset</span>: number.indexOf(<span class="hljs-string">'E'</span>)
    453. 

  453.     <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0\d*[89]/</span>.test number
    454. 

  454.       <span class="hljs-property">@error</span> <span class="hljs-string">"decimal literal '<span class="hljs-subst">#{number}</span>' must not be prefixed with '0'"</span>, <span class="hljs-attribute">length</span>: lexedLength
    455. 

  455.     <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0\d+/</span>.test number
    456. 

  456.       <span class="hljs-property">@error</span> <span class="hljs-string">"octal literal '<span class="hljs-subst">#{number}</span>' must be prefixed with '0o'"</span>, <span class="hljs-attribute">length</span>: lexedLength
    457. 

  457.     <span class="hljs-keyword">if</span> octalLiteral = <span class="hljs-regexp">/^0o([0-7]+)/</span>.exec number
    458. 

  458.       number = <span class="hljs-string">'0x'</span> + parseInt(octalLiteral[<span class="hljs-number">1</span>], <span class="hljs-number">8</span>).toString <span class="hljs-number">16</span>
    459. 

  459.     <span class="hljs-keyword">if</span> binaryLiteral = <span class="hljs-regexp">/^0b([01]+)/</span>.exec number
    460. 

  460.       number = <span class="hljs-string">'0x'</span> + parseInt(binaryLiteral[<span class="hljs-number">1</span>], <span class="hljs-number">2</span>).toString <span class="hljs-number">16</span>
    461. 

  461.     <span class="hljs-property">@token</span> <span class="hljs-string">'NUMBER'</span>, number, <span class="hljs-number">0</span>, lexedLength
    462. 

  462.     lexedLength</pre></div></div>
    463. 

  463.             
    464. 

  464.         </li>
    465. 

  465.         
    466. 

  466.         
    467. 

  467.         <li id="section-15">
    468. 

  468.             <div class="annotation">
    469. 

  469.               
    470. 

  470.               <div class="pilwrap ">
    471. 

  471.                 <a class="pilcrow" href="#section-15">&#182;</a>
    472. 

  472.               </div>
    473. 

  473.               <p>Matches strings, including multi-line strings, as well as heredocs, with or without
    474. 

  474. interpolation.</p>
    475. 

  475. 

  476.             </div>
    477. 

  477.             
    478. 

  478.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">stringToken</span>:<span class="hljs-function"> -&gt;</span>
    479. 

  479.     [quote] = STRING_START.exec(<span class="hljs-property">@chunk</span>) || []
    480. 

  480.     <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> quote
    481. 

  481.     regex = <span class="hljs-keyword">switch</span> quote
    482. 

  482.       <span class="hljs-keyword">when</span> <span class="hljs-string">"'"</span>   <span class="hljs-keyword">then</span> STRING_SINGLE
    483. 

  483.       <span class="hljs-keyword">when</span> <span class="hljs-string">'"'</span>   <span class="hljs-keyword">then</span> STRING_DOUBLE
    484. 

  484.       <span class="hljs-keyword">when</span> <span class="hljs-string">"'''"</span> <span class="hljs-keyword">then</span> HEREDOC_SINGLE
    485. 

  485.       <span class="hljs-keyword">when</span> <span class="hljs-string">'"""'</span> <span class="hljs-keyword">then</span> HEREDOC_DOUBLE
    486. 

  486.     heredoc = quote.length <span class="hljs-keyword">is</span> <span class="hljs-number">3</span>
    487. 

  487. 

  488.     {tokens, <span class="hljs-attribute">index</span>: end} = <span class="hljs-property">@matchWithInterpolations</span> regex, quote
    489. 

  489.     $ = tokens.length - <span class="hljs-number">1</span>
    490. 

  490. 

  491.     delimiter = quote.charAt(<span class="hljs-number">0</span>)
    492. 

  492.     <span class="hljs-keyword">if</span> heredoc</pre></div></div>
    493. 

  493.             
    494. 

  494.         </li>
    495. 

  495.         
    496. 

  496.         
    497. 

  497.         <li id="section-16">
    498. 

  498.             <div class="annotation">
    499. 

  499.               
    500. 

  500.               <div class="pilwrap ">
    501. 

  501.                 <a class="pilcrow" href="#section-16">&#182;</a>
    502. 

  502.               </div>
    503. 

  503.               <p>Find the smallest indentation. It will be removed from all lines later.</p>
    504. 

  504. 

  505.             </div>
    506. 

  506.             
    507. 

  507.             <div class="content"><div class='highlight'><pre>      indent = <span class="hljs-literal">null</span>
    508. 

  508.       doc = (token[<span class="hljs-number">1</span>] <span class="hljs-keyword">for</span> token, i <span class="hljs-keyword">in</span> tokens <span class="hljs-keyword">when</span> token[<span class="hljs-number">0</span>] <span class="hljs-keyword">is</span> <span class="hljs-string">'NEOSTRING'</span>).join <span class="hljs-string">'#{}'</span>
    509. 

  509.       <span class="hljs-keyword">while</span> match = HEREDOC_INDENT.exec doc
    510. 

  510.         attempt = match[<span class="hljs-number">1</span>]
    511. 

  511.         indent = attempt <span class="hljs-keyword">if</span> indent <span class="hljs-keyword">is</span> <span class="hljs-literal">null</span> <span class="hljs-keyword">or</span> <span class="hljs-number">0</span> &lt; attempt.length &lt; indent.length
    512. 

  512.       indentRegex = <span class="hljs-regexp">/// ^<span class="hljs-subst">#{indent}</span> ///</span>gm <span class="hljs-keyword">if</span> indent
    513. 

  513.       <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {delimiter}, <span class="hljs-function"><span class="hljs-params">(value, i)</span> =&gt;</span>
    514. 

  514.         value = <span class="hljs-property">@formatString</span> value
    515. 

  515.         value = value.replace LEADING_BLANK_LINE,  <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> i <span class="hljs-keyword">is</span> <span class="hljs-number">0</span>
    516. 

  516.         value = value.replace TRAILING_BLANK_LINE, <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> i <span class="hljs-keyword">is</span> $
    517. 

  517.         value = value.replace indentRegex, <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> indentRegex
    518. 

  518.         value
    519. 

  519.     <span class="hljs-keyword">else</span>
    520. 

  520.       <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {delimiter}, <span class="hljs-function"><span class="hljs-params">(value, i)</span> =&gt;</span>
    521. 

  521.         value = <span class="hljs-property">@formatString</span> value
    522. 

  522.         value = value.replace SIMPLE_STRING_OMIT, <span class="hljs-function"><span class="hljs-params">(match, offset)</span> -&gt;</span>
    523. 

  523.           <span class="hljs-keyword">if</span> (i <span class="hljs-keyword">is</span> <span class="hljs-number">0</span> <span class="hljs-keyword">and</span> offset <span class="hljs-keyword">is</span> <span class="hljs-number">0</span>) <span class="hljs-keyword">or</span>
    524. 

  524.              (i <span class="hljs-keyword">is</span> $ <span class="hljs-keyword">and</span> offset + match.length <span class="hljs-keyword">is</span> value.length)
    525. 

  525.             <span class="hljs-string">''</span>
    526. 

  526.           <span class="hljs-keyword">else</span>
    527. 

  527.             <span class="hljs-string">' '</span>
    528. 

  528.         value
    529. 

  529. 

  530.     end</pre></div></div>
    531. 

  531.             
    532. 

  532.         </li>
    533. 

  533.         
    534. 

  534.         
    535. 

  535.         <li id="section-17">
    536. 

  536.             <div class="annotation">
    537. 

  537.               
    538. 

  538.               <div class="pilwrap ">
    539. 

  539.                 <a class="pilcrow" href="#section-17">&#182;</a>
    540. 

  540.               </div>
    541. 

  541.               <p>Matches and consumes comments.</p>
    542. 

  542. 

  543.             </div>
    544. 

  544.             
    545. 

  545.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">commentToken</span>:<span class="hljs-function"> -&gt;</span>
    546. 

  546.     <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = <span class="hljs-property">@chunk</span>.match COMMENT
    547. 

  547.     [comment, here] = match
    548. 

  548.     <span class="hljs-keyword">if</span> here
    549. 

  549.       <span class="hljs-keyword">if</span> match = HERECOMMENT_ILLEGAL.exec comment
    550. 

  550.         <span class="hljs-property">@error</span> <span class="hljs-string">"block comments cannot contain <span class="hljs-subst">#{match[<span class="hljs-number">0</span>]}</span>"</span>,
    551. 

  551.           <span class="hljs-attribute">offset</span>: match.index, <span class="hljs-attribute">length</span>: match[<span class="hljs-number">0</span>].length
    552. 

  552.       <span class="hljs-keyword">if</span> here.indexOf(<span class="hljs-string">'\n'</span>) &gt;= <span class="hljs-number">0</span>
    553. 

  553.         here = here.replace <span class="hljs-regexp">/// \n <span class="hljs-subst">#{repeat <span class="hljs-string">' '</span>, <span class="hljs-property">@indent</span>}</span> ///</span>g, <span class="hljs-string">'\n'</span>
    554. 

  554.       <span class="hljs-property">@token</span> <span class="hljs-string">'HERECOMMENT'</span>, here, <span class="hljs-number">0</span>, comment.length
    555. 

  555.     comment.length</pre></div></div>
    556. 

  556.             
    557. 

  557.         </li>
    558. 

  558.         
    559. 

  559.         
    560. 

  560.         <li id="section-18">
    561. 

  561.             <div class="annotation">
    562. 

  562.               
    563. 

  563.               <div class="pilwrap ">
    564. 

  564.                 <a class="pilcrow" href="#section-18">&#182;</a>
    565. 

  565.               </div>
    566. 

  566.               <p>Matches JavaScript interpolated directly into the source via backticks.</p>
    567. 

  567. 

  568.             </div>
    569. 

  569.             
    570. 

  570.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">jsToken</span>:<span class="hljs-function"> -&gt;</span>
    571. 

  571.     <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <span class="hljs-property">@chunk</span>.charAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> <span class="hljs-string">'`'</span> <span class="hljs-keyword">and</span> match = JSTOKEN.exec <span class="hljs-property">@chunk</span>
    572. 

  572.     <span class="hljs-property">@token</span> <span class="hljs-string">'JS'</span>, (script = match[<span class="hljs-number">0</span>])[<span class="hljs-number">1.</span>..-<span class="hljs-number">1</span>], <span class="hljs-number">0</span>, script.length
    573. 

  573.     script.length</pre></div></div>
    574. 

  574.             
    575. 

  575.         </li>
    576. 

  576.         
    577. 

  577.         
    578. 

  578.         <li id="section-19">
    579. 

  579.             <div class="annotation">
    580. 

  580.               
    581. 

  581.               <div class="pilwrap ">
    582. 

  582.                 <a class="pilcrow" href="#section-19">&#182;</a>
    583. 

  583.               </div>
    584. 

  584.               <p>Matches regular expression literals, as well as multiline extended ones.
    585. 

  585. Lexing regular expressions is difficult to distinguish from division, so we
    586. 

  586. borrow some basic heuristics from JavaScript and Ruby.</p>
    587. 

  587. 

  588.             </div>
    589. 

  589.             
    590. 

  590.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">regexToken</span>:<span class="hljs-function"> -&gt;</span>
    591. 

  591.     <span class="hljs-keyword">switch</span>
    592. 

  592.       <span class="hljs-keyword">when</span> match = REGEX_ILLEGAL.exec <span class="hljs-property">@chunk</span>
    593. 

  593.         <span class="hljs-property">@error</span> <span class="hljs-string">"regular expressions cannot begin with <span class="hljs-subst">#{match[<span class="hljs-number">2</span>]}</span>"</span>,
    594. 

  594.           <span class="hljs-attribute">offset</span>: match.index + match[<span class="hljs-number">1</span>].length
    595. 

  595.       <span class="hljs-keyword">when</span> match = <span class="hljs-property">@matchWithInterpolations</span> HEREGEX, <span class="hljs-string">'///'</span>
    596. 

  596.         {tokens, index} = match
    597. 

  597.       <span class="hljs-keyword">when</span> match = REGEX.exec <span class="hljs-property">@chunk</span>
    598. 

  598.         [regex, body, closed] = match
    599. 

  599.         <span class="hljs-property">@validateEscapes</span> body, <span class="hljs-attribute">isRegex</span>: <span class="hljs-literal">yes</span>, <span class="hljs-attribute">offsetInChunk</span>: <span class="hljs-number">1</span>
    600. 

  600.         index = regex.length
    601. 

  601.         [..., prev] = <span class="hljs-property">@tokens</span>
    602. 

  602.         <span class="hljs-keyword">if</span> prev
    603. 

  603.           <span class="hljs-keyword">if</span> prev.spaced <span class="hljs-keyword">and</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> CALLABLE
    604. 

  604.             <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> closed <span class="hljs-keyword">or</span> POSSIBLY_DIVISION.test regex
    605. 

  605.           <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> NOT_REGEX
    606. 

  606.             <span class="hljs-keyword">return</span> <span class="hljs-number">0</span>
    607. 

  607.         <span class="hljs-property">@error</span> <span class="hljs-string">'missing / (unclosed regex)'</span> <span class="hljs-keyword">unless</span> closed
    608. 

  608.       <span class="hljs-keyword">else</span>
    609. 

  609.         <span class="hljs-keyword">return</span> <span class="hljs-number">0</span>
    610. 

  610. 

  611.     [flags] = REGEX_FLAGS.exec <span class="hljs-property">@chunk</span>[index..]
    612. 

  612.     end = index + flags.length
    613. 

  613.     origin = <span class="hljs-property">@makeToken</span> <span class="hljs-string">'REGEX'</span>, <span class="hljs-literal">null</span>, <span class="hljs-number">0</span>, end
    614. 

  614.     <span class="hljs-keyword">switch</span>
    615. 

  615.       <span class="hljs-keyword">when</span> <span class="hljs-keyword">not</span> VALID_FLAGS.test flags
    616. 

  616.         <span class="hljs-property">@error</span> <span class="hljs-string">"invalid regular expression flags <span class="hljs-subst">#{flags}</span>"</span>, <span class="hljs-attribute">offset</span>: index, <span class="hljs-attribute">length</span>: flags.length
    617. 

  617.       <span class="hljs-keyword">when</span> regex <span class="hljs-keyword">or</span> tokens.length <span class="hljs-keyword">is</span> <span class="hljs-number">1</span>
    618. 

  618.         body ?= <span class="hljs-property">@formatHeregex</span> tokens[<span class="hljs-number">0</span>][<span class="hljs-number">1</span>]
    619. 

  619.         <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX'</span>, <span class="hljs-string">"<span class="hljs-subst">#{<span class="hljs-property">@makeDelimitedLiteral</span> body, delimiter: <span class="hljs-string">'/'</span>}</span><span class="hljs-subst">#{flags}</span>"</span>, <span class="hljs-number">0</span>, end, origin
    620. 

  620.       <span class="hljs-keyword">else</span>
    621. 

  621.         <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX_START'</span>, <span class="hljs-string">'('</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, origin
    622. 

  622.         <span class="hljs-property">@token</span> <span class="hljs-string">'IDENTIFIER'</span>, <span class="hljs-string">'RegExp'</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>
    623. 

  623.         <span class="hljs-property">@token</span> <span class="hljs-string">'CALL_START'</span>, <span class="hljs-string">'('</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>
    624. 

  624.         <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {<span class="hljs-attribute">delimiter</span>: <span class="hljs-string">'"'</span>, <span class="hljs-attribute">double</span>: <span class="hljs-literal">yes</span>}, <span class="hljs-property">@formatHeregex</span>
    625. 

  625.         <span class="hljs-keyword">if</span> flags
    626. 

  626.           <span class="hljs-property">@token</span> <span class="hljs-string">','</span>, <span class="hljs-string">','</span>, index, <span class="hljs-number">0</span>
    627. 

  627.           <span class="hljs-property">@token</span> <span class="hljs-string">'STRING'</span>, <span class="hljs-string">'"'</span> + flags + <span class="hljs-string">'"'</span>, index, flags.length
    628. 

  628.         <span class="hljs-property">@token</span> <span class="hljs-string">')'</span>, <span class="hljs-string">')'</span>, end, <span class="hljs-number">0</span>
    629. 

  629.         <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX_END'</span>, <span class="hljs-string">')'</span>, end, <span class="hljs-number">0</span>
    630. 

  630. 

  631.     end</pre></div></div>
    632. 

  632.             
    633. 

  633.         </li>
    634. 

  634.         
    635. 

  635.         
    636. 

  636.         <li id="section-20">
    637. 

  637.             <div class="annotation">
    638. 

  638.               
    639. 

  639.               <div class="pilwrap ">
    640. 

  640.                 <a class="pilcrow" href="#section-20">&#182;</a>
    641. 

  641.               </div>
    642. 

  642.               <p>Matches newlines, indents, and outdents, and determines which is which.
    643. 

  643. If we can detect that the current line is continued onto the next line,
    644. 

  644. then the newline is suppressed:</p>
    645. 

  645. <pre><code>elements
    646. 

  646.   .each( ... )
    647. 

  647.   .map( ... )
    648. 

  648. </code></pre><p>Keeps track of the level of indentation, because a single outdent token
    649. 

  649. can close multiple indents, so we need to know how far in we happen to be.</p>
    650. 

  650. 

  651.             </div>
    652. 

  652.             
    653. 

  653.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">lineToken</span>:<span class="hljs-function"> -&gt;</span>
    654. 

  654.     <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = MULTI_DENT.exec <span class="hljs-property">@chunk</span>
    655. 

  655.     indent = match[<span class="hljs-number">0</span>]
    656. 

  656.     <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">no</span>
    657. 

  657.     size = indent.length - <span class="hljs-number">1</span> - indent.lastIndexOf <span class="hljs-string">'\n'</span>
    658. 

  658.     noNewlines = <span class="hljs-property">@unfinished</span>()
    659. 

  659.     <span class="hljs-keyword">if</span> size - <span class="hljs-property">@indebt</span> <span class="hljs-keyword">is</span> <span class="hljs-property">@indent</span>
    660. 

  660.       <span class="hljs-keyword">if</span> noNewlines <span class="hljs-keyword">then</span> <span class="hljs-property">@suppressNewlines</span>() <span class="hljs-keyword">else</span> <span class="hljs-property">@newlineToken</span> <span class="hljs-number">0</span>
    661. 

  661.       <span class="hljs-keyword">return</span> indent.length
    662. 

  662. 

  663.     <span class="hljs-keyword">if</span> size &gt; <span class="hljs-property">@indent</span>
    664. 

  664.       <span class="hljs-keyword">if</span> noNewlines
    665. 

  665.         <span class="hljs-property">@indebt</span> = size - <span class="hljs-property">@indent</span>
    666. 

  666.         <span class="hljs-property">@suppressNewlines</span>()
    667. 

  667.         <span class="hljs-keyword">return</span> indent.length
    668. 

  668.       <span class="hljs-keyword">unless</span> <span class="hljs-property">@tokens</span>.length
    669. 

  669.         <span class="hljs-property">@baseIndent</span> = <span class="hljs-property">@indent</span> = size
    670. 

  670.         <span class="hljs-keyword">return</span> indent.length
    671. 

  671.       diff = size - <span class="hljs-property">@indent</span> + <span class="hljs-property">@outdebt</span>
    672. 

  672.       <span class="hljs-property">@token</span> <span class="hljs-string">'INDENT'</span>, diff, indent.length - size, size
    673. 

  673.       <span class="hljs-property">@indents</span>.push diff
    674. 

  674.       <span class="hljs-property">@ends</span>.push {<span class="hljs-attribute">tag</span>: <span class="hljs-string">'OUTDENT'</span>}
    675. 

  675.       <span class="hljs-property">@outdebt</span> = <span class="hljs-property">@indebt</span> = <span class="hljs-number">0</span>
    676. 

  676.       <span class="hljs-property">@indent</span> = size
    677. 

  677.     <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> size &lt; <span class="hljs-property">@baseIndent</span>
    678. 

  678.       <span class="hljs-property">@error</span> <span class="hljs-string">'missing indentation'</span>, <span class="hljs-attribute">offset</span>: indent.length
    679. 

  679.     <span class="hljs-keyword">else</span>
    680. 

  680.       <span class="hljs-property">@indebt</span> = <span class="hljs-number">0</span>
    681. 

  681.       <span class="hljs-property">@outdentToken</span> <span class="hljs-property">@indent</span> - size, noNewlines, indent.length
    682. 

  682.     indent.length</pre></div></div>
    683. 

  683.             
    684. 

  684.         </li>
    685. 

  685.         
    686. 

  686.         
    687. 

  687.         <li id="section-21">
    688. 

  688.             <div class="annotation">
    689. 

  689.               
    690. 

  690.               <div class="pilwrap ">
    691. 

  691.                 <a class="pilcrow" href="#section-21">&#182;</a>
    692. 

  692.               </div>
    693. 

  693.               <p>Record an outdent token or multiple tokens, if we happen to be moving back
    694. 

  694. inwards past several recorded indents. Sets new @indent value.</p>
    695. 

  695. 

  696.             </div>
    697. 

  697.             
    698. 

  698.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">outdentToken</span>: <span class="hljs-function"><span class="hljs-params">(moveOut, noNewlines, outdentLength)</span> -&gt;</span>
    699. 

  699.     decreasedIndent = <span class="hljs-property">@indent</span> - moveOut
    700. 

  700.     <span class="hljs-keyword">while</span> moveOut &gt; <span class="hljs-number">0</span>
    701. 

  701.       lastIndent = <span class="hljs-property">@indents</span>[<span class="hljs-property">@indents</span>.length - <span class="hljs-number">1</span>]
    702. 

  702.       <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> lastIndent
    703. 

  703.         moveOut = <span class="hljs-number">0</span>
    704. 

  704.       <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> lastIndent <span class="hljs-keyword">is</span> <span class="hljs-property">@outdebt</span>
    705. 

  705.         moveOut -= <span class="hljs-property">@outdebt</span>
    706. 

  706.         <span class="hljs-property">@outdebt</span> = <span class="hljs-number">0</span>
    707. 

  707.       <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> lastIndent &lt; <span class="hljs-property">@outdebt</span>
    708. 

  708.         <span class="hljs-property">@outdebt</span> -= lastIndent
    709. 

  709.         moveOut  -= lastIndent
    710. 

  710.       <span class="hljs-keyword">else</span>
    711. 

  711.         dent = <span class="hljs-property">@indents</span>.pop() + <span class="hljs-property">@outdebt</span>
    712. 

  712.         <span class="hljs-keyword">if</span> outdentLength <span class="hljs-keyword">and</span> <span class="hljs-property">@chunk</span>[outdentLength] <span class="hljs-keyword">in</span> INDENTABLE_CLOSERS
    713. 

  713.           decreasedIndent -= dent - moveOut
    714. 

  714.           moveOut = dent
    715. 

  715.         <span class="hljs-property">@outdebt</span> = <span class="hljs-number">0</span></pre></div></div>
    716. 

  716.             
    717. 

  717.         </li>
    718. 

  718.         
    719. 

  719.         
    720. 

  720.         <li id="section-22">
    721. 

  721.             <div class="annotation">
    722. 

  722.               
    723. 

  723.               <div class="pilwrap ">
    724. 

  724.                 <a class="pilcrow" href="#section-22">&#182;</a>
    725. 

  725.               </div>
    726. 

  726.               <p>pair might call outdentToken, so preserve decreasedIndent</p>
    727. 

  727. 

  728.             </div>
    729. 

  729.             
    730. 

  730.             <div class="content"><div class='highlight'><pre>        <span class="hljs-property">@pair</span> <span class="hljs-string">'OUTDENT'</span>
    731. 

  731.         <span class="hljs-property">@token</span> <span class="hljs-string">'OUTDENT'</span>, moveOut, <span class="hljs-number">0</span>, outdentLength
    732. 

  732.         moveOut -= dent
    733. 

  733.     <span class="hljs-property">@outdebt</span> -= moveOut <span class="hljs-keyword">if</span> dent
    734. 

  734.     <span class="hljs-property">@tokens</span>.pop() <span class="hljs-keyword">while</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">';'</span>
    735. 

  735. 

  736.     <span class="hljs-property">@token</span> <span class="hljs-string">'TERMINATOR'</span>, <span class="hljs-string">'\n'</span>, outdentLength, <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'TERMINATOR'</span> <span class="hljs-keyword">or</span> noNewlines
    737. 

  737.     <span class="hljs-property">@indent</span> = decreasedIndent
    738. 

  738.     <span class="hljs-keyword">this</span></pre></div></div>
    739. 

  739.             
    740. 

  740.         </li>
    741. 

  741.         
    742. 

  742.         
    743. 

  743.         <li id="section-23">
    744. 

  744.             <div class="annotation">
    745. 

  745.               
    746. 

  746.               <div class="pilwrap ">
    747. 

  747.                 <a class="pilcrow" href="#section-23">&#182;</a>
    748. 

  748.               </div>
    749. 

  749.               <p>Matches and consumes non-meaningful whitespace. Tag the previous token
    750. 

  750. as being “spaced”, because there are some cases where it makes a difference.</p>
    751. 

  751. 

  752.             </div>
    753. 

  753.             
    754. 

  754.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">whitespaceToken</span>:<span class="hljs-function"> -&gt;</span>
    755. 

  755.     <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> (match = WHITESPACE.exec <span class="hljs-property">@chunk</span>) <span class="hljs-keyword">or</span>
    756. 

  756.                     (nline = <span class="hljs-property">@chunk</span>.charAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> <span class="hljs-string">'\n'</span>)
    757. 

  757.     [..., prev] = <span class="hljs-property">@tokens</span>
    758. 

  758.     prev[<span class="hljs-keyword">if</span> match <span class="hljs-keyword">then</span> <span class="hljs-string">'spaced'</span> <span class="hljs-keyword">else</span> <span class="hljs-string">'newLine'</span>] = <span class="hljs-literal">true</span> <span class="hljs-keyword">if</span> prev
    759. 

  759.     <span class="hljs-keyword">if</span> match <span class="hljs-keyword">then</span> match[<span class="hljs-number">0</span>].length <span class="hljs-keyword">else</span> <span class="hljs-number">0</span></pre></div></div>
    760. 

  760.             
    761. 

  761.         </li>
    762. 

  762.         
    763. 

  763.         
    764. 

  764.         <li id="section-24">
    765. 

  765.             <div class="annotation">
    766. 

  766.               
    767. 

  767.               <div class="pilwrap ">
    768. 

  768.                 <a class="pilcrow" href="#section-24">&#182;</a>
    769. 

  769.               </div>
    770. 

  770.               <p>Generate a newline token. Consecutive newlines get merged together.</p>
    771. 

  771. 

  772.             </div>
    773. 

  773.             
    774. 

  774.             <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">newlineToken</span>: <span class="hljs-function"><span class="hljs-params">(offset)</span> -&gt;</span>
    775. 

  775.     <span class="hljs-property">@tokens</span>.pop() <span class="hljs-keyword">while</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">';'</span>
    776. 

  776.     <span class="hljs-property">@token</span> <span class="hljs-string">'TERMINATOR'</span>, <span class="hljs-string">'\n'</span>, offset, <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <s…
    777.