PageRenderTime 69ms CodeModel.GetById 20ms app.highlight 36ms RepoModel.GetById 2ms app.codeStats 0ms

/documentation/docs/lexer.html

http://github.com/jashkenas/coffee-script
HTML | 2024 lines | 1865 code | 159 blank | 0 comment | 0 complexity | 9c4487192ff7778f37ad5ad37144c273 MD5 | raw file

Large files files are truncated, but you can click here to view the full file

  1<!DOCTYPE html>
  2
  3<html>
  4<head>
  5  <title>lexer.coffee</title>
  6  <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  7  <meta name="viewport" content="width=device-width, target-densitydpi=160dpi, initial-scale=1.0; maximum-scale=1.0; user-scalable=0;">
  8  <link rel="stylesheet" media="all" href="docco.css" />
  9</head>
 10<body>
 11  <div id="container">
 12    <div id="background"></div>
 13    
 14      <ul id="jump_to">
 15        <li>
 16          <a class="large" href="javascript:void(0);">Jump To &hellip;</a>
 17          <a class="small" href="javascript:void(0);">+</a>
 18          <div id="jump_wrapper">
 19          <div id="jump_page_wrapper">
 20            <div id="jump_page">
 21              
 22                
 23                <a class="source" href="browser.html">
 24                  browser.coffee
 25                </a>
 26              
 27                
 28                <a class="source" href="cake.html">
 29                  cake.coffee
 30                </a>
 31              
 32                
 33                <a class="source" href="coffee-script.html">
 34                  coffee-script.coffee
 35                </a>
 36              
 37                
 38                <a class="source" href="command.html">
 39                  command.coffee
 40                </a>
 41              
 42                
 43                <a class="source" href="grammar.html">
 44                  grammar.coffee
 45                </a>
 46              
 47                
 48                <a class="source" href="helpers.html">
 49                  helpers.coffee
 50                </a>
 51              
 52                
 53                <a class="source" href="index.html">
 54                  index.coffee
 55                </a>
 56              
 57                
 58                <a class="source" href="lexer.html">
 59                  lexer.coffee
 60                </a>
 61              
 62                
 63                <a class="source" href="nodes.html">
 64                  nodes.coffee
 65                </a>
 66              
 67                
 68                <a class="source" href="optparse.html">
 69                  optparse.coffee
 70                </a>
 71              
 72                
 73                <a class="source" href="register.html">
 74                  register.coffee
 75                </a>
 76              
 77                
 78                <a class="source" href="repl.html">
 79                  repl.coffee
 80                </a>
 81              
 82                
 83                <a class="source" href="rewriter.html">
 84                  rewriter.coffee
 85                </a>
 86              
 87                
 88                <a class="source" href="scope.html">
 89                  scope.litcoffee
 90                </a>
 91              
 92                
 93                <a class="source" href="sourcemap.html">
 94                  sourcemap.litcoffee
 95                </a>
 96              
 97            </div>
 98          </div>
 99        </li>
100      </ul>
101    
102    <ul class="sections">
103        
104          <li id="title">
105              <div class="annotation">
106                  <h1>lexer.coffee</h1>
107              </div>
108          </li>
109        
110        
111        
112        <li id="section-1">
113            <div class="annotation">
114              
115              <div class="pilwrap ">
116                <a class="pilcrow" href="#section-1">&#182;</a>
117              </div>
118              <p>The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt
119matches against the beginning of the source code. When a match is found,
120a token is produced, we consume the match, and start again. Tokens are in the
121form:</p>
122<pre><code>[tag, value, locationData]
123</code></pre><p>where locationData is {first_line, first_column, last_line, last_column}, which is a
124format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>.  These
125are read by jison in the <code>parser.lexer</code> function defined in coffee-script.coffee.</p>
126
127            </div>
128            
129            <div class="content"><div class='highlight'><pre>
130{Rewriter, INVERSES} = <span class="hljs-built_in">require</span> <span class="hljs-string">'./rewriter'</span></pre></div></div>
131            
132        </li>
133        
134        
135        <li id="section-2">
136            <div class="annotation">
137              
138              <div class="pilwrap ">
139                <a class="pilcrow" href="#section-2">&#182;</a>
140              </div>
141              <p>Import the helpers we need.</p>
142
143            </div>
144            
145            <div class="content"><div class='highlight'><pre>{count, starts, compact, repeat, invertLiterate,
146locationDataToString,  throwSyntaxError} = <span class="hljs-built_in">require</span> <span class="hljs-string">'./helpers'</span></pre></div></div>
147            
148        </li>
149        
150        
151        <li id="section-3">
152            <div class="annotation">
153              
154              <div class="pilwrap ">
155                <a class="pilcrow" href="#section-3">&#182;</a>
156              </div>
157              <h2 id="the-lexer-class">The Lexer Class</h2>
158
159            </div>
160            
161        </li>
162        
163        
164        <li id="section-4">
165            <div class="annotation">
166              
167              <div class="pilwrap ">
168                <a class="pilcrow" href="#section-4">&#182;</a>
169              </div>
170              
171            </div>
172            
173        </li>
174        
175        
176        <li id="section-5">
177            <div class="annotation">
178              
179              <div class="pilwrap ">
180                <a class="pilcrow" href="#section-5">&#182;</a>
181              </div>
182              <p>The Lexer class reads a stream of CoffeeScript and divvies it up into tagged
183tokens. Some potential ambiguity in the grammar has been avoided by
184pushing some extra smarts into the Lexer.</p>
185
186            </div>
187            
188            <div class="content"><div class='highlight'><pre><span class="hljs-built_in">exports</span>.Lexer = <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Lexer</span></span></pre></div></div>
189            
190        </li>
191        
192        
193        <li id="section-6">
194            <div class="annotation">
195              
196              <div class="pilwrap ">
197                <a class="pilcrow" href="#section-6">&#182;</a>
198              </div>
199              <p><strong>tokenize</strong> is the Lexer’s main method. Scan by attempting to match tokens
200one at a time, using a regular expression anchored at the start of the
201remaining code, or a custom recursive token-matching method
202(for interpolations). When the next token has been recorded, we move forward
203within the code past the token, and begin again.</p>
204<p>Each tokenizing method is responsible for returning the number of characters
205it has consumed.</p>
206<p>Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a>.</p>
207
208            </div>
209            
210            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">tokenize</span>: <span class="hljs-function"><span class="hljs-params">(code, opts = {})</span> -&gt;</span>
211    <span class="hljs-property">@literate</span>   = opts.literate  <span class="hljs-comment"># Are we lexing literate CoffeeScript?</span>
212    <span class="hljs-property">@indent</span>     = <span class="hljs-number">0</span>              <span class="hljs-comment"># The current indentation level.</span>
213    <span class="hljs-property">@baseIndent</span> = <span class="hljs-number">0</span>              <span class="hljs-comment"># The overall minimum indentation level</span>
214    <span class="hljs-property">@indebt</span>     = <span class="hljs-number">0</span>              <span class="hljs-comment"># The over-indentation at the current level.</span>
215    <span class="hljs-property">@outdebt</span>    = <span class="hljs-number">0</span>              <span class="hljs-comment"># The under-outdentation at the current level.</span>
216    <span class="hljs-property">@indents</span>    = []             <span class="hljs-comment"># The stack of all current indentation levels.</span>
217    <span class="hljs-property">@ends</span>       = []             <span class="hljs-comment"># The stack for pairing up tokens.</span>
218    <span class="hljs-property">@tokens</span>     = []             <span class="hljs-comment"># Stream of parsed tokens in the form `['TYPE', value, location data]`.</span>
219    <span class="hljs-property">@seenFor</span>    = <span class="hljs-literal">no</span>             <span class="hljs-comment"># Used to recognize FORIN and FOROF tokens.</span>
220
221    <span class="hljs-property">@chunkLine</span> =
222      opts.line <span class="hljs-keyword">or</span> <span class="hljs-number">0</span>         <span class="hljs-comment"># The start line for the current @chunk.</span>
223    <span class="hljs-property">@chunkColumn</span> =
224      opts.column <span class="hljs-keyword">or</span> <span class="hljs-number">0</span>       <span class="hljs-comment"># The start column of the current @chunk.</span>
225    code = <span class="hljs-property">@clean</span> code         <span class="hljs-comment"># The stripped, cleaned original source code.</span></pre></div></div>
226            
227        </li>
228        
229        
230        <li id="section-7">
231            <div class="annotation">
232              
233              <div class="pilwrap ">
234                <a class="pilcrow" href="#section-7">&#182;</a>
235              </div>
236              <p>At every position, run through this list of attempted matches,
237short-circuiting if any of them succeed. Their order determines precedence:
238<code>@literalToken</code> is the fallback catch-all.</p>
239
240            </div>
241            
242            <div class="content"><div class='highlight'><pre>    i = <span class="hljs-number">0</span>
243    <span class="hljs-keyword">while</span> <span class="hljs-property">@chunk</span> = code[i..]
244      consumed = \
245           <span class="hljs-property">@identifierToken</span>() <span class="hljs-keyword">or</span>
246           <span class="hljs-property">@commentToken</span>()    <span class="hljs-keyword">or</span>
247           <span class="hljs-property">@whitespaceToken</span>() <span class="hljs-keyword">or</span>
248           <span class="hljs-property">@lineToken</span>()       <span class="hljs-keyword">or</span>
249           <span class="hljs-property">@stringToken</span>()     <span class="hljs-keyword">or</span>
250           <span class="hljs-property">@numberToken</span>()     <span class="hljs-keyword">or</span>
251           <span class="hljs-property">@regexToken</span>()      <span class="hljs-keyword">or</span>
252           <span class="hljs-property">@jsToken</span>()         <span class="hljs-keyword">or</span>
253           <span class="hljs-property">@literalToken</span>()</pre></div></div>
254            
255        </li>
256        
257        
258        <li id="section-8">
259            <div class="annotation">
260              
261              <div class="pilwrap ">
262                <a class="pilcrow" href="#section-8">&#182;</a>
263              </div>
264              <p>Update position</p>
265
266            </div>
267            
268            <div class="content"><div class='highlight'><pre>      [<span class="hljs-property">@chunkLine</span>, <span class="hljs-property">@chunkColumn</span>] = <span class="hljs-property">@getLineAndColumnFromChunk</span> consumed
269
270      i += consumed
271
272      <span class="hljs-keyword">return</span> {<span class="hljs-property">@tokens</span>, <span class="hljs-attribute">index</span>: i} <span class="hljs-keyword">if</span> opts.untilBalanced <span class="hljs-keyword">and</span> <span class="hljs-property">@ends</span>.length <span class="hljs-keyword">is</span> <span class="hljs-number">0</span>
273
274    <span class="hljs-property">@closeIndentation</span>()
275    <span class="hljs-property">@error</span> <span class="hljs-string">"missing <span class="hljs-subst">#{end.tag}</span>"</span>, end.origin[<span class="hljs-number">2</span>] <span class="hljs-keyword">if</span> end = <span class="hljs-property">@ends</span>.pop()
276    <span class="hljs-keyword">return</span> <span class="hljs-property">@tokens</span> <span class="hljs-keyword">if</span> opts.rewrite <span class="hljs-keyword">is</span> <span class="hljs-literal">off</span>
277    (<span class="hljs-keyword">new</span> Rewriter).rewrite <span class="hljs-property">@tokens</span></pre></div></div>
278            
279        </li>
280        
281        
282        <li id="section-9">
283            <div class="annotation">
284              
285              <div class="pilwrap ">
286                <a class="pilcrow" href="#section-9">&#182;</a>
287              </div>
288              <p>Preprocess the code to remove leading and trailing whitespace, carriage
289returns, etc. If we’re lexing literate CoffeeScript, strip external Markdown
290by removing all lines that aren’t indented by at least four spaces or a tab.</p>
291
292            </div>
293            
294            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">clean</span>: <span class="hljs-function"><span class="hljs-params">(code)</span> -&gt;</span>
295    code = code.slice(<span class="hljs-number">1</span>) <span class="hljs-keyword">if</span> code.charCodeAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> BOM
296    code = code.replace(<span class="hljs-regexp">/\r/g</span>, <span class="hljs-string">''</span>).replace TRAILING_SPACES, <span class="hljs-string">''</span>
297    <span class="hljs-keyword">if</span> WHITESPACE.test code
298      code = <span class="hljs-string">"\n<span class="hljs-subst">#{code}</span>"</span>
299      <span class="hljs-property">@chunkLine</span>--
300    code = invertLiterate code <span class="hljs-keyword">if</span> <span class="hljs-property">@literate</span>
301    code</pre></div></div>
302            
303        </li>
304        
305        
306        <li id="section-10">
307            <div class="annotation">
308              
309              <div class="pilwrap ">
310                <a class="pilcrow" href="#section-10">&#182;</a>
311              </div>
312              <h2 id="tokenizers">Tokenizers</h2>
313
314            </div>
315            
316        </li>
317        
318        
319        <li id="section-11">
320            <div class="annotation">
321              
322              <div class="pilwrap ">
323                <a class="pilcrow" href="#section-11">&#182;</a>
324              </div>
325              
326            </div>
327            
328        </li>
329        
330        
331        <li id="section-12">
332            <div class="annotation">
333              
334              <div class="pilwrap ">
335                <a class="pilcrow" href="#section-12">&#182;</a>
336              </div>
337              <p>Matches identifying literals: variables, keywords, method names, etc.
338Check to ensure that JavaScript reserved words aren’t being used as
339identifiers. Because CoffeeScript reserves a handful of keywords that are
340allowed in JavaScript, we’re careful not to tag them as keywords when
341referenced as property names here, so you can still do <code>jQuery.is()</code> even
342though <code>is</code> means <code>===</code> otherwise.</p>
343
344            </div>
345            
346            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">identifierToken</span>:<span class="hljs-function"> -&gt;</span>
347    <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = IDENTIFIER.exec <span class="hljs-property">@chunk</span>
348    [input, id, colon] = match</pre></div></div>
349            
350        </li>
351        
352        
353        <li id="section-13">
354            <div class="annotation">
355              
356              <div class="pilwrap ">
357                <a class="pilcrow" href="#section-13">&#182;</a>
358              </div>
359              <p>Preserve length of id for location data</p>
360
361            </div>
362            
363            <div class="content"><div class='highlight'><pre>    idLength = id.length
364    poppedToken = <span class="hljs-literal">undefined</span>
365
366    <span class="hljs-keyword">if</span> id <span class="hljs-keyword">is</span> <span class="hljs-string">'own'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'FOR'</span>
367      <span class="hljs-property">@token</span> <span class="hljs-string">'OWN'</span>, id
368      <span class="hljs-keyword">return</span> id.length
369    <span class="hljs-keyword">if</span> id <span class="hljs-keyword">is</span> <span class="hljs-string">'from'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'YIELD'</span>
370      <span class="hljs-property">@token</span> <span class="hljs-string">'FROM'</span>, id
371      <span class="hljs-keyword">return</span> id.length
372    [..., prev] = <span class="hljs-property">@tokens</span>
373    forcedIdentifier = colon <span class="hljs-keyword">or</span> prev? <span class="hljs-keyword">and</span>
374      (prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> [<span class="hljs-string">'.'</span>, <span class="hljs-string">'?.'</span>, <span class="hljs-string">'::'</span>, <span class="hljs-string">'?::'</span>] <span class="hljs-keyword">or</span>
375      <span class="hljs-keyword">not</span> prev.spaced <span class="hljs-keyword">and</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">is</span> <span class="hljs-string">'@'</span>)
376    tag = <span class="hljs-string">'IDENTIFIER'</span>
377
378    <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> forcedIdentifier <span class="hljs-keyword">and</span> (id <span class="hljs-keyword">in</span> JS_KEYWORDS <span class="hljs-keyword">or</span> id <span class="hljs-keyword">in</span> COFFEE_KEYWORDS)
379      tag = id.toUpperCase()
380      <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'WHEN'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">in</span> LINE_BREAK
381        tag = <span class="hljs-string">'LEADING_WHEN'</span>
382      <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'FOR'</span>
383        <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">yes</span>
384      <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'UNLESS'</span>
385        tag = <span class="hljs-string">'IF'</span>
386      <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">in</span> UNARY
387        tag = <span class="hljs-string">'UNARY'</span>
388      <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">in</span> RELATION
389        <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">isnt</span> <span class="hljs-string">'INSTANCEOF'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@seenFor</span>
390          tag = <span class="hljs-string">'FOR'</span> + tag
391          <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">no</span>
392        <span class="hljs-keyword">else</span>
393          tag = <span class="hljs-string">'RELATION'</span>
394          <span class="hljs-keyword">if</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'!'</span>
395            poppedToken = <span class="hljs-property">@tokens</span>.pop()
396            id = <span class="hljs-string">'!'</span> + id
397
398    <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> JS_FORBIDDEN
399      <span class="hljs-keyword">if</span> forcedIdentifier
400        tag = <span class="hljs-string">'IDENTIFIER'</span>
401        id  = <span class="hljs-keyword">new</span> String id
402        id.reserved = <span class="hljs-literal">yes</span>
403      <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> RESERVED
404        <span class="hljs-property">@error</span> <span class="hljs-string">"reserved word '<span class="hljs-subst">#{id}</span>'"</span>, <span class="hljs-attribute">length</span>: id.length
405
406    <span class="hljs-keyword">unless</span> forcedIdentifier
407      <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> COFFEE_ALIASES
408        alias = id
409        id = COFFEE_ALIAS_MAP[id]
410      tag = <span class="hljs-keyword">switch</span> id
411        <span class="hljs-keyword">when</span> <span class="hljs-string">'!'</span>                 <span class="hljs-keyword">then</span> <span class="hljs-string">'UNARY'</span>
412        <span class="hljs-keyword">when</span> <span class="hljs-string">'=='</span>, <span class="hljs-string">'!='</span>          <span class="hljs-keyword">then</span> <span class="hljs-string">'COMPARE'</span>
413        <span class="hljs-keyword">when</span> <span class="hljs-string">'&amp;&amp;'</span>, <span class="hljs-string">'||'</span>          <span class="hljs-keyword">then</span> <span class="hljs-string">'LOGIC'</span>
414        <span class="hljs-keyword">when</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">'false'</span>     <span class="hljs-keyword">then</span> <span class="hljs-string">'BOOL'</span>
415        <span class="hljs-keyword">when</span> <span class="hljs-string">'break'</span>, <span class="hljs-string">'continue'</span> <span class="hljs-keyword">then</span> <span class="hljs-string">'STATEMENT'</span>
416        <span class="hljs-keyword">else</span>  tag
417
418    tagToken = <span class="hljs-property">@token</span> tag, id, <span class="hljs-number">0</span>, idLength
419    tagToken.origin = [tag, alias, tagToken[<span class="hljs-number">2</span>]] <span class="hljs-keyword">if</span> alias
420    tagToken.variable = <span class="hljs-keyword">not</span> forcedIdentifier
421    <span class="hljs-keyword">if</span> poppedToken
422      [tagToken[<span class="hljs-number">2</span>].first_line, tagToken[<span class="hljs-number">2</span>].first_column] =
423        [poppedToken[<span class="hljs-number">2</span>].first_line, poppedToken[<span class="hljs-number">2</span>].first_column]
424    <span class="hljs-keyword">if</span> colon
425      colonOffset = input.lastIndexOf <span class="hljs-string">':'</span>
426      <span class="hljs-property">@token</span> <span class="hljs-string">':'</span>, <span class="hljs-string">':'</span>, colonOffset, colon.length
427
428    input.length</pre></div></div>
429            
430        </li>
431        
432        
433        <li id="section-14">
434            <div class="annotation">
435              
436              <div class="pilwrap ">
437                <a class="pilcrow" href="#section-14">&#182;</a>
438              </div>
439              <p>Matches numbers, including decimals, hex, and exponential notation.
440Be careful not to interfere with ranges-in-progress.</p>
441
442            </div>
443            
444            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">numberToken</span>:<span class="hljs-function"> -&gt;</span>
445    <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = NUMBER.exec <span class="hljs-property">@chunk</span>
446    number = match[<span class="hljs-number">0</span>]
447    lexedLength = number.length
448    <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0[BOX]/</span>.test number
449      <span class="hljs-property">@error</span> <span class="hljs-string">"radix prefix in '<span class="hljs-subst">#{number}</span>' must be lowercase"</span>, <span class="hljs-attribute">offset</span>: <span class="hljs-number">1</span>
450    <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/E/</span>.test(number) <span class="hljs-keyword">and</span> <span class="hljs-keyword">not</span> <span class="hljs-regexp">/^0x/</span>.test number
451      <span class="hljs-property">@error</span> <span class="hljs-string">"exponential notation in '<span class="hljs-subst">#{number}</span>' must be indicated with a lowercase 'e'"</span>,
452        <span class="hljs-attribute">offset</span>: number.indexOf(<span class="hljs-string">'E'</span>)
453    <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0\d*[89]/</span>.test number
454      <span class="hljs-property">@error</span> <span class="hljs-string">"decimal literal '<span class="hljs-subst">#{number}</span>' must not be prefixed with '0'"</span>, <span class="hljs-attribute">length</span>: lexedLength
455    <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0\d+/</span>.test number
456      <span class="hljs-property">@error</span> <span class="hljs-string">"octal literal '<span class="hljs-subst">#{number}</span>' must be prefixed with '0o'"</span>, <span class="hljs-attribute">length</span>: lexedLength
457    <span class="hljs-keyword">if</span> octalLiteral = <span class="hljs-regexp">/^0o([0-7]+)/</span>.exec number
458      number = <span class="hljs-string">'0x'</span> + parseInt(octalLiteral[<span class="hljs-number">1</span>], <span class="hljs-number">8</span>).toString <span class="hljs-number">16</span>
459    <span class="hljs-keyword">if</span> binaryLiteral = <span class="hljs-regexp">/^0b([01]+)/</span>.exec number
460      number = <span class="hljs-string">'0x'</span> + parseInt(binaryLiteral[<span class="hljs-number">1</span>], <span class="hljs-number">2</span>).toString <span class="hljs-number">16</span>
461    <span class="hljs-property">@token</span> <span class="hljs-string">'NUMBER'</span>, number, <span class="hljs-number">0</span>, lexedLength
462    lexedLength</pre></div></div>
463            
464        </li>
465        
466        
467        <li id="section-15">
468            <div class="annotation">
469              
470              <div class="pilwrap ">
471                <a class="pilcrow" href="#section-15">&#182;</a>
472              </div>
473              <p>Matches strings, including multi-line strings, as well as heredocs, with or without
474interpolation.</p>
475
476            </div>
477            
478            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">stringToken</span>:<span class="hljs-function"> -&gt;</span>
479    [quote] = STRING_START.exec(<span class="hljs-property">@chunk</span>) || []
480    <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> quote
481    regex = <span class="hljs-keyword">switch</span> quote
482      <span class="hljs-keyword">when</span> <span class="hljs-string">"'"</span>   <span class="hljs-keyword">then</span> STRING_SINGLE
483      <span class="hljs-keyword">when</span> <span class="hljs-string">'"'</span>   <span class="hljs-keyword">then</span> STRING_DOUBLE
484      <span class="hljs-keyword">when</span> <span class="hljs-string">"'''"</span> <span class="hljs-keyword">then</span> HEREDOC_SINGLE
485      <span class="hljs-keyword">when</span> <span class="hljs-string">'"""'</span> <span class="hljs-keyword">then</span> HEREDOC_DOUBLE
486    heredoc = quote.length <span class="hljs-keyword">is</span> <span class="hljs-number">3</span>
487
488    {tokens, <span class="hljs-attribute">index</span>: end} = <span class="hljs-property">@matchWithInterpolations</span> regex, quote
489    $ = tokens.length - <span class="hljs-number">1</span>
490
491    delimiter = quote.charAt(<span class="hljs-number">0</span>)
492    <span class="hljs-keyword">if</span> heredoc</pre></div></div>
493            
494        </li>
495        
496        
497        <li id="section-16">
498            <div class="annotation">
499              
500              <div class="pilwrap ">
501                <a class="pilcrow" href="#section-16">&#182;</a>
502              </div>
503              <p>Find the smallest indentation. It will be removed from all lines later.</p>
504
505            </div>
506            
507            <div class="content"><div class='highlight'><pre>      indent = <span class="hljs-literal">null</span>
508      doc = (token[<span class="hljs-number">1</span>] <span class="hljs-keyword">for</span> token, i <span class="hljs-keyword">in</span> tokens <span class="hljs-keyword">when</span> token[<span class="hljs-number">0</span>] <span class="hljs-keyword">is</span> <span class="hljs-string">'NEOSTRING'</span>).join <span class="hljs-string">'#{}'</span>
509      <span class="hljs-keyword">while</span> match = HEREDOC_INDENT.exec doc
510        attempt = match[<span class="hljs-number">1</span>]
511        indent = attempt <span class="hljs-keyword">if</span> indent <span class="hljs-keyword">is</span> <span class="hljs-literal">null</span> <span class="hljs-keyword">or</span> <span class="hljs-number">0</span> &lt; attempt.length &lt; indent.length
512      indentRegex = <span class="hljs-regexp">/// ^<span class="hljs-subst">#{indent}</span> ///</span>gm <span class="hljs-keyword">if</span> indent
513      <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {delimiter}, <span class="hljs-function"><span class="hljs-params">(value, i)</span> =&gt;</span>
514        value = <span class="hljs-property">@formatString</span> value
515        value = value.replace LEADING_BLANK_LINE,  <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> i <span class="hljs-keyword">is</span> <span class="hljs-number">0</span>
516        value = value.replace TRAILING_BLANK_LINE, <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> i <span class="hljs-keyword">is</span> $
517        value = value.replace indentRegex, <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> indentRegex
518        value
519    <span class="hljs-keyword">else</span>
520      <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {delimiter}, <span class="hljs-function"><span class="hljs-params">(value, i)</span> =&gt;</span>
521        value = <span class="hljs-property">@formatString</span> value
522        value = value.replace SIMPLE_STRING_OMIT, <span class="hljs-function"><span class="hljs-params">(match, offset)</span> -&gt;</span>
523          <span class="hljs-keyword">if</span> (i <span class="hljs-keyword">is</span> <span class="hljs-number">0</span> <span class="hljs-keyword">and</span> offset <span class="hljs-keyword">is</span> <span class="hljs-number">0</span>) <span class="hljs-keyword">or</span>
524             (i <span class="hljs-keyword">is</span> $ <span class="hljs-keyword">and</span> offset + match.length <span class="hljs-keyword">is</span> value.length)
525            <span class="hljs-string">''</span>
526          <span class="hljs-keyword">else</span>
527            <span class="hljs-string">' '</span>
528        value
529
530    end</pre></div></div>
531            
532        </li>
533        
534        
535        <li id="section-17">
536            <div class="annotation">
537              
538              <div class="pilwrap ">
539                <a class="pilcrow" href="#section-17">&#182;</a>
540              </div>
541              <p>Matches and consumes comments.</p>
542
543            </div>
544            
545            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">commentToken</span>:<span class="hljs-function"> -&gt;</span>
546    <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = <span class="hljs-property">@chunk</span>.match COMMENT
547    [comment, here] = match
548    <span class="hljs-keyword">if</span> here
549      <span class="hljs-keyword">if</span> match = HERECOMMENT_ILLEGAL.exec comment
550        <span class="hljs-property">@error</span> <span class="hljs-string">"block comments cannot contain <span class="hljs-subst">#{match[<span class="hljs-number">0</span>]}</span>"</span>,
551          <span class="hljs-attribute">offset</span>: match.index, <span class="hljs-attribute">length</span>: match[<span class="hljs-number">0</span>].length
552      <span class="hljs-keyword">if</span> here.indexOf(<span class="hljs-string">'\n'</span>) &gt;= <span class="hljs-number">0</span>
553        here = here.replace <span class="hljs-regexp">/// \n <span class="hljs-subst">#{repeat <span class="hljs-string">' '</span>, <span class="hljs-property">@indent</span>}</span> ///</span>g, <span class="hljs-string">'\n'</span>
554      <span class="hljs-property">@token</span> <span class="hljs-string">'HERECOMMENT'</span>, here, <span class="hljs-number">0</span>, comment.length
555    comment.length</pre></div></div>
556            
557        </li>
558        
559        
560        <li id="section-18">
561            <div class="annotation">
562              
563              <div class="pilwrap ">
564                <a class="pilcrow" href="#section-18">&#182;</a>
565              </div>
566              <p>Matches JavaScript interpolated directly into the source via backticks.</p>
567
568            </div>
569            
570            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">jsToken</span>:<span class="hljs-function"> -&gt;</span>
571    <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <span class="hljs-property">@chunk</span>.charAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> <span class="hljs-string">'`'</span> <span class="hljs-keyword">and</span> match = JSTOKEN.exec <span class="hljs-property">@chunk</span>
572    <span class="hljs-property">@token</span> <span class="hljs-string">'JS'</span>, (script = match[<span class="hljs-number">0</span>])[<span class="hljs-number">1.</span>..-<span class="hljs-number">1</span>], <span class="hljs-number">0</span>, script.length
573    script.length</pre></div></div>
574            
575        </li>
576        
577        
578        <li id="section-19">
579            <div class="annotation">
580              
581              <div class="pilwrap ">
582                <a class="pilcrow" href="#section-19">&#182;</a>
583              </div>
584              <p>Matches regular expression literals, as well as multiline extended ones.
585Lexing regular expressions is difficult to distinguish from division, so we
586borrow some basic heuristics from JavaScript and Ruby.</p>
587
588            </div>
589            
590            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">regexToken</span>:<span class="hljs-function"> -&gt;</span>
591    <span class="hljs-keyword">switch</span>
592      <span class="hljs-keyword">when</span> match = REGEX_ILLEGAL.exec <span class="hljs-property">@chunk</span>
593        <span class="hljs-property">@error</span> <span class="hljs-string">"regular expressions cannot begin with <span class="hljs-subst">#{match[<span class="hljs-number">2</span>]}</span>"</span>,
594          <span class="hljs-attribute">offset</span>: match.index + match[<span class="hljs-number">1</span>].length
595      <span class="hljs-keyword">when</span> match = <span class="hljs-property">@matchWithInterpolations</span> HEREGEX, <span class="hljs-string">'///'</span>
596        {tokens, index} = match
597      <span class="hljs-keyword">when</span> match = REGEX.exec <span class="hljs-property">@chunk</span>
598        [regex, body, closed] = match
599        <span class="hljs-property">@validateEscapes</span> body, <span class="hljs-attribute">isRegex</span>: <span class="hljs-literal">yes</span>, <span class="hljs-attribute">offsetInChunk</span>: <span class="hljs-number">1</span>
600        index = regex.length
601        [..., prev] = <span class="hljs-property">@tokens</span>
602        <span class="hljs-keyword">if</span> prev
603          <span class="hljs-keyword">if</span> prev.spaced <span class="hljs-keyword">and</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> CALLABLE
604            <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> closed <span class="hljs-keyword">or</span> POSSIBLY_DIVISION.test regex
605          <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> NOT_REGEX
606            <span class="hljs-keyword">return</span> <span class="hljs-number">0</span>
607        <span class="hljs-property">@error</span> <span class="hljs-string">'missing / (unclosed regex)'</span> <span class="hljs-keyword">unless</span> closed
608      <span class="hljs-keyword">else</span>
609        <span class="hljs-keyword">return</span> <span class="hljs-number">0</span>
610
611    [flags] = REGEX_FLAGS.exec <span class="hljs-property">@chunk</span>[index..]
612    end = index + flags.length
613    origin = <span class="hljs-property">@makeToken</span> <span class="hljs-string">'REGEX'</span>, <span class="hljs-literal">null</span>, <span class="hljs-number">0</span>, end
614    <span class="hljs-keyword">switch</span>
615      <span class="hljs-keyword">when</span> <span class="hljs-keyword">not</span> VALID_FLAGS.test flags
616        <span class="hljs-property">@error</span> <span class="hljs-string">"invalid regular expression flags <span class="hljs-subst">#{flags}</span>"</span>, <span class="hljs-attribute">offset</span>: index, <span class="hljs-attribute">length</span>: flags.length
617      <span class="hljs-keyword">when</span> regex <span class="hljs-keyword">or</span> tokens.length <span class="hljs-keyword">is</span> <span class="hljs-number">1</span>
618        body ?= <span class="hljs-property">@formatHeregex</span> tokens[<span class="hljs-number">0</span>][<span class="hljs-number">1</span>]
619        <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX'</span>, <span class="hljs-string">"<span class="hljs-subst">#{<span class="hljs-property">@makeDelimitedLiteral</span> body, delimiter: <span class="hljs-string">'/'</span>}</span><span class="hljs-subst">#{flags}</span>"</span>, <span class="hljs-number">0</span>, end, origin
620      <span class="hljs-keyword">else</span>
621        <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX_START'</span>, <span class="hljs-string">'('</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, origin
622        <span class="hljs-property">@token</span> <span class="hljs-string">'IDENTIFIER'</span>, <span class="hljs-string">'RegExp'</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>
623        <span class="hljs-property">@token</span> <span class="hljs-string">'CALL_START'</span>, <span class="hljs-string">'('</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>
624        <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {<span class="hljs-attribute">delimiter</span>: <span class="hljs-string">'"'</span>, <span class="hljs-attribute">double</span>: <span class="hljs-literal">yes</span>}, <span class="hljs-property">@formatHeregex</span>
625        <span class="hljs-keyword">if</span> flags
626          <span class="hljs-property">@token</span> <span class="hljs-string">','</span>, <span class="hljs-string">','</span>, index, <span class="hljs-number">0</span>
627          <span class="hljs-property">@token</span> <span class="hljs-string">'STRING'</span>, <span class="hljs-string">'"'</span> + flags + <span class="hljs-string">'"'</span>, index, flags.length
628        <span class="hljs-property">@token</span> <span class="hljs-string">')'</span>, <span class="hljs-string">')'</span>, end, <span class="hljs-number">0</span>
629        <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX_END'</span>, <span class="hljs-string">')'</span>, end, <span class="hljs-number">0</span>
630
631    end</pre></div></div>
632            
633        </li>
634        
635        
636        <li id="section-20">
637            <div class="annotation">
638              
639              <div class="pilwrap ">
640                <a class="pilcrow" href="#section-20">&#182;</a>
641              </div>
642              <p>Matches newlines, indents, and outdents, and determines which is which.
643If we can detect that the current line is continued onto the next line,
644then the newline is suppressed:</p>
645<pre><code>elements
646  .each( ... )
647  .map( ... )
648</code></pre><p>Keeps track of the level of indentation, because a single outdent token
649can close multiple indents, so we need to know how far in we happen to be.</p>
650
651            </div>
652            
653            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">lineToken</span>:<span class="hljs-function"> -&gt;</span>
654    <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = MULTI_DENT.exec <span class="hljs-property">@chunk</span>
655    indent = match[<span class="hljs-number">0</span>]
656    <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">no</span>
657    size = indent.length - <span class="hljs-number">1</span> - indent.lastIndexOf <span class="hljs-string">'\n'</span>
658    noNewlines = <span class="hljs-property">@unfinished</span>()
659    <span class="hljs-keyword">if</span> size - <span class="hljs-property">@indebt</span> <span class="hljs-keyword">is</span> <span class="hljs-property">@indent</span>
660      <span class="hljs-keyword">if</span> noNewlines <span class="hljs-keyword">then</span> <span class="hljs-property">@suppressNewlines</span>() <span class="hljs-keyword">else</span> <span class="hljs-property">@newlineToken</span> <span class="hljs-number">0</span>
661      <span class="hljs-keyword">return</span> indent.length
662
663    <span class="hljs-keyword">if</span> size &gt; <span class="hljs-property">@indent</span>
664      <span class="hljs-keyword">if</span> noNewlines
665        <span class="hljs-property">@indebt</span> = size - <span class="hljs-property">@indent</span>
666        <span class="hljs-property">@suppressNewlines</span>()
667        <span class="hljs-keyword">return</span> indent.length
668      <span class="hljs-keyword">unless</span> <span class="hljs-property">@tokens</span>.length
669        <span class="hljs-property">@baseIndent</span> = <span class="hljs-property">@indent</span> = size
670        <span class="hljs-keyword">return</span> indent.length
671      diff = size - <span class="hljs-property">@indent</span> + <span class="hljs-property">@outdebt</span>
672      <span class="hljs-property">@token</span> <span class="hljs-string">'INDENT'</span>, diff, indent.length - size, size
673      <span class="hljs-property">@indents</span>.push diff
674      <span class="hljs-property">@ends</span>.push {<span class="hljs-attribute">tag</span>: <span class="hljs-string">'OUTDENT'</span>}
675      <span class="hljs-property">@outdebt</span> = <span class="hljs-property">@indebt</span> = <span class="hljs-number">0</span>
676      <span class="hljs-property">@indent</span> = size
677    <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> size &lt; <span class="hljs-property">@baseIndent</span>
678      <span class="hljs-property">@error</span> <span class="hljs-string">'missing indentation'</span>, <span class="hljs-attribute">offset</span>: indent.length
679    <span class="hljs-keyword">else</span>
680      <span class="hljs-property">@indebt</span> = <span class="hljs-number">0</span>
681      <span class="hljs-property">@outdentToken</span> <span class="hljs-property">@indent</span> - size, noNewlines, indent.length
682    indent.length</pre></div></div>
683            
684        </li>
685        
686        
687        <li id="section-21">
688            <div class="annotation">
689              
690              <div class="pilwrap ">
691                <a class="pilcrow" href="#section-21">&#182;</a>
692              </div>
693              <p>Record an outdent token or multiple tokens, if we happen to be moving back
694inwards past several recorded indents. Sets new @indent value.</p>
695
696            </div>
697            
698            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">outdentToken</span>: <span class="hljs-function"><span class="hljs-params">(moveOut, noNewlines, outdentLength)</span> -&gt;</span>
699    decreasedIndent = <span class="hljs-property">@indent</span> - moveOut
700    <span class="hljs-keyword">while</span> moveOut &gt; <span class="hljs-number">0</span>
701      lastIndent = <span class="hljs-property">@indents</span>[<span class="hljs-property">@indents</span>.length - <span class="hljs-number">1</span>]
702      <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> lastIndent
703        moveOut = <span class="hljs-number">0</span>
704      <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> lastIndent <span class="hljs-keyword">is</span> <span class="hljs-property">@outdebt</span>
705        moveOut -= <span class="hljs-property">@outdebt</span>
706        <span class="hljs-property">@outdebt</span> = <span class="hljs-number">0</span>
707      <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> lastIndent &lt; <span class="hljs-property">@outdebt</span>
708        <span class="hljs-property">@outdebt</span> -= lastIndent
709        moveOut  -= lastIndent
710      <span class="hljs-keyword">else</span>
711        dent = <span class="hljs-property">@indents</span>.pop() + <span class="hljs-property">@outdebt</span>
712        <span class="hljs-keyword">if</span> outdentLength <span class="hljs-keyword">and</span> <span class="hljs-property">@chunk</span>[outdentLength] <span class="hljs-keyword">in</span> INDENTABLE_CLOSERS
713          decreasedIndent -= dent - moveOut
714          moveOut = dent
715        <span class="hljs-property">@outdebt</span> = <span class="hljs-number">0</span></pre></div></div>
716            
717        </li>
718        
719        
720        <li id="section-22">
721            <div class="annotation">
722              
723              <div class="pilwrap ">
724                <a class="pilcrow" href="#section-22">&#182;</a>
725              </div>
726              <p>pair might call outdentToken, so preserve decreasedIndent</p>
727
728            </div>
729            
730            <div class="content"><div class='highlight'><pre>        <span class="hljs-property">@pair</span> <span class="hljs-string">'OUTDENT'</span>
731        <span class="hljs-property">@token</span> <span class="hljs-string">'OUTDENT'</span>, moveOut, <span class="hljs-number">0</span>, outdentLength
732        moveOut -= dent
733    <span class="hljs-property">@outdebt</span> -= moveOut <span class="hljs-keyword">if</span> dent
734    <span class="hljs-property">@tokens</span>.pop() <span class="hljs-keyword">while</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">';'</span>
735
736    <span class="hljs-property">@token</span> <span class="hljs-string">'TERMINATOR'</span>, <span class="hljs-string">'\n'</span>, outdentLength, <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'TERMINATOR'</span> <span class="hljs-keyword">or</span> noNewlines
737    <span class="hljs-property">@indent</span> = decreasedIndent
738    <span class="hljs-keyword">this</span></pre></div></div>
739            
740        </li>
741        
742        
743        <li id="section-23">
744            <div class="annotation">
745              
746              <div class="pilwrap ">
747                <a class="pilcrow" href="#section-23">&#182;</a>
748              </div>
749              <p>Matches and consumes non-meaningful whitespace. Tag the previous token
750as being “spaced”, because there are some cases where it makes a difference.</p>
751
752            </div>
753            
754            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">whitespaceToken</span>:<span class="hljs-function"> -&gt;</span>
755    <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> (match = WHITESPACE.exec <span class="hljs-property">@chunk</span>) <span class="hljs-keyword">or</span>
756                    (nline = <span class="hljs-property">@chunk</span>.charAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> <span class="hljs-string">'\n'</span>)
757    [..., prev] = <span class="hljs-property">@tokens</span>
758    prev[<span class="hljs-keyword">if</span> match <span class="hljs-keyword">then</span> <span class="hljs-string">'spaced'</span> <span class="hljs-keyword">else</span> <span class="hljs-string">'newLine'</span>] = <span class="hljs-literal">true</span> <span class="hljs-keyword">if</span> prev
759    <span class="hljs-keyword">if</span> match <span class="hljs-keyword">then</span> match[<span class="hljs-number">0</span>].length <span class="hljs-keyword">else</span> <span class="hljs-number">0</span></pre></div></div>
760            
761        </li>
762        
763        
764        <li id="section-24">
765            <div class="annotation">
766              
767              <div class="pilwrap ">
768                <a class="pilcrow" href="#section-24">&#182;</a>
769              </div>
770              <p>Generate a newline token. Consecutive newlines get merged together.</p>
771
772            </div>
773            
774            <div class="content"><div class='highlight'><pre>  <span class="hljs-attribute">newlineToken</span>: <span class="hljs-function"><span class="hljs-params">(offset)</span> -&gt;</span>
775    <span class="hljs-property">@tokens</span>.pop() <span class="hljs-keyword">while</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">';'</span>
776    <span class="hljs-property">@token</span> <span class="hljs-string">'TERMINATOR'</span>, <span class="hljs-string">'\n'</span>, offset, <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <s

Large files files are truncated, but you can click here to view the full file