/documentation/docs/lexer.html
HTML | 2024 lines | 1865 code | 159 blank | 0 comment | 0 complexity | 9c4487192ff7778f37ad5ad37144c273 MD5 | raw file
Large files files are truncated, but you can click here to view the full file
1<!DOCTYPE html> 2 3<html> 4<head> 5 <title>lexer.coffee</title> 6 <meta http-equiv="content-type" content="text/html; charset=UTF-8"> 7 <meta name="viewport" content="width=device-width, target-densitydpi=160dpi, initial-scale=1.0; maximum-scale=1.0; user-scalable=0;"> 8 <link rel="stylesheet" media="all" href="docco.css" /> 9</head> 10<body> 11 <div id="container"> 12 <div id="background"></div> 13 14 <ul id="jump_to"> 15 <li> 16 <a class="large" href="javascript:void(0);">Jump To …</a> 17 <a class="small" href="javascript:void(0);">+</a> 18 <div id="jump_wrapper"> 19 <div id="jump_page_wrapper"> 20 <div id="jump_page"> 21 22 23 <a class="source" href="browser.html"> 24 browser.coffee 25 </a> 26 27 28 <a class="source" href="cake.html"> 29 cake.coffee 30 </a> 31 32 33 <a class="source" href="coffee-script.html"> 34 coffee-script.coffee 35 </a> 36 37 38 <a class="source" href="command.html"> 39 command.coffee 40 </a> 41 42 43 <a class="source" href="grammar.html"> 44 grammar.coffee 45 </a> 46 47 48 <a class="source" href="helpers.html"> 49 helpers.coffee 50 </a> 51 52 53 <a class="source" href="index.html"> 54 index.coffee 55 </a> 56 57 58 <a class="source" href="lexer.html"> 59 lexer.coffee 60 </a> 61 62 63 <a class="source" href="nodes.html"> 64 nodes.coffee 65 </a> 66 67 68 <a class="source" href="optparse.html"> 69 optparse.coffee 70 </a> 71 72 73 <a class="source" href="register.html"> 74 register.coffee 75 </a> 76 77 78 <a class="source" href="repl.html"> 79 repl.coffee 80 </a> 81 82 83 <a class="source" href="rewriter.html"> 84 rewriter.coffee 85 </a> 86 87 88 <a class="source" href="scope.html"> 89 scope.litcoffee 90 </a> 91 92 93 <a class="source" href="sourcemap.html"> 94 sourcemap.litcoffee 95 </a> 96 97 </div> 98 </div> 99 </li> 100 </ul> 101 102 <ul class="sections"> 103 104 <li id="title"> 105 <div class="annotation"> 106 <h1>lexer.coffee</h1> 107 </div> 108 </li> 109 110 111 112 <li id="section-1"> 113 <div class="annotation"> 114 115 <div class="pilwrap "> 116 <a class="pilcrow" href="#section-1">¶</a> 117 </div> 118 <p>The CoffeeScript Lexer. Uses a series of token-matching regexes to attempt 119matches against the beginning of the source code. When a match is found, 120a token is produced, we consume the match, and start again. Tokens are in the 121form:</p> 122<pre><code>[tag, value, locationData] 123</code></pre><p>where locationData is {first_line, first_column, last_line, last_column}, which is a 124format that can be fed directly into <a href="http://github.com/zaach/jison">Jison</a>. These 125are read by jison in the <code>parser.lexer</code> function defined in coffee-script.coffee.</p> 126 127 </div> 128 129 <div class="content"><div class='highlight'><pre> 130{Rewriter, INVERSES} = <span class="hljs-built_in">require</span> <span class="hljs-string">'./rewriter'</span></pre></div></div> 131 132 </li> 133 134 135 <li id="section-2"> 136 <div class="annotation"> 137 138 <div class="pilwrap "> 139 <a class="pilcrow" href="#section-2">¶</a> 140 </div> 141 <p>Import the helpers we need.</p> 142 143 </div> 144 145 <div class="content"><div class='highlight'><pre>{count, starts, compact, repeat, invertLiterate, 146locationDataToString, throwSyntaxError} = <span class="hljs-built_in">require</span> <span class="hljs-string">'./helpers'</span></pre></div></div> 147 148 </li> 149 150 151 <li id="section-3"> 152 <div class="annotation"> 153 154 <div class="pilwrap "> 155 <a class="pilcrow" href="#section-3">¶</a> 156 </div> 157 <h2 id="the-lexer-class">The Lexer Class</h2> 158 159 </div> 160 161 </li> 162 163 164 <li id="section-4"> 165 <div class="annotation"> 166 167 <div class="pilwrap "> 168 <a class="pilcrow" href="#section-4">¶</a> 169 </div> 170 171 </div> 172 173 </li> 174 175 176 <li id="section-5"> 177 <div class="annotation"> 178 179 <div class="pilwrap "> 180 <a class="pilcrow" href="#section-5">¶</a> 181 </div> 182 <p>The Lexer class reads a stream of CoffeeScript and divvies it up into tagged 183tokens. Some potential ambiguity in the grammar has been avoided by 184pushing some extra smarts into the Lexer.</p> 185 186 </div> 187 188 <div class="content"><div class='highlight'><pre><span class="hljs-built_in">exports</span>.Lexer = <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Lexer</span></span></pre></div></div> 189 190 </li> 191 192 193 <li id="section-6"> 194 <div class="annotation"> 195 196 <div class="pilwrap "> 197 <a class="pilcrow" href="#section-6">¶</a> 198 </div> 199 <p><strong>tokenize</strong> is the Lexer’s main method. Scan by attempting to match tokens 200one at a time, using a regular expression anchored at the start of the 201remaining code, or a custom recursive token-matching method 202(for interpolations). When the next token has been recorded, we move forward 203within the code past the token, and begin again.</p> 204<p>Each tokenizing method is responsible for returning the number of characters 205it has consumed.</p> 206<p>Before returning the token stream, run it through the <a href="rewriter.html">Rewriter</a>.</p> 207 208 </div> 209 210 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">tokenize</span>: <span class="hljs-function"><span class="hljs-params">(code, opts = {})</span> -></span> 211 <span class="hljs-property">@literate</span> = opts.literate <span class="hljs-comment"># Are we lexing literate CoffeeScript?</span> 212 <span class="hljs-property">@indent</span> = <span class="hljs-number">0</span> <span class="hljs-comment"># The current indentation level.</span> 213 <span class="hljs-property">@baseIndent</span> = <span class="hljs-number">0</span> <span class="hljs-comment"># The overall minimum indentation level</span> 214 <span class="hljs-property">@indebt</span> = <span class="hljs-number">0</span> <span class="hljs-comment"># The over-indentation at the current level.</span> 215 <span class="hljs-property">@outdebt</span> = <span class="hljs-number">0</span> <span class="hljs-comment"># The under-outdentation at the current level.</span> 216 <span class="hljs-property">@indents</span> = [] <span class="hljs-comment"># The stack of all current indentation levels.</span> 217 <span class="hljs-property">@ends</span> = [] <span class="hljs-comment"># The stack for pairing up tokens.</span> 218 <span class="hljs-property">@tokens</span> = [] <span class="hljs-comment"># Stream of parsed tokens in the form `['TYPE', value, location data]`.</span> 219 <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">no</span> <span class="hljs-comment"># Used to recognize FORIN and FOROF tokens.</span> 220 221 <span class="hljs-property">@chunkLine</span> = 222 opts.line <span class="hljs-keyword">or</span> <span class="hljs-number">0</span> <span class="hljs-comment"># The start line for the current @chunk.</span> 223 <span class="hljs-property">@chunkColumn</span> = 224 opts.column <span class="hljs-keyword">or</span> <span class="hljs-number">0</span> <span class="hljs-comment"># The start column of the current @chunk.</span> 225 code = <span class="hljs-property">@clean</span> code <span class="hljs-comment"># The stripped, cleaned original source code.</span></pre></div></div> 226 227 </li> 228 229 230 <li id="section-7"> 231 <div class="annotation"> 232 233 <div class="pilwrap "> 234 <a class="pilcrow" href="#section-7">¶</a> 235 </div> 236 <p>At every position, run through this list of attempted matches, 237short-circuiting if any of them succeed. Their order determines precedence: 238<code>@literalToken</code> is the fallback catch-all.</p> 239 240 </div> 241 242 <div class="content"><div class='highlight'><pre> i = <span class="hljs-number">0</span> 243 <span class="hljs-keyword">while</span> <span class="hljs-property">@chunk</span> = code[i..] 244 consumed = \ 245 <span class="hljs-property">@identifierToken</span>() <span class="hljs-keyword">or</span> 246 <span class="hljs-property">@commentToken</span>() <span class="hljs-keyword">or</span> 247 <span class="hljs-property">@whitespaceToken</span>() <span class="hljs-keyword">or</span> 248 <span class="hljs-property">@lineToken</span>() <span class="hljs-keyword">or</span> 249 <span class="hljs-property">@stringToken</span>() <span class="hljs-keyword">or</span> 250 <span class="hljs-property">@numberToken</span>() <span class="hljs-keyword">or</span> 251 <span class="hljs-property">@regexToken</span>() <span class="hljs-keyword">or</span> 252 <span class="hljs-property">@jsToken</span>() <span class="hljs-keyword">or</span> 253 <span class="hljs-property">@literalToken</span>()</pre></div></div> 254 255 </li> 256 257 258 <li id="section-8"> 259 <div class="annotation"> 260 261 <div class="pilwrap "> 262 <a class="pilcrow" href="#section-8">¶</a> 263 </div> 264 <p>Update position</p> 265 266 </div> 267 268 <div class="content"><div class='highlight'><pre> [<span class="hljs-property">@chunkLine</span>, <span class="hljs-property">@chunkColumn</span>] = <span class="hljs-property">@getLineAndColumnFromChunk</span> consumed 269 270 i += consumed 271 272 <span class="hljs-keyword">return</span> {<span class="hljs-property">@tokens</span>, <span class="hljs-attribute">index</span>: i} <span class="hljs-keyword">if</span> opts.untilBalanced <span class="hljs-keyword">and</span> <span class="hljs-property">@ends</span>.length <span class="hljs-keyword">is</span> <span class="hljs-number">0</span> 273 274 <span class="hljs-property">@closeIndentation</span>() 275 <span class="hljs-property">@error</span> <span class="hljs-string">"missing <span class="hljs-subst">#{end.tag}</span>"</span>, end.origin[<span class="hljs-number">2</span>] <span class="hljs-keyword">if</span> end = <span class="hljs-property">@ends</span>.pop() 276 <span class="hljs-keyword">return</span> <span class="hljs-property">@tokens</span> <span class="hljs-keyword">if</span> opts.rewrite <span class="hljs-keyword">is</span> <span class="hljs-literal">off</span> 277 (<span class="hljs-keyword">new</span> Rewriter).rewrite <span class="hljs-property">@tokens</span></pre></div></div> 278 279 </li> 280 281 282 <li id="section-9"> 283 <div class="annotation"> 284 285 <div class="pilwrap "> 286 <a class="pilcrow" href="#section-9">¶</a> 287 </div> 288 <p>Preprocess the code to remove leading and trailing whitespace, carriage 289returns, etc. If we’re lexing literate CoffeeScript, strip external Markdown 290by removing all lines that aren’t indented by at least four spaces or a tab.</p> 291 292 </div> 293 294 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">clean</span>: <span class="hljs-function"><span class="hljs-params">(code)</span> -></span> 295 code = code.slice(<span class="hljs-number">1</span>) <span class="hljs-keyword">if</span> code.charCodeAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> BOM 296 code = code.replace(<span class="hljs-regexp">/\r/g</span>, <span class="hljs-string">''</span>).replace TRAILING_SPACES, <span class="hljs-string">''</span> 297 <span class="hljs-keyword">if</span> WHITESPACE.test code 298 code = <span class="hljs-string">"\n<span class="hljs-subst">#{code}</span>"</span> 299 <span class="hljs-property">@chunkLine</span>-- 300 code = invertLiterate code <span class="hljs-keyword">if</span> <span class="hljs-property">@literate</span> 301 code</pre></div></div> 302 303 </li> 304 305 306 <li id="section-10"> 307 <div class="annotation"> 308 309 <div class="pilwrap "> 310 <a class="pilcrow" href="#section-10">¶</a> 311 </div> 312 <h2 id="tokenizers">Tokenizers</h2> 313 314 </div> 315 316 </li> 317 318 319 <li id="section-11"> 320 <div class="annotation"> 321 322 <div class="pilwrap "> 323 <a class="pilcrow" href="#section-11">¶</a> 324 </div> 325 326 </div> 327 328 </li> 329 330 331 <li id="section-12"> 332 <div class="annotation"> 333 334 <div class="pilwrap "> 335 <a class="pilcrow" href="#section-12">¶</a> 336 </div> 337 <p>Matches identifying literals: variables, keywords, method names, etc. 338Check to ensure that JavaScript reserved words aren’t being used as 339identifiers. Because CoffeeScript reserves a handful of keywords that are 340allowed in JavaScript, we’re careful not to tag them as keywords when 341referenced as property names here, so you can still do <code>jQuery.is()</code> even 342though <code>is</code> means <code>===</code> otherwise.</p> 343 344 </div> 345 346 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">identifierToken</span>:<span class="hljs-function"> -></span> 347 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = IDENTIFIER.exec <span class="hljs-property">@chunk</span> 348 [input, id, colon] = match</pre></div></div> 349 350 </li> 351 352 353 <li id="section-13"> 354 <div class="annotation"> 355 356 <div class="pilwrap "> 357 <a class="pilcrow" href="#section-13">¶</a> 358 </div> 359 <p>Preserve length of id for location data</p> 360 361 </div> 362 363 <div class="content"><div class='highlight'><pre> idLength = id.length 364 poppedToken = <span class="hljs-literal">undefined</span> 365 366 <span class="hljs-keyword">if</span> id <span class="hljs-keyword">is</span> <span class="hljs-string">'own'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'FOR'</span> 367 <span class="hljs-property">@token</span> <span class="hljs-string">'OWN'</span>, id 368 <span class="hljs-keyword">return</span> id.length 369 <span class="hljs-keyword">if</span> id <span class="hljs-keyword">is</span> <span class="hljs-string">'from'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'YIELD'</span> 370 <span class="hljs-property">@token</span> <span class="hljs-string">'FROM'</span>, id 371 <span class="hljs-keyword">return</span> id.length 372 [..., prev] = <span class="hljs-property">@tokens</span> 373 forcedIdentifier = colon <span class="hljs-keyword">or</span> prev? <span class="hljs-keyword">and</span> 374 (prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> [<span class="hljs-string">'.'</span>, <span class="hljs-string">'?.'</span>, <span class="hljs-string">'::'</span>, <span class="hljs-string">'?::'</span>] <span class="hljs-keyword">or</span> 375 <span class="hljs-keyword">not</span> prev.spaced <span class="hljs-keyword">and</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">is</span> <span class="hljs-string">'@'</span>) 376 tag = <span class="hljs-string">'IDENTIFIER'</span> 377 378 <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> forcedIdentifier <span class="hljs-keyword">and</span> (id <span class="hljs-keyword">in</span> JS_KEYWORDS <span class="hljs-keyword">or</span> id <span class="hljs-keyword">in</span> COFFEE_KEYWORDS) 379 tag = id.toUpperCase() 380 <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'WHEN'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">in</span> LINE_BREAK 381 tag = <span class="hljs-string">'LEADING_WHEN'</span> 382 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'FOR'</span> 383 <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">yes</span> 384 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">is</span> <span class="hljs-string">'UNLESS'</span> 385 tag = <span class="hljs-string">'IF'</span> 386 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">in</span> UNARY 387 tag = <span class="hljs-string">'UNARY'</span> 388 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">in</span> RELATION 389 <span class="hljs-keyword">if</span> tag <span class="hljs-keyword">isnt</span> <span class="hljs-string">'INSTANCEOF'</span> <span class="hljs-keyword">and</span> <span class="hljs-property">@seenFor</span> 390 tag = <span class="hljs-string">'FOR'</span> + tag 391 <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">no</span> 392 <span class="hljs-keyword">else</span> 393 tag = <span class="hljs-string">'RELATION'</span> 394 <span class="hljs-keyword">if</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'!'</span> 395 poppedToken = <span class="hljs-property">@tokens</span>.pop() 396 id = <span class="hljs-string">'!'</span> + id 397 398 <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> JS_FORBIDDEN 399 <span class="hljs-keyword">if</span> forcedIdentifier 400 tag = <span class="hljs-string">'IDENTIFIER'</span> 401 id = <span class="hljs-keyword">new</span> String id 402 id.reserved = <span class="hljs-literal">yes</span> 403 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> RESERVED 404 <span class="hljs-property">@error</span> <span class="hljs-string">"reserved word '<span class="hljs-subst">#{id}</span>'"</span>, <span class="hljs-attribute">length</span>: id.length 405 406 <span class="hljs-keyword">unless</span> forcedIdentifier 407 <span class="hljs-keyword">if</span> id <span class="hljs-keyword">in</span> COFFEE_ALIASES 408 alias = id 409 id = COFFEE_ALIAS_MAP[id] 410 tag = <span class="hljs-keyword">switch</span> id 411 <span class="hljs-keyword">when</span> <span class="hljs-string">'!'</span> <span class="hljs-keyword">then</span> <span class="hljs-string">'UNARY'</span> 412 <span class="hljs-keyword">when</span> <span class="hljs-string">'=='</span>, <span class="hljs-string">'!='</span> <span class="hljs-keyword">then</span> <span class="hljs-string">'COMPARE'</span> 413 <span class="hljs-keyword">when</span> <span class="hljs-string">'&&'</span>, <span class="hljs-string">'||'</span> <span class="hljs-keyword">then</span> <span class="hljs-string">'LOGIC'</span> 414 <span class="hljs-keyword">when</span> <span class="hljs-string">'true'</span>, <span class="hljs-string">'false'</span> <span class="hljs-keyword">then</span> <span class="hljs-string">'BOOL'</span> 415 <span class="hljs-keyword">when</span> <span class="hljs-string">'break'</span>, <span class="hljs-string">'continue'</span> <span class="hljs-keyword">then</span> <span class="hljs-string">'STATEMENT'</span> 416 <span class="hljs-keyword">else</span> tag 417 418 tagToken = <span class="hljs-property">@token</span> tag, id, <span class="hljs-number">0</span>, idLength 419 tagToken.origin = [tag, alias, tagToken[<span class="hljs-number">2</span>]] <span class="hljs-keyword">if</span> alias 420 tagToken.variable = <span class="hljs-keyword">not</span> forcedIdentifier 421 <span class="hljs-keyword">if</span> poppedToken 422 [tagToken[<span class="hljs-number">2</span>].first_line, tagToken[<span class="hljs-number">2</span>].first_column] = 423 [poppedToken[<span class="hljs-number">2</span>].first_line, poppedToken[<span class="hljs-number">2</span>].first_column] 424 <span class="hljs-keyword">if</span> colon 425 colonOffset = input.lastIndexOf <span class="hljs-string">':'</span> 426 <span class="hljs-property">@token</span> <span class="hljs-string">':'</span>, <span class="hljs-string">':'</span>, colonOffset, colon.length 427 428 input.length</pre></div></div> 429 430 </li> 431 432 433 <li id="section-14"> 434 <div class="annotation"> 435 436 <div class="pilwrap "> 437 <a class="pilcrow" href="#section-14">¶</a> 438 </div> 439 <p>Matches numbers, including decimals, hex, and exponential notation. 440Be careful not to interfere with ranges-in-progress.</p> 441 442 </div> 443 444 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">numberToken</span>:<span class="hljs-function"> -></span> 445 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = NUMBER.exec <span class="hljs-property">@chunk</span> 446 number = match[<span class="hljs-number">0</span>] 447 lexedLength = number.length 448 <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0[BOX]/</span>.test number 449 <span class="hljs-property">@error</span> <span class="hljs-string">"radix prefix in '<span class="hljs-subst">#{number}</span>' must be lowercase"</span>, <span class="hljs-attribute">offset</span>: <span class="hljs-number">1</span> 450 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/E/</span>.test(number) <span class="hljs-keyword">and</span> <span class="hljs-keyword">not</span> <span class="hljs-regexp">/^0x/</span>.test number 451 <span class="hljs-property">@error</span> <span class="hljs-string">"exponential notation in '<span class="hljs-subst">#{number}</span>' must be indicated with a lowercase 'e'"</span>, 452 <span class="hljs-attribute">offset</span>: number.indexOf(<span class="hljs-string">'E'</span>) 453 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0\d*[89]/</span>.test number 454 <span class="hljs-property">@error</span> <span class="hljs-string">"decimal literal '<span class="hljs-subst">#{number}</span>' must not be prefixed with '0'"</span>, <span class="hljs-attribute">length</span>: lexedLength 455 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> <span class="hljs-regexp">/^0\d+/</span>.test number 456 <span class="hljs-property">@error</span> <span class="hljs-string">"octal literal '<span class="hljs-subst">#{number}</span>' must be prefixed with '0o'"</span>, <span class="hljs-attribute">length</span>: lexedLength 457 <span class="hljs-keyword">if</span> octalLiteral = <span class="hljs-regexp">/^0o([0-7]+)/</span>.exec number 458 number = <span class="hljs-string">'0x'</span> + parseInt(octalLiteral[<span class="hljs-number">1</span>], <span class="hljs-number">8</span>).toString <span class="hljs-number">16</span> 459 <span class="hljs-keyword">if</span> binaryLiteral = <span class="hljs-regexp">/^0b([01]+)/</span>.exec number 460 number = <span class="hljs-string">'0x'</span> + parseInt(binaryLiteral[<span class="hljs-number">1</span>], <span class="hljs-number">2</span>).toString <span class="hljs-number">16</span> 461 <span class="hljs-property">@token</span> <span class="hljs-string">'NUMBER'</span>, number, <span class="hljs-number">0</span>, lexedLength 462 lexedLength</pre></div></div> 463 464 </li> 465 466 467 <li id="section-15"> 468 <div class="annotation"> 469 470 <div class="pilwrap "> 471 <a class="pilcrow" href="#section-15">¶</a> 472 </div> 473 <p>Matches strings, including multi-line strings, as well as heredocs, with or without 474interpolation.</p> 475 476 </div> 477 478 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">stringToken</span>:<span class="hljs-function"> -></span> 479 [quote] = STRING_START.exec(<span class="hljs-property">@chunk</span>) || [] 480 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> quote 481 regex = <span class="hljs-keyword">switch</span> quote 482 <span class="hljs-keyword">when</span> <span class="hljs-string">"'"</span> <span class="hljs-keyword">then</span> STRING_SINGLE 483 <span class="hljs-keyword">when</span> <span class="hljs-string">'"'</span> <span class="hljs-keyword">then</span> STRING_DOUBLE 484 <span class="hljs-keyword">when</span> <span class="hljs-string">"'''"</span> <span class="hljs-keyword">then</span> HEREDOC_SINGLE 485 <span class="hljs-keyword">when</span> <span class="hljs-string">'"""'</span> <span class="hljs-keyword">then</span> HEREDOC_DOUBLE 486 heredoc = quote.length <span class="hljs-keyword">is</span> <span class="hljs-number">3</span> 487 488 {tokens, <span class="hljs-attribute">index</span>: end} = <span class="hljs-property">@matchWithInterpolations</span> regex, quote 489 $ = tokens.length - <span class="hljs-number">1</span> 490 491 delimiter = quote.charAt(<span class="hljs-number">0</span>) 492 <span class="hljs-keyword">if</span> heredoc</pre></div></div> 493 494 </li> 495 496 497 <li id="section-16"> 498 <div class="annotation"> 499 500 <div class="pilwrap "> 501 <a class="pilcrow" href="#section-16">¶</a> 502 </div> 503 <p>Find the smallest indentation. It will be removed from all lines later.</p> 504 505 </div> 506 507 <div class="content"><div class='highlight'><pre> indent = <span class="hljs-literal">null</span> 508 doc = (token[<span class="hljs-number">1</span>] <span class="hljs-keyword">for</span> token, i <span class="hljs-keyword">in</span> tokens <span class="hljs-keyword">when</span> token[<span class="hljs-number">0</span>] <span class="hljs-keyword">is</span> <span class="hljs-string">'NEOSTRING'</span>).join <span class="hljs-string">'#{}'</span> 509 <span class="hljs-keyword">while</span> match = HEREDOC_INDENT.exec doc 510 attempt = match[<span class="hljs-number">1</span>] 511 indent = attempt <span class="hljs-keyword">if</span> indent <span class="hljs-keyword">is</span> <span class="hljs-literal">null</span> <span class="hljs-keyword">or</span> <span class="hljs-number">0</span> < attempt.length < indent.length 512 indentRegex = <span class="hljs-regexp">/// ^<span class="hljs-subst">#{indent}</span> ///</span>gm <span class="hljs-keyword">if</span> indent 513 <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {delimiter}, <span class="hljs-function"><span class="hljs-params">(value, i)</span> =></span> 514 value = <span class="hljs-property">@formatString</span> value 515 value = value.replace LEADING_BLANK_LINE, <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> i <span class="hljs-keyword">is</span> <span class="hljs-number">0</span> 516 value = value.replace TRAILING_BLANK_LINE, <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> i <span class="hljs-keyword">is</span> $ 517 value = value.replace indentRegex, <span class="hljs-string">''</span> <span class="hljs-keyword">if</span> indentRegex 518 value 519 <span class="hljs-keyword">else</span> 520 <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {delimiter}, <span class="hljs-function"><span class="hljs-params">(value, i)</span> =></span> 521 value = <span class="hljs-property">@formatString</span> value 522 value = value.replace SIMPLE_STRING_OMIT, <span class="hljs-function"><span class="hljs-params">(match, offset)</span> -></span> 523 <span class="hljs-keyword">if</span> (i <span class="hljs-keyword">is</span> <span class="hljs-number">0</span> <span class="hljs-keyword">and</span> offset <span class="hljs-keyword">is</span> <span class="hljs-number">0</span>) <span class="hljs-keyword">or</span> 524 (i <span class="hljs-keyword">is</span> $ <span class="hljs-keyword">and</span> offset + match.length <span class="hljs-keyword">is</span> value.length) 525 <span class="hljs-string">''</span> 526 <span class="hljs-keyword">else</span> 527 <span class="hljs-string">' '</span> 528 value 529 530 end</pre></div></div> 531 532 </li> 533 534 535 <li id="section-17"> 536 <div class="annotation"> 537 538 <div class="pilwrap "> 539 <a class="pilcrow" href="#section-17">¶</a> 540 </div> 541 <p>Matches and consumes comments.</p> 542 543 </div> 544 545 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">commentToken</span>:<span class="hljs-function"> -></span> 546 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = <span class="hljs-property">@chunk</span>.match COMMENT 547 [comment, here] = match 548 <span class="hljs-keyword">if</span> here 549 <span class="hljs-keyword">if</span> match = HERECOMMENT_ILLEGAL.exec comment 550 <span class="hljs-property">@error</span> <span class="hljs-string">"block comments cannot contain <span class="hljs-subst">#{match[<span class="hljs-number">0</span>]}</span>"</span>, 551 <span class="hljs-attribute">offset</span>: match.index, <span class="hljs-attribute">length</span>: match[<span class="hljs-number">0</span>].length 552 <span class="hljs-keyword">if</span> here.indexOf(<span class="hljs-string">'\n'</span>) >= <span class="hljs-number">0</span> 553 here = here.replace <span class="hljs-regexp">/// \n <span class="hljs-subst">#{repeat <span class="hljs-string">' '</span>, <span class="hljs-property">@indent</span>}</span> ///</span>g, <span class="hljs-string">'\n'</span> 554 <span class="hljs-property">@token</span> <span class="hljs-string">'HERECOMMENT'</span>, here, <span class="hljs-number">0</span>, comment.length 555 comment.length</pre></div></div> 556 557 </li> 558 559 560 <li id="section-18"> 561 <div class="annotation"> 562 563 <div class="pilwrap "> 564 <a class="pilcrow" href="#section-18">¶</a> 565 </div> 566 <p>Matches JavaScript interpolated directly into the source via backticks.</p> 567 568 </div> 569 570 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">jsToken</span>:<span class="hljs-function"> -></span> 571 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <span class="hljs-property">@chunk</span>.charAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> <span class="hljs-string">'`'</span> <span class="hljs-keyword">and</span> match = JSTOKEN.exec <span class="hljs-property">@chunk</span> 572 <span class="hljs-property">@token</span> <span class="hljs-string">'JS'</span>, (script = match[<span class="hljs-number">0</span>])[<span class="hljs-number">1.</span>..-<span class="hljs-number">1</span>], <span class="hljs-number">0</span>, script.length 573 script.length</pre></div></div> 574 575 </li> 576 577 578 <li id="section-19"> 579 <div class="annotation"> 580 581 <div class="pilwrap "> 582 <a class="pilcrow" href="#section-19">¶</a> 583 </div> 584 <p>Matches regular expression literals, as well as multiline extended ones. 585Lexing regular expressions is difficult to distinguish from division, so we 586borrow some basic heuristics from JavaScript and Ruby.</p> 587 588 </div> 589 590 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">regexToken</span>:<span class="hljs-function"> -></span> 591 <span class="hljs-keyword">switch</span> 592 <span class="hljs-keyword">when</span> match = REGEX_ILLEGAL.exec <span class="hljs-property">@chunk</span> 593 <span class="hljs-property">@error</span> <span class="hljs-string">"regular expressions cannot begin with <span class="hljs-subst">#{match[<span class="hljs-number">2</span>]}</span>"</span>, 594 <span class="hljs-attribute">offset</span>: match.index + match[<span class="hljs-number">1</span>].length 595 <span class="hljs-keyword">when</span> match = <span class="hljs-property">@matchWithInterpolations</span> HEREGEX, <span class="hljs-string">'///'</span> 596 {tokens, index} = match 597 <span class="hljs-keyword">when</span> match = REGEX.exec <span class="hljs-property">@chunk</span> 598 [regex, body, closed] = match 599 <span class="hljs-property">@validateEscapes</span> body, <span class="hljs-attribute">isRegex</span>: <span class="hljs-literal">yes</span>, <span class="hljs-attribute">offsetInChunk</span>: <span class="hljs-number">1</span> 600 index = regex.length 601 [..., prev] = <span class="hljs-property">@tokens</span> 602 <span class="hljs-keyword">if</span> prev 603 <span class="hljs-keyword">if</span> prev.spaced <span class="hljs-keyword">and</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> CALLABLE 604 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> closed <span class="hljs-keyword">or</span> POSSIBLY_DIVISION.test regex 605 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> prev[<span class="hljs-number">0</span>] <span class="hljs-keyword">in</span> NOT_REGEX 606 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> 607 <span class="hljs-property">@error</span> <span class="hljs-string">'missing / (unclosed regex)'</span> <span class="hljs-keyword">unless</span> closed 608 <span class="hljs-keyword">else</span> 609 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> 610 611 [flags] = REGEX_FLAGS.exec <span class="hljs-property">@chunk</span>[index..] 612 end = index + flags.length 613 origin = <span class="hljs-property">@makeToken</span> <span class="hljs-string">'REGEX'</span>, <span class="hljs-literal">null</span>, <span class="hljs-number">0</span>, end 614 <span class="hljs-keyword">switch</span> 615 <span class="hljs-keyword">when</span> <span class="hljs-keyword">not</span> VALID_FLAGS.test flags 616 <span class="hljs-property">@error</span> <span class="hljs-string">"invalid regular expression flags <span class="hljs-subst">#{flags}</span>"</span>, <span class="hljs-attribute">offset</span>: index, <span class="hljs-attribute">length</span>: flags.length 617 <span class="hljs-keyword">when</span> regex <span class="hljs-keyword">or</span> tokens.length <span class="hljs-keyword">is</span> <span class="hljs-number">1</span> 618 body ?= <span class="hljs-property">@formatHeregex</span> tokens[<span class="hljs-number">0</span>][<span class="hljs-number">1</span>] 619 <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX'</span>, <span class="hljs-string">"<span class="hljs-subst">#{<span class="hljs-property">@makeDelimitedLiteral</span> body, delimiter: <span class="hljs-string">'/'</span>}</span><span class="hljs-subst">#{flags}</span>"</span>, <span class="hljs-number">0</span>, end, origin 620 <span class="hljs-keyword">else</span> 621 <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX_START'</span>, <span class="hljs-string">'('</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, origin 622 <span class="hljs-property">@token</span> <span class="hljs-string">'IDENTIFIER'</span>, <span class="hljs-string">'RegExp'</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span> 623 <span class="hljs-property">@token</span> <span class="hljs-string">'CALL_START'</span>, <span class="hljs-string">'('</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span> 624 <span class="hljs-property">@mergeInterpolationTokens</span> tokens, {<span class="hljs-attribute">delimiter</span>: <span class="hljs-string">'"'</span>, <span class="hljs-attribute">double</span>: <span class="hljs-literal">yes</span>}, <span class="hljs-property">@formatHeregex</span> 625 <span class="hljs-keyword">if</span> flags 626 <span class="hljs-property">@token</span> <span class="hljs-string">','</span>, <span class="hljs-string">','</span>, index, <span class="hljs-number">0</span> 627 <span class="hljs-property">@token</span> <span class="hljs-string">'STRING'</span>, <span class="hljs-string">'"'</span> + flags + <span class="hljs-string">'"'</span>, index, flags.length 628 <span class="hljs-property">@token</span> <span class="hljs-string">')'</span>, <span class="hljs-string">')'</span>, end, <span class="hljs-number">0</span> 629 <span class="hljs-property">@token</span> <span class="hljs-string">'REGEX_END'</span>, <span class="hljs-string">')'</span>, end, <span class="hljs-number">0</span> 630 631 end</pre></div></div> 632 633 </li> 634 635 636 <li id="section-20"> 637 <div class="annotation"> 638 639 <div class="pilwrap "> 640 <a class="pilcrow" href="#section-20">¶</a> 641 </div> 642 <p>Matches newlines, indents, and outdents, and determines which is which. 643If we can detect that the current line is continued onto the next line, 644then the newline is suppressed:</p> 645<pre><code>elements 646 .each( ... ) 647 .map( ... ) 648</code></pre><p>Keeps track of the level of indentation, because a single outdent token 649can close multiple indents, so we need to know how far in we happen to be.</p> 650 651 </div> 652 653 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">lineToken</span>:<span class="hljs-function"> -></span> 654 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> match = MULTI_DENT.exec <span class="hljs-property">@chunk</span> 655 indent = match[<span class="hljs-number">0</span>] 656 <span class="hljs-property">@seenFor</span> = <span class="hljs-literal">no</span> 657 size = indent.length - <span class="hljs-number">1</span> - indent.lastIndexOf <span class="hljs-string">'\n'</span> 658 noNewlines = <span class="hljs-property">@unfinished</span>() 659 <span class="hljs-keyword">if</span> size - <span class="hljs-property">@indebt</span> <span class="hljs-keyword">is</span> <span class="hljs-property">@indent</span> 660 <span class="hljs-keyword">if</span> noNewlines <span class="hljs-keyword">then</span> <span class="hljs-property">@suppressNewlines</span>() <span class="hljs-keyword">else</span> <span class="hljs-property">@newlineToken</span> <span class="hljs-number">0</span> 661 <span class="hljs-keyword">return</span> indent.length 662 663 <span class="hljs-keyword">if</span> size > <span class="hljs-property">@indent</span> 664 <span class="hljs-keyword">if</span> noNewlines 665 <span class="hljs-property">@indebt</span> = size - <span class="hljs-property">@indent</span> 666 <span class="hljs-property">@suppressNewlines</span>() 667 <span class="hljs-keyword">return</span> indent.length 668 <span class="hljs-keyword">unless</span> <span class="hljs-property">@tokens</span>.length 669 <span class="hljs-property">@baseIndent</span> = <span class="hljs-property">@indent</span> = size 670 <span class="hljs-keyword">return</span> indent.length 671 diff = size - <span class="hljs-property">@indent</span> + <span class="hljs-property">@outdebt</span> 672 <span class="hljs-property">@token</span> <span class="hljs-string">'INDENT'</span>, diff, indent.length - size, size 673 <span class="hljs-property">@indents</span>.push diff 674 <span class="hljs-property">@ends</span>.push {<span class="hljs-attribute">tag</span>: <span class="hljs-string">'OUTDENT'</span>} 675 <span class="hljs-property">@outdebt</span> = <span class="hljs-property">@indebt</span> = <span class="hljs-number">0</span> 676 <span class="hljs-property">@indent</span> = size 677 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> size < <span class="hljs-property">@baseIndent</span> 678 <span class="hljs-property">@error</span> <span class="hljs-string">'missing indentation'</span>, <span class="hljs-attribute">offset</span>: indent.length 679 <span class="hljs-keyword">else</span> 680 <span class="hljs-property">@indebt</span> = <span class="hljs-number">0</span> 681 <span class="hljs-property">@outdentToken</span> <span class="hljs-property">@indent</span> - size, noNewlines, indent.length 682 indent.length</pre></div></div> 683 684 </li> 685 686 687 <li id="section-21"> 688 <div class="annotation"> 689 690 <div class="pilwrap "> 691 <a class="pilcrow" href="#section-21">¶</a> 692 </div> 693 <p>Record an outdent token or multiple tokens, if we happen to be moving back 694inwards past several recorded indents. Sets new @indent value.</p> 695 696 </div> 697 698 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">outdentToken</span>: <span class="hljs-function"><span class="hljs-params">(moveOut, noNewlines, outdentLength)</span> -></span> 699 decreasedIndent = <span class="hljs-property">@indent</span> - moveOut 700 <span class="hljs-keyword">while</span> moveOut > <span class="hljs-number">0</span> 701 lastIndent = <span class="hljs-property">@indents</span>[<span class="hljs-property">@indents</span>.length - <span class="hljs-number">1</span>] 702 <span class="hljs-keyword">if</span> <span class="hljs-keyword">not</span> lastIndent 703 moveOut = <span class="hljs-number">0</span> 704 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> lastIndent <span class="hljs-keyword">is</span> <span class="hljs-property">@outdebt</span> 705 moveOut -= <span class="hljs-property">@outdebt</span> 706 <span class="hljs-property">@outdebt</span> = <span class="hljs-number">0</span> 707 <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> lastIndent < <span class="hljs-property">@outdebt</span> 708 <span class="hljs-property">@outdebt</span> -= lastIndent 709 moveOut -= lastIndent 710 <span class="hljs-keyword">else</span> 711 dent = <span class="hljs-property">@indents</span>.pop() + <span class="hljs-property">@outdebt</span> 712 <span class="hljs-keyword">if</span> outdentLength <span class="hljs-keyword">and</span> <span class="hljs-property">@chunk</span>[outdentLength] <span class="hljs-keyword">in</span> INDENTABLE_CLOSERS 713 decreasedIndent -= dent - moveOut 714 moveOut = dent 715 <span class="hljs-property">@outdebt</span> = <span class="hljs-number">0</span></pre></div></div> 716 717 </li> 718 719 720 <li id="section-22"> 721 <div class="annotation"> 722 723 <div class="pilwrap "> 724 <a class="pilcrow" href="#section-22">¶</a> 725 </div> 726 <p>pair might call outdentToken, so preserve decreasedIndent</p> 727 728 </div> 729 730 <div class="content"><div class='highlight'><pre> <span class="hljs-property">@pair</span> <span class="hljs-string">'OUTDENT'</span> 731 <span class="hljs-property">@token</span> <span class="hljs-string">'OUTDENT'</span>, moveOut, <span class="hljs-number">0</span>, outdentLength 732 moveOut -= dent 733 <span class="hljs-property">@outdebt</span> -= moveOut <span class="hljs-keyword">if</span> dent 734 <span class="hljs-property">@tokens</span>.pop() <span class="hljs-keyword">while</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">';'</span> 735 736 <span class="hljs-property">@token</span> <span class="hljs-string">'TERMINATOR'</span>, <span class="hljs-string">'\n'</span>, outdentLength, <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <span class="hljs-property">@tag</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">'TERMINATOR'</span> <span class="hljs-keyword">or</span> noNewlines 737 <span class="hljs-property">@indent</span> = decreasedIndent 738 <span class="hljs-keyword">this</span></pre></div></div> 739 740 </li> 741 742 743 <li id="section-23"> 744 <div class="annotation"> 745 746 <div class="pilwrap "> 747 <a class="pilcrow" href="#section-23">¶</a> 748 </div> 749 <p>Matches and consumes non-meaningful whitespace. Tag the previous token 750as being “spaced”, because there are some cases where it makes a difference.</p> 751 752 </div> 753 754 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">whitespaceToken</span>:<span class="hljs-function"> -></span> 755 <span class="hljs-keyword">return</span> <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> (match = WHITESPACE.exec <span class="hljs-property">@chunk</span>) <span class="hljs-keyword">or</span> 756 (nline = <span class="hljs-property">@chunk</span>.charAt(<span class="hljs-number">0</span>) <span class="hljs-keyword">is</span> <span class="hljs-string">'\n'</span>) 757 [..., prev] = <span class="hljs-property">@tokens</span> 758 prev[<span class="hljs-keyword">if</span> match <span class="hljs-keyword">then</span> <span class="hljs-string">'spaced'</span> <span class="hljs-keyword">else</span> <span class="hljs-string">'newLine'</span>] = <span class="hljs-literal">true</span> <span class="hljs-keyword">if</span> prev 759 <span class="hljs-keyword">if</span> match <span class="hljs-keyword">then</span> match[<span class="hljs-number">0</span>].length <span class="hljs-keyword">else</span> <span class="hljs-number">0</span></pre></div></div> 760 761 </li> 762 763 764 <li id="section-24"> 765 <div class="annotation"> 766 767 <div class="pilwrap "> 768 <a class="pilcrow" href="#section-24">¶</a> 769 </div> 770 <p>Generate a newline token. Consecutive newlines get merged together.</p> 771 772 </div> 773 774 <div class="content"><div class='highlight'><pre> <span class="hljs-attribute">newlineToken</span>: <span class="hljs-function"><span class="hljs-params">(offset)</span> -></span> 775 <span class="hljs-property">@tokens</span>.pop() <span class="hljs-keyword">while</span> <span class="hljs-property">@value</span>() <span class="hljs-keyword">is</span> <span class="hljs-string">';'</span> 776 <span class="hljs-property">@token</span> <span class="hljs-string">'TERMINATOR'</span>, <span class="hljs-string">'\n'</span>, offset, <span class="hljs-number">0</span> <span class="hljs-keyword">unless</span> <s…
Large files files are truncated, but you can click here to view the full file