lexer.html | searchcode

/lib/antlr-2.7.5/doc/lexer.html

Large files files are truncated, but you can click here to view the full file

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
	<title>Lexical Analysis with ANTLR</title> 
</head>
<body bgcolor="#FFFFFF">
<h2><a id="Lexical_Analysis_with_ANTLR"  name="Lexical_Analysis_with_ANTLR"name="_bb1">Lexical Analysis with ANTLR</a></h2> 
<p>
	A <em>lexer</em> (often called a scanner) breaks up an input stream of characters into vocabulary symbols for a parser, which applies a grammatical structure to that symbol stream. Because ANTLR employs the same recognition mechanism for lexing, parsing, and tree parsing, ANTLR-generated lexers are much stronger than DFA-based lexers such as those generated by DLG (from PCCTS 1.33) and lex.
</p>
<p>
	The increase in lexing power comes at the cost of some inconvenience in lexer specification and indeed requires a serious shift your thoughts about lexical analysis. See a <a href="lexer.html#dfacompare">comparison of LL(k) and DFA-based lexical analysis</a>.
</p>
<p>
	ANTLR generates predicated-LL(k) lexers, which means that you can have semantic and syntactic predicates and use k&gt;1 lookahead. The other advantages are:
<ul>
	<li>
		You can actually read and debug the output as its very similar to what you would build by hand.
	</li>
	<li>
		The syntax for specifying lexical structure is the same for lexers, parsers, and tree parsers.
	</li>
	<li>
		You can have actions executed during the recognition of a single token.
	</li>
	<li>
		You can recognize complicated tokens such as HTML tags or &quot;executable&quot; comments like the javadoc <font face="Courier New">@</font>-tags inside <font size="2" face="Courier New">/** ... */</font> comments. The lexer has a stack, unlike a DFA, so you can match nested structures such as nested comments.
	</li>
</ul>
<p>
	The overall structure of a lexer is:
</p>
<pre>class MyLexer extends Lexer;
options {
  <em>some options</em>
}
{<em>
  lexer class members</em>
}
<em>lexical rules</em></pre>

<h3><a id=""  name=""name="_bb2"></a><a name="lexicalrules">Lexical Rules</a></h3> 
<p>
	Rules defined within a lexer grammar must have a name beginning with an uppercase letter. These rules implicitly match characters on the input stream instead of tokens on the token stream. Referenced grammar elements include token references (implicit lexer rule references), characters, and strings. Lexer rules are processed in the exact same manner as parser rules and, hence, may specify arguments and return values; further, lexer rules can also have local variables and use recursion. The following rule defines a rule called <font size="2" face="Courier New">ID</font> that is available as a token type in the parser.
</p>
<pre>ID : ( 'a'..'z' )+
   ;</pre> 
<p>
	This rule would become part of the resulting lexer and would appear as a method called <font size="2" face="Courier New">mID()</font> that looks sort of like this:

<tt><pre>
    public final void mID(...)
        throws RecognitionException,
               CharStreamException, TokenStreamException
    {
        ...
        _loop3:
        do {
            if (((LA(1) >= 'a' && LA(1) <= 'z'))) {
                matchRange('a','z');
            }
        } while (...);
        ...
    }
</pre></tt>

<p>
It is a good idea to become familiar with ANTLR's output--the generated lexers are human-readable and make a lot of concepts more transparent.

<h4><a id="Skipping_characters"  name="Skipping_characters"name="_bb4">Skipping characters</a></h4> 
<p>
	To have the characters matched by a rule ignored, set the token type to<font size="2" face="Courier New"> Token.SKIP</font>. For example,
</p>
<pre>WS : ( ' ' | '\t' | '\n' { newline(); } | '\r' )+
     { $setType(Token.SKIP); }
   ;</pre>

Skipped tokens force the lexer to reset and try for another
token.  Skipped tokens are never sent back to the parser.

<h4><a id="Distinguishing_between_lexer_rules"  name="Distinguishing_between_lexer_rules"name="_bb5">Distinguishing between lexer rules</a></h4> 

<p>As with most lexer generators like <tt>lex</tt>, you simply list a
set of lexical rules that match tokens.  The tool then automatically
generates code to map the next input character(s) to a rule likely to
match.  Because ANTLR generates recursive-descent lexers just like it
does for parsers and tree parsers, ANTLR automatically generates a
method for a fictitious rule called <tt>nextToken</tt> that predicts
which of your lexer rules will match upon seeing the character
lookahead.  You can think of this method as just a big "switch" that
routes recognition flow to the appropriate rule (the code may be much
more complicated than a simple <tt>switch</tt>-statement, however).
Method <tt>nextToken</tt> is the only method of <tt>TokenStream</tt>
(in Java):

<tt><pre>
public interface TokenStream {
    public Token nextToken() throws TokenStreamException;
}
</pre></tt>

A parser feeds off a lookahead buffer and the buffer pulls from any
<tt>TokenStream</tt>.

Consider the following two ANTLR lexer rules:

<tt><pre>
INT : ('0'..'9')+;
WS : ' ' | '\t' | '\r' | '\n';
</pre></tt>

<p>
You will see something like the following method in lexer generated by
ANTLR:

<tt><pre>
public Token nextToken() throws TokenStreamException {
    ...
    for (;;) {
        Token _token = null;
        int _ttype = Token.INVALID_TYPE;
        resetText();
        ...
        switch (LA(1)) {
          case '0': case '1': case '2': case '3':
          case '4': case '5': case '6': case '7':
          case '8': case '9':
            mINT(); break;
          case '\t': case '\n': case '\r': case ' ':
            mWS(); break;
          default: // error
        }
        ...
    }
}
</pre></tt>

<p> <b>What happens when the same character predicts more than a single
lexical rule</b>?  ANTLR generates an nondeterminism warning between the
offending rules, indicating you need to make sure your rules do not
have common left-prefixes.  ANTLR does not follow the common lexer
rule of &quot;first definition wins&quot; (the alternatives within a
rule, however, still follow this rule). Instead, sufficient power is
given to handle the two most common cases of ambiguity, namely
&quot;keywords vs. identifiers&quot;, and &quot;common prefixes&quot;;
and for especially nasty cases you can use syntactic or semantic
predicates.</p>

<p> <b>What if you want to break up the definition of a complicated
rule into multiple rules</b>? Surely you don't want every rule to
result in a complete Token object in this case. Some rules are only
around to help other rules construct tokens. To distinguish these
"helper" rules from rules that result in tokens, use the
<tt>protected</tt> modifier. This overloading of the access-visibility
Java term occurs because if the rule is not visible, it cannot be
"seen" by the parser (yes, this nomeclature sucks).  See also <a
href="http://www.jguru.com/faq/view.jsp?EID=125"><b>What is a
"protected" lexer rule</b></a>.

<p>
Another, more practical, way to look at this is to note that only
non-protected rules get called by <tt>nextToken</tt> and, hence, only
non-protected rules can generate tokens that get shoved down the
TokenStream pipe to the parser.


<h4><a id="Return_values"  name="Return_values"name="_bb3">Return values</a></h4> 
<p>
	All rules return a token object (conceptually) automatically, which contains the text matched for the rule and its token type at least.&nbsp; To specify a user-defined return value, define a return value and set it in an action:
</p>
<pre>protected
INT returns [int v]
    :   (&#145;0&#146;..&#146;9&#146;)+ { v=Integer.valueOf($getText); }
    ;</pre> 
<p>
Note that only protected rules can have a return type since regular lexer rules generally are invoked by <tt>nextToken()</tt> and the parser cannot access the return value, leading to confusion.
</p>

<h3><a id="Predicated-LL(k)_Lexing"  name="Predicated-LL(k)_Lexing"name="_predllk">Predicated-LL(k) Lexing</a></h3> 
<p>

<p>
	Lexer rules allow your parser to match <i>context-free</i> structures on the input character stream as opposed to the much weaker <i>regular</i> structures (using a DFA--deterministic finite automaton). For example, consider that matching nested curly braces with a DFA must be done using a counter whereas nested curlies are trivially matched with a context-free grammar: 
</p>
<pre><tt>ACTION
    :   '{' ( ACTION | ~'}' )* '}'
    ;</tt>    </pre> 
<p>
The recursion from rule ACTION to ACTION, of course, is the dead giveaway that this is not an ordinary lexer rule. 
</p>
<p>
	Because the same algorithms are used to analyze lexer and parser rules, lexer rules may use more than a single symbol of lookahead, can use semantic predicates, and can specify syntactic predicates to look arbitrarily ahead, thus, providing recognition capabilities beyond the LL(k) languages into the <i>context-sensitive</i>. Here is a simple example that requires k&gt;1 lookahead: 
</p>
<pre><tt>ESCAPE_CHAR
    :   '\\' 't' // two char of lookahead needed,
    |   '\\' 'n' // due to common left-prefix
    ;</tt>    </pre> 
<p>
	To illustrate the use of syntactic predicates for lexer rules, consider the problem of distinguishing between floating point numbers and ranges in Pascal. Input <tt>3..4</tt> must be broken up into 3 tokens: <tt>INT</tt>, <tt>RANGE</tt>, followed by <tt>INT</tt>. Input <tt>3.4</tt>, on the other hand, must be sent to the parser as a <tt>REAL</tt>. The trouble is that the series of digits before the first <tt>'.'</tt> can be arbitrarily long. The scanner then must consume the first <tt>'.'</tt> to see if the next character is a <tt>'.'</tt>, which would imply that it must back up and consider the first series of digits an integer. Using a non-backtracking lexer makes this task very difficult; without bracktracking, your lexer has to be able to respond with more than a single token at one time. However, a syntactic predicate can be used to specify what arbitrary lookahead is necessary: 
</p>
<pre><tt>class Pascal extends Parser;

prog:   INT
        (   RANGE INT
            { System.out.println(&quot;INT .. INT&quot;); }
        |   EOF
            { System.out.println(&quot;plain old INT&quot;); }
        )
    |   REAL { System.out.println(&quot;token REAL&quot;); }
    ;

class LexPascal extends Lexer;

WS  :   (' '
    |   '\t'
    |   '\n'
    |   '\r')+
        { $setType(Token.SKIP); }
    ;

protected
INT :   ('0'..'9')+
    ;

protected
REAL:   INT '.' INT
    ;

RANGE
    :   &quot;..&quot;
    ;

RANGE_OR_INT
    :   ( INT &quot;..&quot; ) =&gt; INT  { $setType(INT); }
    |   ( INT '.' )  =&gt; REAL { $setType(REAL); }
    |   INT                  { $setType(INT); }
    ;</tt>    </pre> 
<p>
	ANTLR lexer rules are even able to handle FORTRAN assignments and other difficult lexical constructs. Consider the following <tt>DO</tt> loop: 
</p>
<pre><tt>DO 100 I = 1,10</tt></pre> 
<p>
	If the comma were replaced with a period, the loop would become an assignment to a weird variable called &quot;<tt>DO100I</tt>&quot;: 
</p>
<pre><tt>DO 100 I = 1.10</tt></pre> 
<p>
	The following rules correctly differentiate the two cases:
</p>
<pre>DO_OR_VAR
    :   (DO_HEADER)=&gt; &quot;DO&quot; { <tt>$setType(</tt>DO); }
    |   VARIABLE { <tt>$setType(</tt>VARIABLE); }
    ;

protected
DO_HEADER
options { ignore=WS; }
    :   &quot;DO&quot; INT VARIABLE '=' EXPR ','
    ;

protected INT : ('0'..'9')+;

protected WS : ' ';

protected
VARIABLE
    :   'A'..'Z'
        ('A'..'Z' | ' ' | '0'..'9')*
        { /* strip space from end */ }
    ;

// just an int or float
protected EXPR
    :   INT ( '.' (INT)? )?
    ;
</pre>

<p> The previous examples discuss differentiating lexical rules via
lots of lookahead (fixed k or arbitrary).  There are other situations
where you have to turn on and off certain lexical rules (making
certain tokens valid and invalid) depending on prior context or
semantic information.  One of the best examples is matching a token
only if it starts on the left edge of a line (i.e., column 1).
Without being able to test the state of the lexer's column counter,
you cannot do a decent job.  Here is a simple <tt>DEFINE</tt> rule
that is only matched if the semantic predicate is true.

<tt><pre>
DEFINE
    :   {getColumn()==1}? "#define" ID
    ;
</pre></tt>

<p> Semantic predicates on the <b>left-edge</b> of
<b>single-alternative</b> lexical rules get hoisted into the
<tt>nextToken</tt> prediction mechanism.  Adding the predicate to a
rule makes it so that it is not a candidate for recognition until the
predicate evaluates to true.  In this case, the method for
<tt>DEFINE</tt> would never be entered, even if the lookahead
predicted <tt>#define</tt>, if the column &gt; 1.

<p> Another useful example involves context-sensitive recognition such
as when you want to match a token only if your lexer is in a particular
context (e.g., the lexer previously matched some trigger sequence).  If
you are matching tokens that separate rows of data such as
"<tt>----</tt>", you probably only want to match this if the "begin
table" sequence has been found.

<tt><pre>
BEGIN_TABLE
    :   '[' {this.inTable=true;} // enter table context
    ;

ROW_SEP
    :   {this.inTable}? "----"
    ;

END_TABLE
    :   ']' {this.inTable=false;} // exit table context
    ;
</pre></tt>

This predicate hoisting ability is another way to simulate lexical
states from DFA-based lexer generators like <tt>lex</tt>, though
predicates are much more powerful.  (You could even turn on certain
rules according to the phase of the moon). ;)

<h3><a id="Keywords_and_literals"  name="Keywords_and_literals"name="_bb7">Keywords and literals</a></h3> 
<p>
	Many languages have a general &quot;identifier&quot; lexical rule, and keywords that are special cases of the identifier pattern. A typical identifier token is defined as: 
</p>
<pre><tt>ID : LETTER (LETTER | DIGIT)*;</tt></pre> 
<p>
	This is often in conflict with keywords. ANTLR solves this problem by letting you put fixed keywords into a literals table. The literals table (which is usally implemented as a hash table in the lexer) is checked after each token is matched, so that the literals effectively override the more general identifier pattern. Literals are created in one of two ways. First, any double-quoted string used in a parser is automatically entered into the literals table of the associated lexer. Second, literals may be specified in the lexer grammar by means of the <a href="options.html#literal">literal option</a>. In addition, the <a href="options.html#testLiterals">testLiterals option</a> gives you fine-grained control over the generation of literal-testing code. 
</p>
<h3><a id="Common_prefixes"  name="Common_prefixes"name="_bb8">Common prefixes</a></h3> 
<p>
	Fixed-length common prefixes in lexer rules are best handled by increasing the <a href="options.html#k">lookahead depth</a> of the lexer. For example, some operators from Java: 
</p>
<pre><tt>class MyLexer extends Lexer;
options {
  k=4;
}
GT : &quot;&gt;&quot;;
GE : &quot;&gt;=&quot;;
RSHIFT : &quot;&gt;&gt;&quot;;
RSHIFT_ASSIGN : &quot;&gt;&gt;=&quot;;
UNSIGNED_RSHIFT : &quot;&gt;&gt;&gt;&quot;;
UNSIGNED_RSHIFT_ASSIGN : &quot;&gt;&gt;&gt;=&quot;;</tt></pre>

 <h3><a id="Token_definition_files"  name="Token_definition_files"name="_bb9">Token definition files</a></h3> 
<p>
	Token definitions can be transferred from one grammar to another by way of token definition files. This is accomplished using the <a href="options.html#importVocab">importVocab</a> and <a href="options.html#exportVocab">exportVocab</a> options.
</p>
<h3><a id="Character_classes"  name="Character_classes"name="_bb10">Character classes</a></h3> 
<p>
	Use the <font face="Courier New">~</font> operator to invert a character or set of characters.&nbsp; For example, to match any character other than newline, the following rule references ~'\n'.
</p>
<pre>SL_COMMENT: &quot;//&quot; (~'\n')* '\n';</pre> 
<p>
	The <font face="Courier New">~</font> operator also inverts a character set:
</p>
<pre>NOT_WS: ~(' ' | '\t' | '\n' | '\r');</pre> 
<p>
	The range operator can be used to create sequential character sets:
</p>
<pre>DIGIT : '0'..'9' ;</pre>

<h3><a id="Token_Attributes"  name="Token_Attributes"name="_bb11">Token Attributes</a></h3> 
<p>
	See the next section.
</p>
<h3><a id=""  name=""name="_bb12"></a><a name="lexicallookahead">Lexical lookahead and the end-of-token symbol</a></h3> 
<p>
	A unique situation occurs when analyzing lexical grammars, one which is similar to the end-of-file condition when analyzing regular grammars.&nbsp; Consider how you would compute lookahead sets for the ('b' | ) subrule in following rule B:
</p>
<pre>class L extends Lexer;

A	:	B 'b'
	;

protected  // only called from another lex rule
B	:	'x' ('b' | )
	;</pre> 
<p>
	The lookahead for the first alternative of the subrule is clearly 'b'.&nbsp; The second alternative is empty and the lookahead set is the set of all characters that can follow references to the subrule, which is the follow set for rule B.&nbsp; In this case, the 'b' character follows the reference to B and is therefore the lookahead set for the empty alt indirectly.&nbsp; Because 'b' begins both alternatives, the parsing decision for the subrule is nondeterminism or ambiguous as we sometimes say.&nbsp; ANTLR will justly generate a warning for this subrule (unless you use the <font face="Courier New">warnWhenFollowAmbig</font> option).
</p>
<p>
	Now, consider what would make sense for the lookahead if rule A did not exist and rule B was not protected (it was a complete token rather than a &quot;subtoken&quot;):
</p>
<pre>B	:	'x' ('b' | )
	;</pre> 
<p>
	In this case, the empty alternative finds only the end of the rule as the lookahead with no other rules referencing it.&nbsp; In the worst case, <strong>any</strong> character could follow this rule (i.e., start the next token or error sequence).&nbsp; So, should not the lookahead for the empty alternative be the entire character vocabulary? &nbsp; And should not this result in a nondeterminism warning as it must conflict with the 'b' alternative?&nbsp; Conceptually, yes to both questions.&nbsp; From a practical standpoint, however, you are clearly saying &quot;heh, match a 'b' on the end of token B if you find one.&quot;&nbsp; I argue that no warning should be generated and ANTLR's policy of matching elements as soon as possible makes sense here as well.
</p>
<p>
	Another reason not to represent the lookahead as the entire vocabulary is that a vocabulary of '\u0000'..'\uFFFF' is really big (one set is 2^16 / 32 long words of memory!).&nbsp; Any alternative with '&lt;end-of-token&gt;' in its lookahead set will be pushed to the ELSE or DEFAULT clause by the code generator so that huge bitsets can be avoided.
</p>
<p>
	The summary is that lookahead purely derived from hitting the end of a lexical rule (unreferenced by other rules) cannot be the cause of a nondeterminism.&nbsp; The following table summarizes a bunch of cases that will help you figure out when ANTLR will complain and when it will not.
</p>
<table border="1" width="100%">
	<tr>
		<td valign="top">
<pre>X	:	'q' ('a')? ('a')?
        ;</pre> 
		</td>
		<td width="100">
			The first subrule is nondeterministic as 'a' from second subrule (and end-of-token) are in the lookahead for exit branch of (...)?
		</td>
	</tr>
	<tr>
		<td valign="top">
<pre>X	:	'q' ('a')? ('c')?
        ;</pre> 
		</td>
		<td width="100">
			No nondeterminism.
		</td>
	</tr>
	<tr>
		<td valign="top">
<pre>Y	:    'y' X 'b'
	;

protected
X	:    'b'
	|
	;</pre> 
		</td>
		<td width="100">
			Nondeterminism in rule X.
		</td>
	</tr>
	<tr>
		<td valign="top">
<pre>X	:	'x' ('a'|'c'|'d')+
	|	'z' ('a')+
	;</pre> 
		</td>
		<td width="100">
			No nondeterminism as exit branch of loops see lookahead computed purely from end-of-token.
		</td>
	</tr>
	<tr>
		<td valign="top">
<pre>Y	:	'y' ('a')+ ('a')?
	;</pre> 
		</td>
		<td width="100">
			Nondeterminism between 'a' of (...)+ and exit branch as the exit can see the 'a' of the optional subrule.&nbsp; This would be a problem even if ('a')? were simply 'a'.&nbsp; A (...)* loop would report the same problem.
		</td>
	</tr>
	<tr>
		<td valign="top">
<pre>X	:	'y' ('a' 'b')+ 'a' 'c'
	;</pre> 
		</td>
		<td width="100">
			At k=1, this is a nondeterminism for the (...)? since 'a' predicts staying in and exiting the loop.&nbsp; At k=2, no nondeterminism.
		</td>
	</tr>
	<tr>
		<td valign="top">
<pre>Q	:	'q' ('a' | )?
	;</pre> 
		</td>
		<td width="100">
			Here, there is an empty alternative inside an optional subrule.&nbsp; A nondeterminism is reported as two paths predict end-of-token.
		</td>
	</tr>
</table>
<p>
	You might be wondering why the first subrule below is ambiguous:
</p>
<pre>('a')? ('a')?</pre> 
<p>
	The answer is that the NFA to DFA conversion would result in a DFA with the 'a' transitions merged into a single state transition!&nbsp; This is ok for a DFA where you cannot have actions anywhere except after a complete match.&nbsp; Remember that ANTLR lets you do the following:
</p>
<pre>('a' {do-this})? ('a' {do-that})?</pre> 
<p>
	One other thing is important to know.&nbsp; Recall that alternatives in lexical rules are reordered according to their lookahead requirements, from highest to lowest.
</p>
<pre>A	:	'a'
	|	'a' 'b'
	;</pre> 
<p>
	At k=2, ANTLR can see 'a' followed by '&lt;end-of-token&gt;' for the first alternative and 'a' followed by 'b' in the second.&nbsp; The lookahead at depth 2 for the first alternative being '&lt;end-of-token&gt;' suppressing a warning that depth two can match any character for the first alternative.&nbsp; To behave naturally and to generate good code when no warning is generated, ANTLR reorders the alternatives so that the code generated is similar to:
</p>
<pre>A() {
	if ( LA(1)=='a' &amp;&amp; LA(2)=='b' ) { // alt 2
		match('a'); match('b');
	}
	else if ( LA(1)=='a' ) { // alt 1
		match('a')
	}
	else {<em>error</em>;}
}</pre> 
<p>
	Note the lack of lookahead test for depth 2 for alternative 1.&nbsp; When an empty alternative is present, ANTLR moves it to the end.&nbsp; For example,
</p>
<pre>A	:	'a'
	|
	|	'a' 'b'
	;</pre> 
<p>
	results in code like this:
</p>
<pre>A() {
	if ( LA(1)=='a' &amp;&amp; LA(2)=='b' ) { // alt 2
		match('a'); match('b');
	}
	else if ( LA(1)=='a' ) { // alt 1
		match('a')
	}
	else {
	}
}</pre> 
<p>
	Note that there is no way for a lexing error to occur here (which makes sense because the rule is optional--though this rule only makes sense when <font face="Courier New">protected</font>).
</p>
<p>
	Semantic predicates get moved along with their associated alternatives when the alternatives are sorted by lookahead depth.&nbsp; It would be weird if the addition of a {true}? predicate (which implicitly exists for each alternative) changed what the lexer recognized!&nbsp; The following rule is reorder so that alternative 2 is tested for first.
</p>
<pre>B	:	{true}? 'a'
	|	'a' 'b'
	;</pre> 
<p>
	Syntactic predicates are <strong>not</strong> reordered.&nbsp; Mentioning the predicate after the rule it conflicts with results in an ambiguity such as is in this rule:
</p>
<pre>F	:	'c'
	|	('c')=&gt; 'c'
	;</pre> 
<p>
	Other alternatives are, however, reordered with respect to the syntactic predicates even when a switch is generated for the LL(1) components and the syntactic predicates are pushed the default case.&nbsp; The following rule illustrates the point.
</p>
<pre>F	:	'b'
	|	{/* empty-path */}
	|	('c')=&gt; 'c'
	|	'c'
	|	'd'
	|	'e'
	;</pre> 
<p>
	Rule F's decision is generated as follows:
</p>
<pre>        switch ( la_1) {
        case 'b':
        {
            match('b');
            break;
        }
        case 'd':
        {
            match('d');
            break;
        }
        case 'e':
        {
            match('e');
            break;
        }
        default:
            boolean synPredMatched15 = false;
            if (((la_1=='c'))) {
                int _m15 = mark();
                synPredMatched15 = true;
                guessing++;
                try {
                    match('c');
                }
                catch (RecognitionException pe) {
                    synPredMatched15 = false;
                }
                rewind(_m15);
                guessing--;
            }
            if ( synPredMatched15 ) {
                match('c');
            }
            else if ((la_1=='c')) {
                match('c');
            }
            else {
                if ( guessing==0 ) {
                    /* empty-path */
                }
            }
        }</pre> 
<p>
	Notice how the empty path got moved after the test for the 'c' alternative.
</p>
<h3><a id="Scanning_Binary_Files"  name="Scanning_Binary_Files"name="Scanning Binary Files">Scanning Binary Files</a></h3> 
<p>
	Character literals are not limited to printable ASCII characters.&nbsp; To demonstrate the concept, imagine that you want to parse a binary file that contains strings and short integers.&nbsp; To distinguish between them, marker bytes are used according to the following format:
</p>
<table border="1" width="100%">
	<tr>
		<th width="50%">
			format
		</th>
		<th width="50%" align="center">
			description
		</th>
	</tr>
	<tr>
		<td width="50%">
			'\0' <em>highbyte lowbyte</em>
		</td>
		<td width="50%" align="center">
			Short integer
		</td>
	</tr>
	<tr>
		<td width="50%">
			'\1' <em>string of non-'\2' chars</em> '\2'
		</td>
		<td width="50%" align="center">
			String
		</td>
	</tr>
</table>
<p>
	Sample input (274 followed by &quot;a test&quot;) might look like the following in hex (output from UNIX <strong>od</strong> <strong>-h</strong> command):
</p>
<pre>0000000000    00 01 12 01 61 20 74 65 73 74 02 </pre> 
<p>
	or as viewed as characters:
</p>
<pre>0000000000    \0 001 022 001 a      t  e  s  t 002</pre> 
<p>
	The parser is trivially just a (...)+ around the two types of input tokens:
</p>
<pre>class DataParser extends Parser;

file:   (   sh:SHORT
            {System.out.println(sh.getText());}
        |   st:STRING
            {System.out.println(&quot;\&quot;&quot;+
               st.getText()+&quot;\&quot;&quot;);}
        )+
    ;</pre> 
<p>
	All of the interesting stuff happens in the lexer.&nbsp; First, define the class and set the vocabulary to be all 8 bit binary values:
</p>
<pre>class DataLexer extends Lexer;
options {
    charVocabulary = '\u0000'..'\u00FF';
}</pre> 
<p>
	Then, define the two tokens according to the specifications, with markers around the string and a single marker byte in front of the short:
</p>
<pre>SHORT
    :   // match the marker followed by any 2 bytes
        '\0' high:. lo:.
        {
        // pack the bytes into a two-byte short
        int v = (((int)high)&lt;&lt;8) + lo;
        // make a string out of the value
        $setText(&quot;&quot;+v);
        }
    ;

STRING
    :   '\1'!   // begin string (discard)
        ( ~'\2' )*
        '\2'!   // end string (discard)
    ;</pre> 
<p>
	To invoke the parser, use something like the following:
</p>
<pre>import java.io.*;

class Main {
    public static void main(String[] args) {
        try {
            // use DataInputStream to grab bytes
            DataLexer lexer =
              new DataLexer(
                new DataInputStream(System.in)
              );
            DataParser parser =
                new DataParser(lexer);
            parser.file();
        } catch(Exception e) {
            System.err.println(&quot;exception: &quot;+e);
        }
    }
}</pre>

<h3><a name="unicode"></a>Scanning Unicode Characters</h3> 
<p>
	ANTLR (as of 2.7.1) allows you to recognize input composed of Unicode characters; that is, you are not restricted to 8 bit ASCII characters.&nbsp; I would like to emphasize that ANTLR <em>allows</em>, but does yet not <em>support</em> Unicode as there is more work to be done.&nbsp; For example, end-of-file is currently incorrectly specified:
</p>
<pre>CharScanner.EOF_CHAR=(char)-1;</pre> 
<p>
	This must be an integer -1 not char, which is actually
	narrowed to 0xFFFF via the cast. &nbsp; I have to go throught
	the entire code base looking for these problems.&nbsp; Plus,
	we should really have a special syntax to mean &quot;java
	identifier character&quot; and some standard encodings for
	non-Western character sets etc...  I expect 2.7.3 to add nice
	predefined character blocks like <tt>LETTER</tt>.
</p>
<p>
The following is a very simple example of how to match a series of space-separated identifiers.
</p>
<pre>class L extends Lexer;

options {
    // Allow any char but \uFFFF (16 bit -1)
    charVocabulary='\u0000'..'\uFFFE';
}

{
    private static boolean done = false;

    public void uponEOF()
        throws TokenStreamException, CharStreamException
    {
        done=true;
    }
    
    public static void main(String[] args) throws Exception {
        L lexer = new L(System.in);
        while ( !done ) {
            Token t = lexer.nextToken();
            System.out.println(&quot;Token: &quot;+t);
        }
    }
}

ID    :    ID_START_LETTER ( ID_LETTER )*
    ;

WS    :    (' '|'\n') {$setType(Token.SKIP);}
    ;

protected
ID_START_LETTER
    :    '$'
    |    '_'
    |    'a'..'z'
    |    '\u0080'..'\ufffe'
    ;

protected
ID_LETTER
    :    ID_START_LETTER
    |    '0'..'9'
    ;</pre> 
<p>
	A final note on Unicode.&nbsp; The ~<em>x</em> &quot;not&quot; operator includes everything in your specified vocabulary (up to 16 bit character space) except <em>x</em>. &nbsp; For example,
</p>
<pre>~('$'|'a'..'z')</pre> 
<p>
	results in every unicode character except '$' and lowercase latin-1 letters, assuming your charVocabulary is 0..FFFF.
</p>
<h3><a id="Manipulating_Token_Text_and_Objects"  name="Manipulating_Token_Text_and_Objects"name="Manipulating Token Text and Objects">Manipulating Token Text and Objects</a></h3> 
<p>
	Once you have specified what to match in a lexical rule, you may ask &quot;what can I discover about what will be matched for each rule element?&quot;&nbsp; ANTLR allows you to label the various elements and, at parse-time, access the text matched for the element. &nbsp; You can even specify the token object to return from the rule and, hence, from the lexer to the parser.&nbsp; This section describes the text and token object handling characteristics of ANTLR.
</p>
<h4><a id="Manipulating_the_Text_of_a_Lexical_Rule"  name="Manipulating_the_Text_of_a_Lexical_Rule"name="_bb14">Manipulating the Text of a Lexical Rule</a></h4> 
<p>
	There are times when you want to look at the text matched for the current rule, alter it, or set the text of a rule to a new string.&nbsp; The most common case is when you want to simply discard the text associated with a few of the elements that are matched for a rule such as quotes.
</p>
<p>
	ANTLR provides the '!' operator that lets you indicate certain elements should not contribute to the text for a token being recognized. The '!' operator is used just like when building trees in the parser. For example, if you are matching the HTML tags and you do not want the '&lt;' and '&gt;' characters returned as part of the token text, you could manually remove them from the token's text before they are returned, but a better way is to suffix the unwanted characters with '!'. For example, the &lt;br&gt; tag might be recognized as follows:
</p>
<pre>BR  :  '&lt;'! &quot;br&quot; '&gt;'! ;	// discard &lt; and &gt;</pre> 
<p>
	Suffixing a lexical rule reference with '!' forces the text matched by the invoked rule to be discarded (it will not appear in the text for the invoking rule).&nbsp; For example, if you do not care about the mantissa of a floating point number, you can suffix the rule that matches it with a '!':
</p>
<pre>FLOAT : INT ('.'! INT!)? ; // keep only first INT</pre> 
<p>
	As a shorthand notation, you may suffix an alternative or rule with '!' to indicate the alternative or rule should not pass any text back to the invoking rule or parser (if nonprotected):
</p>
<pre>// ! on rule: nothing is auto added to text of rule.
rule! : ... ;

// ! on alt: nothing is auto added to text for alt
rule : ... |! ...;</pre> 
<table border="1">
	<tr>
		<th width="175">
			Item suffixed with '!'
		</th>
		<th>
			Effect
		</th>
	</tr>
	<tr>
		<td width="175" align="center">
			char or string literal
		</td>
		<td align="left">
			Do not add text for this atom to current rule's text.
		</td>
	</tr>
	<tr>
		<td width="175" align="center">
			rule reference
		</td>
		<td align="left">
			Do not add text for matched while recognizing this rule to current rule's text.
		</td>
	</tr>
	<tr>
		<td width="175" align="center">
			alternative
		</td>
		<td align="left">
			Nothing that is matched by alternative is added to current rule's text; the enclosing rule contributes nothing to any invoking rule's text.&nbsp; For nonprotected rules, the text for the token returned to parser is blank.
		</td>
	</tr>
	<tr>
		<td width="175" align="center">
			rule definition
		</td>
		<td align="left">
			Nothing that is matched by <strong>any</strong> alternative is added to current rule's text; the rule contributes nothing to any invoking rule's text.&nbsp; For nonprotected rules, the text for the token returned to parser is blank.
		</td>
	</tr>
</table>
<p>
	While the '!' implies that the text is not added to the text for the current rule, you can label an element to access the text (via the token if the element is a rule reference).
</p>
<p>
	In terms of implementation, the characters are always added to the current text buffer, but are carved out when necessary (as this will be the exception rather than the rule, making the normal case efficient).
</p>
<p>
	The '!' operator is great for discarding certain characters or groups of characters, but what about the case where you want to insert characters or totally reset the text for a rule or token?&nbsp; ANTLR provides a series of special methods to do this (we prefix the methods with '$' because Java does not have a macro facility and ANTLR must recognize the special methods in your actions).&nbsp; The following table summarizes.
</p>
<table border="1">
	<tr>
		<th width="175">
			Method
		</th>
		<th>
			Description/Translation
		</th>
	</tr>
	<tr>
		<td align="center" width="175">
			<font face="Courier New">$append(x)</font>
		</td>
		<td>
			Append x to the text of the surrounding rule.&nbsp; Translation: <font face="Courier New">text.append(x)</font>
		</td>
	</tr>
	<tr>
		<td align="center" width="175">
			<font face="Courier New">$setText(x)</font>
		</td>
		<td>
			Set the text of the surrounding rule to x.&nbsp; Translation: <font face="Courier New">text.setLength(_begin); text.append(x)</font>
		</td>
	</tr>
	<tr>
		<td align="center" width="175">
			<font face="Courier New">$getText</font>
		</td>
		<td>
			Return a String of the text for the surrounding rule.&nbsp; Translation;
			<br>
			<font face="Courier New">new String(text.getBuffer(),
				<br>
				_begin,text.length()-_begin)</font>
		</td>
	</tr>
	<tr>
		<td align="center" width="175">
			<font face="Courier New">$setToken(x)</font>
		</td>
		<td>
			Set the token object that this rule is to return.&nbsp; See the section on <a href="#Token Object Creation">Token Object Creation</a>. Translation: <font face="Courier New">_token = x</font>
		</td>
	</tr>
	<tr>
		<td align="center" width="175">
			<font face="Courier New">$setType(x)</font>
		</td>
		<td>
			Set the token type of the surrounding rule.&nbsp; Translation: <font face="Courier New">_ttype = x</font>
		</td>
	</tr>
	<tr>
		<td align="center" width="175">
			<font face="Courier New">setText(x)</font>
		</td>
		<td>
			<font face="Times New Roman">Set the text for the entire token being recognized </font>regardless of what rule the action is in<font face="Times New Roman">. No translation.</font>
		</td>
	</tr>
	<tr>
		<td align="center" width="175">
			<font face="Courier New">getText()</font>
		</td>
		<td>
			<font face="Times New Roman">Get the text for the entire token being recognized </font>regardless of what rule the action is <font face="Times New Roman">in. No translation.</font>
		</td>
	</tr>
</table>
<p>
	One of the great things about an ANTLR generated lexer is that the text of a token can be modified incrementally as the token is recognized (an impossible task for a DFA-based lexer):
</p>
<pre>STRING: '&quot;' ( ESCAPE | ~('&quot;'|'\\') )* '&quot;' ;

protected
ESCAPE
    :    '\\'
         ( 'n' { $setText(&quot;\n&quot;); }
         | 'r' { $setText(&quot;\r&quot;); }
         | 't' { $setText(&quot;\t&quot;); }
         | '&quot;' { $setText(&quot;\&quot;&quot;); }
         )
    ;</pre> <h4><a id=""  name=""name="_bb15"></a><a name="Token Object Creation">Token Object Creation</a></h4> 
<p>
	Because lexical rules can call other rules just like in the parser, you sometimes want to know what text was matched for that portion of the token being matched. To support this, ANTLR allows you to label lexical rules and obtain a <font face="Courier New">Token</font> object representing the text, token type, line number, etc... matched for that rule reference.&nbsp;&nbsp; This ability corresponds to be able to access the text matched for a lexical state in a DFA-based lexer.&nbsp; For example, here is a simple rule that prints out the text matched for a rule reference, INT.
</p>
<pre>INDEX	:	'[' i:INT ']'
		{System.out.println(i.getText());}
	;</pre> <pre>INT	:	('0'..'9')+ ;</pre> 
<p>
	If you moved the labeled reference and action to a parser, it would the same thing (match an integer and print it out).
</p>
<p>
	All lexical rules <em>conceptually</em> return a <font face="Courier New">Token</font> object, but in practice this would be inefficient. ANTLR generates methods so that a token object is created only if any invoking reference is labeled (indicating they want the token object).&nbsp; Imagine another rule that calls INT without a label.
</p>
<pre>FLOAT	:	INT ('.' INT)? ;</pre>
<p>
</font>In this case, no token object is created for either reference to INT<font size="2" face="Courier New">.</font><font face="Times New Roman" size="3">&nbsp; You will notice a boolean argument to every lexical rule that tells it whether or not a token object should be created and returned (via a member variable).&nbsp; </font>All nonprotected rules (those that are &quot;exposed&quot; to the parser) must always generate tokens, which are passed back to the parser.
</p>
<h4><a id="Heterogeneous_Token_Object_Streams"  name="Heterogeneous_Token_Object_Streams"name="_bb16">Heterogeneous Token Object Streams</a></h4> 
<p>
	While token creation is normally handled automatically, you can also manually specify the token object to be returned from a lexical rule. The advantage is that you can pass heterogeneous token objects back to the parser, which is extremely useful for parsing languagues with complicated tokens such as HTML (the <font face="Courier New">&lt;img&gt;</font> and <font face="Courier New">&lt;table&gt;</font> tokens, for example, can have lots of attributes).&nbsp; Here is a rule for the &lt;img&gt; tag that returns a token object of type ImageToken:
</p>
<pre>IMAGE
{
  Attributes attrs;
}
  :  &quot;&lt;img &quot; attrs=ATTRIBUTES '&gt;'
     {
     ImageToken t = new ImageToken(IMAGE,$getText);
     t.setAttributes(attrs);
     $setToken(t);
     }
  ;
ATTRIBUTES returns [Attributes a]
  :  ...
  ;</pre> 
<p>
	The <font face="Courier New">$setToken</font> function specifies that its argument is to be returned when the rule exits.&nbsp; The parser will receive this specific object instead of a <font face="Courier New">CommonToken</font> or whatever else you may have specified with the <font face="Courier New">Lexer.setTokenObjectClass</font> method. &nbsp; The action in rule <font face="Courier New">IMAGE</font> references a token type, <font face="Courier New">IMAGE</font>, and a lexical rule references, <font face="Courier New">ATTRIBUTES</font>, which matches all of the attributes of an image tag and returns them in a data structure called <font face="Courier New">Attributes</font>.
</p>
<p>
	What would it mean for rule <font face="Courier New">IMAGE</font> to be protected (i.e., referenced only from other lexical rules rather than from <font face="Courier New">nextToken</font>)? &nbsp; Any invoking labeled rule reference would receive the object (not the parser) and could examine it, or manipulate it, or pass it on to the invoker of that rule.&nbsp; For example, if <font face="Courier New">IMAGE</font> were called from <font face="Courier New">TAGS</font> rather than being nonprotected, rule <font face="Courier New">TAGS</font> would have to pass the token object back to the parser for it.
</p>
<pre>TAGS : IMG:IMAGE
       {$setToken(img);} // pass to parser
     | PARAGRAPH // probably has no special token
     | ...
     ;</pre> 
<p>
	Setting the token object for a nonprotected rule invoked without a label has no effect other than to waste time creating an object that will not be used.
</p>
<p>
	We use a <tt>CharScanner</tt> member <tt>_returnToken</tt> to do the return in order to not conflict with return values used by the grammar developer. For example, 
</p>
<pre>PTAG: &quot;&lt;p&gt;&quot; {$setToken(new ParagraphToken($$));} ; </pre> 
<p>
	which would be translated to something like: 
</p>
<pre>protected final void mPTAG()
  throws RecognitionException, CharStreamException,
         TokenStreamException {
    Token _token = null;
    match(&quot;&lt;p&gt;&quot;);
    _returnToken =
      new ParagraphToken(<em>text-of-current-rule</em>);
}</pre> <h3><a id=""  name=""name="_bb17"></a><a name="Filtering Input Streams">Filtering Input Streams</a></h3> 
<p>
	You often want to perform an action upon seeing a pattern or two in a complicated input stream, such as pulling out links in an HTML file.&nbsp; One solution is to take the HTML grammar and just put actions where you want.&nbsp; Using a complete grammar is overkill and you may not have a complete grammar to start with.
</p>
<p>
	ANTLR provides a mechanism similar to AWK that lets you say &quot;here are the patterns I'm interested in--ignore everything else.&quot;&nbsp; Naturally, AWK is limited to regular expressions whereas ANTLR accepts context-free grammars (Uber-AWK?).&nbsp; For example, consider pulling out the &lt;p&gt; and &lt;br&gt; tags from an arbitrary HTML file.&nbsp; Using the filter option, this is easy:
</p>
<pre>class T extends Lexer;
options {
    k=2;
    filter=true;
}

P : &quot;&lt;p&gt;&quot; ;
BR: &quot;&lt;br&gt;&quot; ;</pre> 
<p>
	In this &quot;mode&quot;, there is no possibility of a syntax error.&nbsp; Either the pattern is matched exactly or it is filtered out.
</p>
<p>
	This works very well for many cases, but is not sophisticated enough to handle the situation where you want &quot;almost matches&quot; to be reported as errors. &nbsp; Consider the addition of the &lt;table...&gt; tag to the previous grammar:
</p>
<pre>class T extends Lexer; 
options { 
    k=2; 
    filter = true; 
} 

P : &quot;&lt;p&gt;&quot; ; 
BR: &quot;&lt;br&gt;&quot; ; 
TABLE : &quot;&lt;table&quot; (WS)? (ATTRIBUTE)* (WS)? '&gt;' ; 
WS : ' ' | '\t' | '\n' ; 
ATTRIBUTE : ... ;</pre> 
<p>
	Now, consider input &quot;&lt;table 8 = width ;&gt;&quot; (a bogus table definition). As is, the lexer would simply scarf past this input without &quot;noticing&quot; the invalid table. What if you want to indicate that a bad table definition was found as opposed to ignoring it?&nbsp; Call method
</p>
<pre>setCommitToPath(boolean commit)</pre> 
<p>
	in your TABLE rule to indicate that you want the lexer to commit to recognizing the table tag:
</p>
<pre>TABLE
    :   &quot;&lt;table&quot; (WS)?
        {setCommitToPath(true);}
        (ATTRIBUTE)* (WS)? '&gt;'
    ;</pre> 
<p>
	Input &quot;&lt;table 8 = width ;&gt;&quot; would result in a syntax error.&nbsp; Note the placement after the whitespace recognition; you do not want &lt;tabletop&gt; reported as a bad table (you want to ignore it).
</p>
<p>
	One further complication in filtering: What if the &quot;skip language&quot; (the stuff in between valid tokens or tokens of interest) cannot be correctly handled by simply consuming a character and trying again for a valid token?&nbsp; You may want to ignore comments or strings or whatever.&nbsp; In that case, you can specify a rule that scarfs anything between tokens of interest by using option <font face="Courier New">filter=<em>RULE</em></font>. &nbsp; For example, the grammar below filters for &lt;p&gt; and &lt;br&gt; tags as before, but also prints out any other tag (&lt;...&gt;) encountered.
</p>
<pre>class T extends Lexer;
options {
    k=2;
    filter=IGNORE;
    charVocabulary = '\3'..'\177';
}

P : &quot;&lt;p&gt;&quot; ;
BR: &quot;&lt;br&gt;&quot; ;

protected
IGNORE
    :   '&lt;' (~'&gt;')* '&gt;'
        {System.out.println(&quot;bad tag:&quot;+$getText);}
    |   ( &quot;\r\n&quot; | '\r' | '\n' ) {newline();}
    |   .
    ;</pre> 
<p>
	Notice that the filter rule must track newlines in the general case where the lexer might emit error messages so that the line number is not stuck at 0.
</p>
<p>
	The filter rule is invoked either when the lookahead (in nextToken) predicts none of the nonprotected lexical rules or when one of those rules fails.&nbsp; In the latter case, the input is rolled back before attempting the filter rule.&nbsp; Option <font face="Courier New">filter=true</font> is like having a filter rule such as:
</p>
<pre>IGNORE : . ;</pre> 
<p>
	Actions in regular lexical rules are executed even if the rule fails and the filter rule is called.&nbsp; To do otherwise would require every valid token to be matched twice (once to match and once to do the actions like a syntactic predicate)! Plus, there are few actions in lexer rules (usually they are at the end at which point an error cannot occur).
</p>
<p>
	Is the filter rule called when commit-to-path is true and an error is found in a lexer rule? No, an error is reported as with filter=true.
</p>
<p>
	What happens if there is a syntax error in the filter rule?&nbsp; Well, you can either put an exception handler on the filter rule or accept the default behavior, which is to consume a character and begin looking for another valid token.
</p>
<p>
	In summary, the filter option allows you to:
<ol>
	<li>
		Filter like awk (only perfect matches reported--no such thing as syntax error) 
	</li>
	<li>
		Filter like awk + catch poorly-formed matches (that is, &quot;almost matches&quot; like &lt;table 8=3;&gt; result in an error) 
	</li>
	<li>
		Filter but specify the skip language 
	</li>
</ol>
<h4><a id="ANTLR_Masquerading_as_SED"  name="ANTLR_Masquerading_as_SED"name="ANTLR Masquerading as SED">ANTLR Masquerading as SED</a></h4> 
<p>
	To make ANTLR generate lexers that behave like the UNIX utility sed (copy standard in to standard out except as specified by the replace patterns), use a filter rule that does the input to output copying:
</p>
<font size="2"><pre></font><font size="3"><font face="Courier New">class T extends Lexer;
options {
  k=2;
  filter=IGNORE;
  charVocabulary = '\3'..'\177';
}</font></pre> <pre><font face="Courier New">P  : &quot;&lt;p&gt;&quot; {System.out.print(&quot;&lt;P&gt;&quot;);};
BR : &quot;&lt;br&gt;&quot; {System.out.print(&quot;&lt;BR&gt;&quot;);};</font></pre> <pre><font face="Courier New">protected
IGNORE
  :  ( &quot;\r\n&quot; | '\r' | '\n' )
     {newline(); System.out.println(&quot;&quot;);}
  |  c:. {System.out.print(c);}
  ;</font></pre> </font>
<p>
	This example dumps anything other than &lt;p&gt; and &lt;br&gt; tags to standard out and pushes lowercase &lt;p&gt; and &lt;br&gt; to uppercase. Works great.
</p>
<h4><a id="Nongreedy_Subrules"  name="Nongreedy_Subrules"name="Nongreedy Subrules">Nongreedy Subrules</a></h4> 
<p>
	Quick:&nbsp; What does the following match?
</p>
<pre>BLOCK : '{' (.)* '}';</pre> 
<p>
	Your first reaction is that it matches any set of characters inside of curly quotes. &nbsp; In reality, it matches '{' followed by every single character left on the input stream!&nbsp; Why?&nbsp; Well, because ANTLR loops are <em>greedy</em>--they consume as much input as they can match.&nbsp; Since the wildcard matches any character, it consumes the '}' and beyond.&nbsp; This is a pain for matching strings, comments and so on.
</p>
<p>
	Why can't we switch it around so that it consumes only until it sees something on the input stream that matches what <strong>follows</strong> the loop, such as the '}'? &nbsp; That is, why can't we make loops <em>nongreedy</em>?&nbsp; The answer is we can, but sometimes you want greedy and sometimes you want nongreedy (PERL has both kinds of closure loops now too).&nbsp; Unfortunately, parsers usually want greedy and lexers usually want nongreedy loops.&nbsp; Rather than make the same syntax behave differently in the various situations, Terence decided to leave the semantics of loops as they are (greedy) and make a subrule option to make loops nongreedy.
</p>
<h4><a id="Greedy_Subrules"  name="Greedy_Subrules"name="Greedy Parser Subrules">Greedy Subrules</a></h4> 
<p>
	I have yet to see a case when building a parser grammar where I did not want a subrule to match as much input as possible.&nbsp; For example, the solution to the classic if-then-else clause ambiguity is to match the &quot;else&quot; as soon as possible:
</p>
<pre>stat : &quot;if&quot; expr &quot;then&quot; stat (&quot;else&quot; stat)?
     | ...
     ;</pre> 
<p>
	This ambiguity (which statement should the &quot;else&quot; be attached to) results in a parser nondeterminism.&nbsp; ANTLR warns you about the <font face="Courier New">(...)?</font> subrule as follows:
</p>
<pre>warning: line 3: nondeterminism upon
        k==1:&quot;else&quot;
        between alts 1 and 2 of block</pre> 
<p>
	If, on the other hand, you make it clea…
Large files files are truncated, but you can click here to view the full file