index.html | searchcode

/contrib/lispbuilder-20151218-git/lispbuilder-lexer/index.html

https://gitlab.com/dto/ecl-android-games-src · HTML · 91 lines · 73 code · 18 blank · 0 comment · 0 complexity · ea25c344bd0a93091af3dfad57a3b5ef MD5 · raw file


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">



<html>

  <head>

	<title>LEXER Package</title>

  </head>



  <body>

          <p><em>Text copied from

          <a href="http://www.geocities.com/mparker762/clawk">http://www.geocities.com/mparker762/clawk</a></em></p>



          <h2>LEXER package</h2>



          <p>The LEXER package implements a lexical-analyzer-generator

          called <em>DEFLEXER</em>, which is built on top of both

          REGEX and CLAWK.  Many of the optimizations in the recent

          rewrite of the regex engine went into optimizing the sorts

          of patterns generated by DEFLEX.</p>



          <p>The default lexer doesn't implement full greediness.  If you have a rule for ints followed by a rule for floats, the int rule will match on the part before the decimal before the float rule gets a change to look at it.  You can fix this by specifying <code>:flex-compatible</code> as the first rule.  This gives all patterns a chance to examine the text and takes the one that matches the longest string (first pattern wins in case of a tie).  The down side of this option is that it slows down the analyser.  If you can solve the issue by reordering your rules that's the way to do it.</p>



          <p>I'm currently writing an AWK->CLAWK translator using this

          as the lexer, and it's working fine.  As far as I can tell,

          the DEFLEXER-generated lexing functions should be fast

          enough for production use.</p>



          <p>Currently, the LEX/FLEX/BISON feature of switching

          productions on and off using state variables is not

          supported, but it's a pretty simple feature to add.  If

          you're using LEXER and discover you need this feature,

          let me know.</p>



          <p>It also doesn't yet support prefix and postfix context

          patterns.  This isn't quite so trivial to add, but it's

          planned for a future release of regex, so LEXER will be

          getting it someday.</p>



          <p>Anyway, Here's a simple <em>DEFLEXER</em> example:</p>



<code><pre>

  (deflexer test-lexer

    ("[0-9]+([.][0-9]+([Ee][0-9]+)?)"

      (return (values 'flt (num %0))))

    ("[0-9]+"

      (return (values 'int (int %0))))

    ("[:alpha:][:alnum:]*"

      (return (values 'name %0)))

    ("[:space:]+") )



  > (setq *lex* (test-lexer "1.0 12 fred 10.23e45"))

  &lt;closure&gt;

 

  > (funcall *lex*)

  FLT

  1.0

 

  > (funcall *lex*)

  INT

  12

 

  > (funcall *lex*)

  NAME

  "fred"



  > (funcall *lex*)

  FLT

  1.0229999999999997E46



  > (funcall *lex*)

  NIL

  NIL

</pre></code>



<p>

You can also write this lexer using the <code>:flex-compatible</code> option, in which case you can write the <em>int</em> and <em>flt</em> rules in any order.

</p>

<code><pre>

(deflexer test-lexer

  :flex-compatible

  ("[0-9]+"

    (return (values 'int (int %0))))

  ("[0-9]+([.][0-9]+([Ee][0-9]+)?)"

    (return (values 'flt (num %0))))

  ("[:space:]+")

 )

</pre></code>



    <hr>

    <address><a href="mailto:mparker762@hotmail.com">Michael Parker</a></address>

  </body>

</html>