/vendor/pcre/HACKING

http://github.com/feyeleanor/RubyGoLightly · #! · 418 lines · 329 code · 89 blank · 0 comment · 0 complexity · f0dcafb5fe72ababd7090b722704d2d2 MD5 · raw file

  1. Technical Notes about PCRE
  2. --------------------------
  3. These are very rough technical notes that record potentially useful information
  4. about PCRE internals.
  5. Historical note 1
  6. -----------------
  7. Many years ago I implemented some regular expression functions to an algorithm
  8. suggested by Martin Richards. These were not Unix-like in form, and were quite
  9. restricted in what they could do by comparison with Perl. The interesting part
  10. about the algorithm was that the amount of space required to hold the compiled
  11. form of an expression was known in advance. The code to apply an expression did
  12. not operate by backtracking, as the original Henry Spencer code and current
  13. Perl code does, but instead checked all possibilities simultaneously by keeping
  14. a list of current states and checking all of them as it advanced through the
  15. subject string. In the terminology of Jeffrey Friedl's book, it was a "DFA
  16. algorithm", though it was not a traditional Finite State Machine (FSM). When
  17. the pattern was all used up, all remaining states were possible matches, and
  18. the one matching the longest subset of the subject string was chosen. This did
  19. not necessarily maximize the individual wild portions of the pattern, as is
  20. expected in Unix and Perl-style regular expressions.
  21. Historical note 2
  22. -----------------
  23. By contrast, the code originally written by Henry Spencer (which was
  24. subsequently heavily modified for Perl) compiles the expression twice: once in
  25. a dummy mode in order to find out how much store will be needed, and then for
  26. real. (The Perl version probably doesn't do this any more; I'm talking about
  27. the original library.) The execution function operates by backtracking and
  28. maximizing (or, optionally, minimizing in Perl) the amount of the subject that
  29. matches individual wild portions of the pattern. This is an "NFA algorithm" in
  30. Friedl's terminology.
  31. OK, here's the real stuff
  32. -------------------------
  33. For the set of functions that form the "basic" PCRE library (which are
  34. unrelated to those mentioned above), I tried at first to invent an algorithm
  35. that used an amount of store bounded by a multiple of the number of characters
  36. in the pattern, to save on compiling time. However, because of the greater
  37. complexity in Perl regular expressions, I couldn't do this. In any case, a
  38. first pass through the pattern is helpful for other reasons.
  39. Computing the memory requirement: how it was
  40. --------------------------------------------
  41. Up to and including release 6.7, PCRE worked by running a very degenerate first
  42. pass to calculate a maximum store size, and then a second pass to do the real
  43. compile - which might use a bit less than the predicted amount of memory. The
  44. idea was that this would turn out faster than the Henry Spencer code because
  45. the first pass is degenerate and the second pass can just store stuff straight
  46. into the vector, which it knows is big enough.
  47. Computing the memory requirement: how it is
  48. -------------------------------------------
  49. By the time I was working on a potential 6.8 release, the degenerate first pass
  50. had become very complicated and hard to maintain. Indeed one of the early
  51. things I did for 6.8 was to fix Yet Another Bug in the memory computation. Then
  52. I had a flash of inspiration as to how I could run the real compile function in
  53. a "fake" mode that enables it to compute how much memory it would need, while
  54. actually only ever using a few hundred bytes of working memory, and without too
  55. many tests of the mode that might slow it down. So I re-factored the compiling
  56. functions to work this way. This got rid of about 600 lines of source. It
  57. should make future maintenance and development easier. As this was such a major
  58. change, I never released 6.8, instead upping the number to 7.0 (other quite
  59. major changes are also present in the 7.0 release).
  60. A side effect of this work is that the previous limit of 200 on the nesting
  61. depth of parentheses was removed. However, there is a downside: pcre_compile()
  62. runs more slowly than before (30% or more, depending on the pattern) because it
  63. is doing a full analysis of the pattern. My hope is that this is not a big
  64. issue.
  65. Traditional matching function
  66. -----------------------------
  67. The "traditional", and original, matching function is called pcre_exec(), and
  68. it implements an NFA algorithm, similar to the original Henry Spencer algorithm
  69. and the way that Perl works. Not surprising, since it is intended to be as
  70. compatible with Perl as possible. This is the function most users of PCRE will
  71. use most of the time.
  72. Supplementary matching function
  73. -------------------------------
  74. From PCRE 6.0, there is also a supplementary matching function called
  75. pcre_dfa_exec(). This implements a DFA matching algorithm that searches
  76. simultaneously for all possible matches that start at one point in the subject
  77. string. (Going back to my roots: see Historical Note 1 above.) This function
  78. intreprets the same compiled pattern data as pcre_exec(); however, not all the
  79. facilities are available, and those that are do not always work in quite the
  80. same way. See the user documentation for details.
  81. The algorithm that is used for pcre_dfa_exec() is not a traditional FSM,
  82. because it may have a number of states active at one time. More work would be
  83. needed at compile time to produce a traditional FSM where only one state is
  84. ever active at once. I believe some other regex matchers work this way.
  85. Format of compiled patterns
  86. ---------------------------
  87. The compiled form of a pattern is a vector of bytes, containing items of
  88. variable length. The first byte in an item is an opcode, and the length of the
  89. item is either implicit in the opcode or contained in the data bytes that
  90. follow it.
  91. In many cases below LINK_SIZE data values are specified for offsets within the
  92. compiled pattern. The default value for LINK_SIZE is 2, but PCRE can be
  93. compiled to use 3-byte or 4-byte values for these offsets (impairing the
  94. performance). This is necessary only when patterns whose compiled length is
  95. greater than 64K are going to be processed. In this description, we assume the
  96. "normal" compilation options. Data values that are counts (e.g. for
  97. quantifiers) are always just two bytes long.
  98. A list of the opcodes follows:
  99. Opcodes with no following data
  100. ------------------------------
  101. These items are all just one byte long
  102. OP_END end of pattern
  103. OP_ANY match any one character other than newline
  104. OP_ALLANY match any one character, including newline
  105. OP_ANYBYTE match any single byte, even in UTF-8 mode
  106. OP_SOD match start of data: \A
  107. OP_SOM, start of match (subject + offset): \G
  108. OP_SET_SOM, set start of match (\K)
  109. OP_CIRC ^ (start of data, or after \n in multiline)
  110. OP_NOT_WORD_BOUNDARY \W
  111. OP_WORD_BOUNDARY \w
  112. OP_NOT_DIGIT \D
  113. OP_DIGIT \d
  114. OP_NOT_HSPACE \H
  115. OP_HSPACE \h
  116. OP_NOT_WHITESPACE \S
  117. OP_WHITESPACE \s
  118. OP_NOT_VSPACE \V
  119. OP_VSPACE \v
  120. OP_NOT_WORDCHAR \W
  121. OP_WORDCHAR \w
  122. OP_EODN match end of data or \n at end: \Z
  123. OP_EOD match end of data: \z
  124. OP_DOLL $ (end of data, or before \n in multiline)
  125. OP_EXTUNI match an extended Unicode character
  126. OP_ANYNL match any Unicode newline sequence
  127. OP_ACCEPT )
  128. OP_COMMIT )
  129. OP_FAIL ) These are Perl 5.10's "backtracking
  130. OP_PRUNE ) control verbs".
  131. OP_SKIP )
  132. OP_THEN )
  133. Repeating single characters
  134. ---------------------------
  135. The common repeats (*, +, ?) when applied to a single character use the
  136. following opcodes:
  137. OP_STAR
  138. OP_MINSTAR
  139. OP_POSSTAR
  140. OP_PLUS
  141. OP_MINPLUS
  142. OP_POSPLUS
  143. OP_QUERY
  144. OP_MINQUERY
  145. OP_POSQUERY
  146. In ASCII mode, these are two-byte items; in UTF-8 mode, the length is variable.
  147. Those with "MIN" in their name are the minimizing versions. Those with "POS" in
  148. their names are possessive versions. Each is followed by the character that is
  149. to be repeated. Other repeats make use of
  150. OP_UPTO
  151. OP_MINUPTO
  152. OP_POSUPTO
  153. OP_EXACT
  154. which are followed by a two-byte count (most significant first) and the
  155. repeated character. OP_UPTO matches from 0 to the given number. A repeat with a
  156. non-zero minimum and a fixed maximum is coded as an OP_EXACT followed by an
  157. OP_UPTO (or OP_MINUPTO or OPT_POSUPTO).
  158. Repeating character types
  159. -------------------------
  160. Repeats of things like \d are done exactly as for single characters, except
  161. that instead of a character, the opcode for the type is stored in the data
  162. byte. The opcodes are:
  163. OP_TYPESTAR
  164. OP_TYPEMINSTAR
  165. OP_TYPEPOSSTAR
  166. OP_TYPEPLUS
  167. OP_TYPEMINPLUS
  168. OP_TYPEPOSPLUS
  169. OP_TYPEQUERY
  170. OP_TYPEMINQUERY
  171. OP_TYPEPOSQUERY
  172. OP_TYPEUPTO
  173. OP_TYPEMINUPTO
  174. OP_TYPEPOSUPTO
  175. OP_TYPEEXACT
  176. Match by Unicode property
  177. -------------------------
  178. OP_PROP and OP_NOTPROP are used for positive and negative matches of a
  179. character by testing its Unicode property (the \p and \P escape sequences).
  180. Each is followed by two bytes that encode the desired property as a type and a
  181. value.
  182. Repeats of these items use the OP_TYPESTAR etc. set of opcodes, followed by
  183. three bytes: OP_PROP or OP_NOTPROP and then the desired property type and
  184. value.
  185. Matching literal characters
  186. ---------------------------
  187. The OP_CHAR opcode is followed by a single character that is to be matched
  188. casefully. For caseless matching, OP_CHARNC is used. In UTF-8 mode, the
  189. character may be more than one byte long. (Earlier versions of PCRE used
  190. multi-character strings, but this was changed to allow some new features to be
  191. added.)
  192. Character classes
  193. -----------------
  194. If there is only one character, OP_CHAR or OP_CHARNC is used for a positive
  195. class, and OP_NOT for a negative one (that is, for something like [^a]).
  196. However, in UTF-8 mode, the use of OP_NOT applies only to characters with
  197. values < 128, because OP_NOT is confined to single bytes.
  198. Another set of repeating opcodes (OP_NOTSTAR etc.) are used for a repeated,
  199. negated, single-character class. The normal ones (OP_STAR etc.) are used for a
  200. repeated positive single-character class.
  201. When there's more than one character in a class and all the characters are less
  202. than 256, OP_CLASS is used for a positive class, and OP_NCLASS for a negative
  203. one. In either case, the opcode is followed by a 32-byte bit map containing a 1
  204. bit for every character that is acceptable. The bits are counted from the least
  205. significant end of each byte.
  206. The reason for having both OP_CLASS and OP_NCLASS is so that, in UTF-8 mode,
  207. subject characters with values greater than 256 can be handled correctly. For
  208. OP_CLASS they don't match, whereas for OP_NCLASS they do.
  209. For classes containing characters with values > 255, OP_XCLASS is used. It
  210. optionally uses a bit map (if any characters lie within it), followed by a list
  211. of pairs and single characters. There is a flag character than indicates
  212. whether it's a positive or a negative class.
  213. Back references
  214. ---------------
  215. OP_REF is followed by two bytes containing the reference number.
  216. Repeating character classes and back references
  217. -----------------------------------------------
  218. Single-character classes are handled specially (see above). This section
  219. applies to OP_CLASS and OP_REF. In both cases, the repeat information follows
  220. the base item. The matching code looks at the following opcode to see if it is
  221. one of
  222. OP_CRSTAR
  223. OP_CRMINSTAR
  224. OP_CRPLUS
  225. OP_CRMINPLUS
  226. OP_CRQUERY
  227. OP_CRMINQUERY
  228. OP_CRRANGE
  229. OP_CRMINRANGE
  230. All but the last two are just single-byte items. The others are followed by
  231. four bytes of data, comprising the minimum and maximum repeat counts. There are
  232. no special possessive opcodes for these repeats; a possessive repeat is
  233. compiled into an atomic group.
  234. Brackets and alternation
  235. ------------------------
  236. A pair of non-capturing (round) brackets is wrapped round each expression at
  237. compile time, so alternation always happens in the context of brackets.
  238. [Note for North Americans: "bracket" to some English speakers, including
  239. myself, can be round, square, curly, or pointy. Hence this usage.]
  240. Non-capturing brackets use the opcode OP_BRA. Originally PCRE was limited to 99
  241. capturing brackets and it used a different opcode for each one. From release
  242. 3.5, the limit was removed by putting the bracket number into the data for
  243. higher-numbered brackets. From release 7.0 all capturing brackets are handled
  244. this way, using the single opcode OP_CBRA.
  245. A bracket opcode is followed by LINK_SIZE bytes which give the offset to the
  246. next alternative OP_ALT or, if there aren't any branches, to the matching
  247. OP_KET opcode. Each OP_ALT is followed by LINK_SIZE bytes giving the offset to
  248. the next one, or to the OP_KET opcode. For capturing brackets, the bracket
  249. number immediately follows the offset, always as a 2-byte item.
  250. OP_KET is used for subpatterns that do not repeat indefinitely, while
  251. OP_KETRMIN and OP_KETRMAX are used for indefinite repetitions, minimally or
  252. maximally respectively. All three are followed by LINK_SIZE bytes giving (as a
  253. positive number) the offset back to the matching bracket opcode.
  254. If a subpattern is quantified such that it is permitted to match zero times, it
  255. is preceded by one of OP_BRAZERO, OP_BRAMINZERO, or OP_SKIPZERO. These are
  256. single-byte opcodes that tell the matcher that skipping the following
  257. subpattern entirely is a valid branch. In the case of the first two, not
  258. skipping the pattern is also valid (greedy and non-greedy). The third is used
  259. when a pattern has the quantifier {0,0}. It cannot be entirely discarded,
  260. because it may be called as a subroutine from elsewhere in the regex.
  261. A subpattern with an indefinite maximum repetition is replicated in the
  262. compiled data its minimum number of times (or once with OP_BRAZERO if the
  263. minimum is zero), with the final copy terminating with OP_KETRMIN or OP_KETRMAX
  264. as appropriate.
  265. A subpattern with a bounded maximum repetition is replicated in a nested
  266. fashion up to the maximum number of times, with OP_BRAZERO or OP_BRAMINZERO
  267. before each replication after the minimum, so that, for example, (abc){2,5} is
  268. compiled as (abc)(abc)((abc)((abc)(abc)?)?)?, except that each bracketed group
  269. has the same number.
  270. When a repeated subpattern has an unbounded upper limit, it is checked to see
  271. whether it could match an empty string. If this is the case, the opcode in the
  272. final replication is changed to OP_SBRA or OP_SCBRA. This tells the matcher
  273. that it needs to check for matching an empty string when it hits OP_KETRMIN or
  274. OP_KETRMAX, and if so, to break the loop.
  275. Assertions
  276. ----------
  277. Forward assertions are just like other subpatterns, but starting with one of
  278. the opcodes OP_ASSERT or OP_ASSERT_NOT. Backward assertions use the opcodes
  279. OP_ASSERTBACK and OP_ASSERTBACK_NOT, and the first opcode inside the assertion
  280. is OP_REVERSE, followed by a two byte count of the number of characters to move
  281. back the pointer in the subject string. When operating in UTF-8 mode, the count
  282. is a character count rather than a byte count. A separate count is present in
  283. each alternative of a lookbehind assertion, allowing them to have different
  284. fixed lengths.
  285. Once-only (atomic) subpatterns
  286. ------------------------------
  287. These are also just like other subpatterns, but they start with the opcode
  288. OP_ONCE. The check for matching an empty string in an unbounded repeat is
  289. handled entirely at runtime, so there is just this one opcode.
  290. Conditional subpatterns
  291. -----------------------
  292. These are like other subpatterns, but they start with the opcode OP_COND, or
  293. OP_SCOND for one that might match an empty string in an unbounded repeat. If
  294. the condition is a back reference, this is stored at the start of the
  295. subpattern using the opcode OP_CREF followed by two bytes containing the
  296. reference number. If the condition is "in recursion" (coded as "(?(R)"), or "in
  297. recursion of group x" (coded as "(?(Rx)"), the group number is stored at the
  298. start of the subpattern using the opcode OP_RREF, and a value of zero for "the
  299. whole pattern". For a DEFINE condition, just the single byte OP_DEF is used (it
  300. has no associated data). Otherwise, a conditional subpattern always starts with
  301. one of the assertions.
  302. Recursion
  303. ---------
  304. Recursion either matches the current regex, or some subexpression. The opcode
  305. OP_RECURSE is followed by an value which is the offset to the starting bracket
  306. from the start of the whole pattern. From release 6.5, OP_RECURSE is
  307. automatically wrapped inside OP_ONCE brackets (because otherwise some patterns
  308. broke it). OP_RECURSE is also used for "subroutine" calls, even though they
  309. are not strictly a recursion.
  310. Callout
  311. -------
  312. OP_CALLOUT is followed by one byte of data that holds a callout number in the
  313. range 0 to 254 for manual callouts, or 255 for an automatic callout. In both
  314. cases there follows a two-byte value giving the offset in the pattern to the
  315. start of the following item, and another two-byte item giving the length of the
  316. next item.
  317. Changing options
  318. ----------------
  319. If any of the /i, /m, or /s options are changed within a pattern, an OP_OPT
  320. opcode is compiled, followed by one byte containing the new settings of these
  321. flags. If there are several alternatives, there is an occurrence of OP_OPT at
  322. the start of all those following the first options change, to set appropriate
  323. options for the start of the alternative. Immediately after the end of the
  324. group there is another such item to reset the flags to their previous values. A
  325. change of flag right at the very start of the pattern can be handled entirely
  326. at compile time, and so does not cause anything to be put into the compiled
  327. data.
  328. Philip Hazel
  329. April 2008