/fparsec/main/Doc/html/reference/charstream.html
HTML | 1839 lines | 1804 code | 29 blank | 6 comment | 0 complexity | 0eb8354014409e44363fac4b55d0b5f9 MD5 | raw file
Large files files are truncated, but you can click here to view the full file
1<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> 2<html xmlns="http://www.w3.org/1999/xhtml"> 3<head> 4 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> 5 <title>FParsec.CharStream</title> 6 <link rel="stylesheet" type="text/css" media="all" href="../css/style.css" /> 7 <link rel="stylesheet" type="text/css" media="screen" href="../css/screen-sidebar.css" title="Sidebar" /> 8 <link rel="alternate stylesheet" type="text/css" media="screen" href="../css/screen-no-sidebar.css" title="No sidebar" /> 9 <!--[if IE]> 10 <link rel="stylesheet" type="text/css" media="all" href="../css/style-ie.css" /> 11 <![endif]--> 12 <!--[if IE 6]> 13 <link rel="stylesheet" type="text/css" media="all" href="../css/style-ie6.css" /> 14 <![endif]--> 15 <link rel="stylesheet" type="text/css" media="print" href="../css/print.css" /> 16</head> 17<body> 18 <div id="fixed-layer"> 19 <div id="fixed-wrapper"> 20 <div id="sidebar"> 21 <div id="breadcrumbs-dummy"><span class="breadcrumbs"> </span></div> 22 <div id="nav-tree"> 23 <table class="nav n1"> 24 <tbody class="nav-open n1"> 25 <tr class="nav-entry n1 _1"> 26 <td class="nav-number n1"></td> 27 <td class="nav-title n1"><a href="../index.html">FParsec Documentation</a></td> 28 </tr> 29 <tr class="nav-sub-entries n1 _1"> 30 <td class="nav-sub-entries-number n1"></td> 31 <td class="nav-sub-entries n1"> 32 <table class="nav n2"> 33 <tbody class="nav-before-open n2"> 34 <tr class="nav-entry n2 _1"> 35 <td class="nav-number n2"><a href="../tutorial.html"><span class="section-number">1</span><span class="nav-space"></span></a></td> 36 <td class="nav-title n2"><a href="../tutorial.html">Tutorial</a></td> 37 </tr> 38 </tbody> 39 <tbody class="nav-open n2"> 40 <tr class="nav-entry n2 _2"> 41 <td class="nav-number n2"><a href="index.html"><span class="section-number">2</span><span class="nav-space"></span></a></td> 42 <td class="nav-title n2"><a href="index.html">Reference</a></td> 43 </tr> 44 <tr class="nav-sub-entries n2 _2"> 45 <td class="nav-sub-entries-number n2"></td> 46 <td class="nav-sub-entries n2"> 47 <table class="nav n3"> 48 <tbody class="nav-before-open n3"> 49 <tr class="nav-entry n3 _1"> 50 <td class="nav-number n3"><a href="parser-overview.html"><span class="section-number">1</span><span class="nav-space"></span></a></td> 51 <td class="nav-title n3"><a href="parser-overview.html">Parser overview</a></td> 52 </tr> 53 <tr class="nav-entry n3 _2"> 54 <td class="nav-number n3"><a href="primitives.html"><span class="section-number">2</span><span class="nav-space"></span></a></td> 55 <td class="nav-title n3"><a href="primitives.html">FParsec.Primitives</a></td> 56 </tr> 57 <tr class="nav-entry n3 _3"> 58 <td class="nav-number n3"><a href="charparsers.html"><span class="section-number">3</span><span class="nav-space"></span></a></td> 59 <td class="nav-title n3"><a href="charparsers.html">FParsec.CharParsers</a></td> 60 </tr> 61 <tr class="nav-entry n3 _4"> 62 <td class="nav-number n3"> 63 <a href="operatorprecedenceparser.html"><span class="section-number">4</span><span class="nav-space"></span></a> 64 </td> 65 <td class="nav-title n3"><a href="operatorprecedenceparser.html">FParsec.OperatorPrecedenceParser</a></td> 66 </tr> 67 <tr class="nav-entry n3 _5"> 68 <td class="nav-number n3"><a href="error.html"><span class="section-number">5</span><span class="nav-space"></span></a></td> 69 <td class="nav-title n3"><a href="error.html">FParsec.Error</a></td> 70 </tr> 71 <tr class="nav-entry n3 _6"> 72 <td class="nav-number n3"><a href="state.html"><span class="section-number">6</span><span class="nav-space"></span></a></td> 73 <td class="nav-title n3"><a href="state.html">FParsec.State</a></td> 74 </tr> 75 <tr class="nav-entry n3 _7"> 76 <td class="nav-number n3"><a href="position.html"><span class="section-number">7</span><span class="nav-space"></span></a></td> 77 <td class="nav-title n3"><a href="position.html">FParsec.Position</a></td> 78 </tr> 79 </tbody> 80 <tbody class="nav-open selected n3"> 81 <tr class="nav-entry selected n3 _8"> 82 <td class="nav-number selected n3"><a href="#"><span class="section-number">8</span><span class="nav-space"></span></a></td> 83 <td class="nav-title selected n3"><a href="#">FParsec.CharStream</a></td> 84 </tr> 85 <tr class="nav-sub-entries selected n3 _8"> 86 <td class="nav-sub-entries-number selected n3"></td> 87 <td class="nav-sub-entries selected n3"> 88 <table class="nav n4"> 89 <tbody class="nav-before-open n4"> 90 <tr class="nav-entry n4 _1"> 91 <td class="nav-number n4"><a href="#interface"><span class="section-number">1</span><span class="nav-space"></span></a></td> 92 <td class="nav-title n4"><a href="#interface">Interface</a></td> 93 </tr> 94 <tr class="nav-entry n4 _2"> 95 <td class="nav-number n4"><a href="#remarks"><span class="section-number">2</span><span class="nav-space"></span></a></td> 96 <td class="nav-title n4"><a href="#remarks">Remarks</a></td> 97 </tr> 98 <tr class="nav-entry n4 _3"> 99 <td class="nav-number n4"><a href="#exceptions"><span class="section-number">3</span><span class="nav-space"></span></a></td> 100 <td class="nav-title n4"><a href="#exceptions">I/O exceptions</a></td> 101 </tr> 102 <tr class="nav-entry n4 _4"> 103 <td class="nav-number n4"><a href="#members"><span class="section-number">4</span><span class="nav-space"></span></a></td> 104 <td class="nav-title n4"><a href="#members">Members</a></td> 105 </tr> 106 <tr class="nav-entry n4 _5"> 107 <td class="nav-number n4"><a href="#Iterator"><span class="section-number">5</span><span class="nav-space"></span></a></td> 108 <td class="nav-title n4"><a href="#Iterator">CharStream.Iterator</a></td> 109 </tr> 110 <tr class="nav-sub-entries n4 _5"> 111 <td class="nav-sub-entries-number n4"></td> 112 <td class="nav-sub-entries n4"> 113 <table class="nav n5"> 114 <tbody class="nav-before-open n5"> 115 <tr class="nav-entry n5 _1"> 116 <td class="nav-number n5"> 117 <a href="#Iterator.interface"><span class="section-number">1</span><span class="nav-space"></span></a> 118 </td> 119 <td class="nav-title n5"><a href="#Iterator.interface">Interface</a></td> 120 </tr> 121 <tr class="nav-entry n5 _2"> 122 <td class="nav-number n5"> 123 <a href="#Iterator.members"><span class="section-number">2</span><span class="nav-space"></span></a> 124 </td> 125 <td class="nav-title n5"><a href="#Iterator.members">Members</a></td> 126 </tr> 127 </tbody> 128 </table> 129 </td> 130 </tr> 131 </tbody> 132 </table> 133 </td> 134 </tr> 135 </tbody> 136 </table> 137 </td> 138 </tr> 139 </tbody> 140 </table> 141 </td> 142 </tr> 143 </tbody> 144 </table> 145 </div> 146 </div> 147 </div> 148 </div> 149 <div id="wrapper"> 150 <div id="main"> 151 <div id="main-content"> 152 <div id="breadcrumbs"> 153 <span class="breadcrumbs"> 154 <span id="breadcrumbs-parents"><a href="../index.html">FParsec Documentation</a><span class="breadcrumbs-sep"> > </span><a href="index.html">Reference</a></span><span class="breadcrumbs-sep"> > </span>FParsec.CharStream 155 </span> 156 </div> 157 <div class="section s2"> 158 <h1 class="title h2"><span class="section-number">2.8</span> FParsec.CharStream</h1> 159 <div class="intro i2"> 160 <p class="para _1"> 161 The <code class="fsharp"><span class="ci">CharStream</span></code> class provides a unified interface for reading the UTF‐16 char content from 162 binary streams or in‐memory char buffers. 163 </p> 164 </div> 165 <div id="interface" class="section s3"> 166 <h2 class="title h3"><span class="section-number">2.8.1</span> Interface</h2> 167 <div class="intro i3"> 168 <div class="para _1"> 169<pre class="code fsharp"><span class="ck">namespace</span> <span class="ci">FParsec</span> 170 171<span class="cp">[<</span><span class="ci">Sealed</span><span class="cp">>]</span> 172<span class="ck">type</span> <span class="ci">CharStream</span> <span class="cp">=</span> 173 <span class="ck">interface</span> <span class="ci">System</span><span class="cm">.</span><span class="ci">IDisposable</span> 174 175 <a id="interface.new_string:B:" href="#members.new_string"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">string</span> <span class="cp">*</span> <span class="ci">index</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> 176 <span class="cr">-></span> <span class="ci">CharStream</span> 177 <a id="interface.new_string_offset:B:" href="#members.new_string_offset"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">string</span> <span class="cp">*</span> <span class="ci">index</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">streamBeginIndex</span><span class="cp">:</span> <span class="ci">int64</span> 178 <span class="cr">-></span> <span class="ci">CharStream</span> 179 180 <a id="interface.new_char-array:B:" href="#members.new_char-array"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">char</span><span class="cp">[]</span> <span class="cp">*</span> <span class="ci">index</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> 181 <span class="cr">-></span> <span class="ci">CharStream</span> 182 <a id="interface.new_char-array_offset:B:" href="#members.new_char-array_offset"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">char</span><span class="cp">[]</span> <span class="cp">*</span> <span class="ci">index</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">streamBeginIndex</span><span class="cp">:</span> <span class="ci">int64</span> 183 <span class="cr">-></span> <span class="ci">CharStream</span> 184 185 <a id="interface.new_char-pointer:B:" href="#members.new_char-pointer"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">NativePtr</span><span class="cp"><</span><span class="ci">char</span><span class="cp">></span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> 186 <span class="cr">-></span> <span class="ci">CharStream</span> 187 <a id="interface.new_char-pointer_offset:B:" href="#members.new_char-pointer_offset"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">NativePtr</span><span class="cp"><</span><span class="ci">char</span><span class="cp">></span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">streamBeginIndex</span><span class="cp">:</span> <span class="ci">int64</span> 188 <span class="cr">-></span> <span class="ci">CharStream</span> 189 190 <a id="interface.new_file-path:B:" href="#members.new_file-path"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">path</span><span class="cp">:</span> <span class="ci">string</span> <span class="cp">*</span> <span class="ci">encoding</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">Text</span><span class="cm">.</span><span class="ci">Encoding</span></a> 191 <span class="cr">-></span> <span class="ci">CharStream</span> 192 <a id="interface.new_file-path_2:B:" href="#members.new_file-path_2"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">path</span><span class="cp">:</span> <span class="ci">string</span> 193 <span class="cp">*</span> <span class="ci">encoding</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">Text</span><span class="cm">.</span><span class="ci">Encoding</span></a> <span class="cp">*</span> <span class="ci">detectEncodingFromByteOrderMarks</span><span class="cp">:</span> <span class="ci">bool</span> 194 <span class="cr">-></span> <span class="ci">CharStream</span> 195 <a id="interface.new_file-path_3:B:" href="#members.new_file-path_3"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">path</span><span class="cp">:</span> <span class="ci">string</span> 196 <span class="cp">*</span> <span class="ci">encoding</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">Text</span><span class="cm">.</span><span class="ci">Encoding</span></a> <span class="cp">*</span> <span class="ci">detectEncodingFromByteOrderMarks</span><span class="cp">:</span> <span class="ci">bool</span> 197 <span class="cp">*</span> <span class="ci">blockSize</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">blockOverlap</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">minRegexSpace</span><span class="cp">:</span> <span class="ci">int</span> 198 <span class="cp">*</span> <span class="ci">byteBufferLength</span><span class="cp">:</span> <span class="ci">int</span> 199 <span class="cr">-></span> <span class="ci">CharStream</span> 200 201 <a id="interface.new_stream:B:" href="#members.new_stream"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">stream</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a> <span class="cp">*</span> <span class="ci">encoding</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">Text</span><span class="cm">.</span><span class="ci">Encoding</span></a> 202 <span class="cr">-></span> <span class="ci">CharStream</span> 203 <a id="interface.new_stream_2:B:" href="#members.new_stream_2"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">stream</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a> <span class="cp">*</span> <span class="ci">leaveOpen</span><span class="cp">:</span> <span class="ci">bool</span> 204 <span class="cp">*</span> <span class="ci">encoding</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">Text</span><span class="cm">.</span><span class="ci">Encoding</span></a> 205 <span class="cr">-></span> <span class="ci">CharStream</span> 206 <a id="interface.new_stream_3:B:" href="#members.new_stream_3"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">stream</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a> <span class="cp">*</span> <span class="ci">leaveOpen</span><span class="cp">:</span> <span class="ci">bool</span> 207 <span class="cp">*</span> <span class="ci">encoding</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">Text</span><span class="cm">.</span><span class="ci">Encoding</span></a> <span class="cp">*</span> <span class="ci">detectEncodingFromByteOrderMarks</span><span class="cp">:</span> <span class="ci">bool</span> 208 <span class="cr">-></span> <span class="ci">CharStream</span> 209 <a id="interface.new_stream_4:B:" href="#members.new_stream_4"><span class="interface-member-marker"><span class="ck">new</span></span></a><span class="cp">:</span> <span class="ci">stream</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a> <span class="cp">*</span> <span class="ci">leaveOpen</span><span class="cp">:</span> <span class="ci">bool</span> 210 <span class="cp">*</span> <span class="ci">encoding</span><span class="cp">:</span> <a href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">System</span><span class="cm">.</span><span class="ci">Text</span><span class="cm">.</span><span class="ci">Encoding</span></a> <span class="cp">*</span> <span class="ci">detectEncodingFromByteOrderMarks</span><span class="cp">:</span> <span class="ci">bool</span> 211 <span class="cp">*</span> <span class="ci">blockSize</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">blockOverlap</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">minRegexSpace</span><span class="cp">:</span> <span class="ci">int</span> 212 <span class="cp">*</span> <span class="ci">byteBufferLength</span><span class="cp">:</span> <span class="ci">int</span> 213 <span class="cr">-></span> <span class="ci">CharStream</span> 214 215 <span class="ck">member</span> <a id="interface.Dispose:B:" href="#members.Dispose"><span class="interface-member-marker"><span class="ci">Dispose</span></span></a><span class="cp">:</span> <span class="ci">unit</span> <span class="cr">-></span> <span class="ci">unit</span> 216 217 <span class="ck">member</span> <a id="interface.Begin:B:" href="#members.Begin"><span class="interface-member-marker"><span class="ci">Begin</span></span></a><span class="cp">:</span> <a href="#Iterator"><span class="ci">Iterator</span></a> 218 <span class="ck">member</span> <a id="interface.Seek:B:" href="#members.Seek"><span class="interface-member-marker"><span class="ci">Seek</span></span></a><span class="cp">:</span> <span class="ci">int64</span> <span class="cr">-></span> <a href="#Iterator"><span class="ci">Iterator</span></a> 219 220 <span class="ck">member</span> <a id="interface.BeginIndex:B:" href="#members.BeginIndex"><span class="interface-member-marker"><span class="ci">BeginIndex</span></span></a><span class="cp">:</span> <span class="ci">int64</span> 221 <span class="ck">member</span> <a id="interface.EndIndex:B:" href="#members.EndIndex"><span class="interface-member-marker"><span class="ci">EndIndex</span></span></a><span class="cp">:</span> <span class="ci">int64</span> 222 223 <span class="ck">static</span> <span class="ck">member</span> <a id="interface.FoldCase:B:" href="#members.FoldCase"><span class="interface-member-marker"><span class="ci">FoldCase</span></span></a><span class="cp">:</span> <span class="ci">string</span> <span class="cr">-></span> <span class="ci">string</span> 224 <span class="ck">static</span> <span class="ck">member</span> <a id="interface.NormalizeNewlines:B:" href="#members.NormalizeNewlines"><span class="interface-member-marker"><span class="ci">NormalizeNewlines</span></span></a><span class="cp">:</span> <span class="ci">string</span> <span class="cr">-></span> <span class="ci">string</span> 225 226 <span class="cp">[<</span><span class="ci">Struct</span><span class="cp">>]</span> 227 <span class="ck">type</span> <span class="bold"><a href="#Iterator"><span class="ci">Iterator</span></a></span> <span class="cp">=</span> <span class="cbc"><span class="left-delimiter">(*</span> see below <span class="right-delimiter">*)</span></span> 228 229</pre> 230 </div> 231 </div> 232 </div> 233 <div id="remarks" class="section s3"> 234 <h2 class="title h3"><span class="section-number">2.8.2</span> Remarks</h2> 235 <div class="intro i3"> 236 <p class="para _1"> 237 The <code class="fsharp"><span class="ci">CharStream</span></code> class provides an <a 238 href="http://en.wikipedia.org/wiki/Iterator">iterator‐based</a> interface for reading UTF‐16 chars from a binary stream or an in‐memory char 239 buffer (e.g. a string). It is optimized for use in backtracking parser applications and supports arbitrary <em>char‐based</em> seeking, even for 240 streams larger than physical memory. 241 </p> 242 <p class="para _2"> 243 A <code class="fsharp"><span class="ci">CharStream</span></code> constructed from a <code class="fsharp"><a 244 href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span 245 class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a></code> or a file path reads the stream block‐wise and only holds 246 the most recently accessed block in memory. The blocks overlap in order to provide efficient access on the boundary between blocks. 247 </p> 248 <p class="para _3"> 249 If the char content is already available as a string or a char array, a <code class="fsharp"><span class="ci">CharStream</span></code> can be 250 directly constructed from the char buffer (without needing to copy the buffer). The overhead of accessing an in‐memory char buffer through a 251 <code class="fsharp"><span class="ci">CharStream</span></code> is minimal. 252 </p> 253 <p class="para _4"> 254 You can access the content of a <code class="fsharp"><span class="ci">CharStream</span></code> only through its <code class="fsharp"><a 255 href="#Iterator"><span class="ci">Iterator</span></a></code> instances. The <code class="fsharp"><a href="#members.Begin"><span 256 class="ci">Begin</span></a></code> property returns an iterator pointing to the first char in the stream. The <code class="fsharp"><a 257 href="#members.Seek"><span class="ci">Seek</span></a></code> method returns an iterator pointing to the char with a given index. 258 </p> 259 <div class="para _5"> 260 <div class="dl multi-para"> 261 <dl class="dl multi-para"> 262 <dt class="_1"><a id="remarks.low-trust-version"></a>Low‐Trust version</dt> 263 <dd class="_1"> 264 <div class="para _1"> 265 <p> 266 If you compile FParsec with the <code class="fsharp"><span class="ci">LOW_TRUST</span></code> conditional compiler symbol, the <code 267 class="fsharp"><span class="ci">CharStream</span></code> class differs from the normal version as follows: 268 </p> 269 <ul class="l1"> 270 <li class="_1"> 271 No <a href="http://msdn.microsoft.com/en-us/library/t2yzs44b.aspx">unverifiable code</a> involving pointers is used. <em>This allows 272 FParsec to be executed in an environment with reduced trust</em>, such as medium trust ASP.NET applications. 273 </li> 274 <li class="_2"> 275 A <code class="fsharp"><span class="ci">CharStream</span></code> that is constructed from a <code class="fsharp"><a 276 href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span 277 class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a></code> or a file path reads the complete file into a single 278 string during construction. <em>This severely limits the maximum practical stream size.</em> 279 </li> 280 <li class="_3"> 281 Although the <code class="fsharp"><span class="ci">CharStream</span></code> class still supports the <code class="fsharp"><span 282 class="ci">IDisposable</span></code> interface, disposing the <code class="fsharp"><span class="ci">CharStream</span></code> instances is 283 no longer necessary, since no resources are hold that need to be explicitly released. 284 </li> 285 </ul> 286 </div> 287 </dd> 288 <dt class="_2"><a id="remarks.non-sequential-access"></a>Non‐sequential access</dt> 289 <dd class="_2"> 290 <p class="para _1"> 291 <span class="small"><span class="italic">This note does not apply to the <a href="#remarks.low-trust-version">Low‐Trust version</a> of 292 FParsec.</span></span><br /> If you construct a <code class="fsharp"><span class="ci">CharStream</span></code> from a <code class="fsharp"><a 293 href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span 294 class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a></code> or a file path and you backtrack over a distance long 295 enough to require the <code class="fsharp"><span class="ci">CharStream</span></code> to reread a previous block, then the underlying 296 <strong>byte stream needs to support seeking</strong>, otherwise a <code class="fsharp"><a 297 href="http://msdn.microsoft.com/en-us/library/system.notsupportedexception.aspx"><span class="ci">NotSupportedException</span></a></code> is 298 thrown. Furthermore, the <strong><a href="http://msdn.microsoft.com/en-us/library/system.text.decoder.aspx">Decoder</a> for the input <a 299 href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx">Encoding</a> must be serializable</strong> if you backtrack to a 300 block other than the first in the stream. Note that <em>file streams created for regular disk files are always seekable and all the .NET 301 standard decoders are serializable</em>. In order to support non‐seekable streams for applications which don’t require extensive 302 backtracking, no exception will be thrown before an operation actually requires backtracking and the necessary capabilities of the stream or 303 decoder are not available. 304 </p> 305 </dd> 306 <dt class="_3">Decoder errors</dt> 307 <dd class="_3"> 308 <p class="para _1"> 309 A <code class="fsharp"><span class="ci">CharStream</span></code> constructed from a binary input stream decodes the input data with the help 310 of a <code class="fsharp"><a href="http://msdn.microsoft.com/en-us/library/system.text.decoder.aspx"><span 311 class="ci">Decoder</span></a></code> instance obtained via the <code class="fsharp"><span class="ci">Encodings</span></code>’s <code 312 class="fsharp"><span class="ci">GetDecoder</span></code> method. Depending on the configuration of the encoding the decoder might throw an 313 exception if it encounters invalid byte sequences, usually a <code class="fsharp"><span class="ci">System</span><span 314 class="cm">.</span><span class="ci">Text</span><span class="cm">.</span><a 315 href="http://msdn.microsoft.com/en-us/library/system.text.decoderfallbackexception.aspx"><span 316 class="ci">DecoderFallbackException</span></a></code> or a <code class="fsharp"><span class="ci">System</span><span class="cm">.</span><span 317 class="ci">IO</span><span class="cm">.</span><a href="http://msdn.microsoft.com/en-us/library/system.argumentexception.aspx"><span 318 class="ci">ArgumentException</span></a></code>. <sup class="fn-mark"><a id="remarks.:FN:1:B:" href="#remarks.:FN:1">[1]</a></sup> 319 </p> 320 </dd> 321 <dt class="_4"><a id="remarks.case-insensitive-matching"></a>Case‐insensitive matching</dt> 322 <dd class="_4"> 323 <p class="para _1"> 324 The <code class="fsharp"><a href="#Iterator.members.MatchCaseFolded"><span class="ci">MatchCaseFolded</span></a></code> member matches the 325 content of the stream “case‐insensitively” with a reference string. In this instance “case‐insensitive” means 326 that before the chars are matched with the reference string they are <a 327 href="http://unicode.org/reports/tr21/tr21-5.html#Caseless_Matching">mapped to a canonical form where case differences are erased</a>. For 328 performance reasons <code class="fsharp"><a href="#Iterator.members.MatchCaseFolded"><span class="ci">MatchCaseFolded</span></a></code> only 329 applies the (non‐Turkic) 1‐to‐1 <a href="http://unicode.org/Public/UNIDATA/CaseFolding.txt">case folding mappings</a> (v. 5.1) for Unicode 330 code points in the Basic Multilingual Plane, i.e. code points below 0x10000. These mappings are sufficient for many case‐insensitive parser 331 grammars encountered in practice, but they are not appropriate for matching arbitrary natural language content. Also note that the <code 332 class="fsharp"><span class="ci">CharStream</span></code> class performs no Unicode <a 333 href="http://unicode.org/reports/tr15/">normalization</a>. 334 </p> 335 </dd> 336 <dt class="_5">Newlines</dt> 337 <dd class="_5"> 338 <p class="para _1"> 339 The <code class="fsharp"><span class="ci">CharStream</span></code> class itself does not apply any automatic newline conversion to the 340 processed input. A useful helper function for this purpose is <code class="fsharp"><a href="#members.NormalizeNewlines"><span 341 class="ci">NormalizeNewlines</span></a></code>, which replaces all ASCII newline character sequences (<code class="fsharp"><span 342 class="cs"><span class="left-delimiter">"</span><span class="ce">\n</span><span class="right-delimiter">"</span></span></code>, <code 343 class="fsharp"><span class="cs"><span class="left-delimiter">"</span><span class="ce">\r</span><span class="ce">\n</span><span 344 class="right-delimiter">"</span></span></code> or <code class="fsharp"><span class="cs"><span class="left-delimiter">"</span><span 345 class="ce">\r</span><span class="right-delimiter">"</span></span></code>) in a given string with <code class="fsharp"><span class="cs"><span 346 class="left-delimiter">"</span><span class="ce">\n</span><span class="right-delimiter">"</span></span></code>. 347 </p> 348 </dd> 349 <dt class="_6">Disposable interface</dt> 350 <dd class="_6"> 351 <p class="para _1"> 352 <span class="small"><span class="italic">This note does not apply to the <a href="#remarks.low-trust-version">Low‐Trust version</a> of 353 FParsec.</span></span><br /> A <code class="fsharp"><span class="ci">CharStream</span></code> holds managed and unmanaged resources that need 354 to be explicitly released. Hence, it is very important that <code class="fsharp"><span class="ci">CharStream</span></code> objects are 355 promptly disposed after use. Where possible <code class="fsharp"><span class="ci">CharStream</span></code> objects should only be used within 356 a “using” block (C#), a “use” expression( F#) or similar constructs in other languages. Members of <code 357 class="fsharp"><a href="#Iterator"><span class="ci">CharStream</span><span class="cm">.</span><span class="ci">Iterator</span></a></code> 358 instances must not be accessed after the corresponding <code class="fsharp"><span class="ci">CharStream</span></code> has been disposed, of 359 course. 360 </p> 361 </dd> 362 <dt class="_7">Thread safety</dt> 363 <dd class="_7"> 364 <p class="para _1"> 365 A <code class="fsharp"><span class="ci">CharStream</span></code> constructed from a <code class="fsharp"><a 366 href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span 367 class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a></code> or a file path is not thread safe and its members may 368 not be accessed by multiple threads, except in the <a href="#remarks.low-trust-version">Low‐Trust version</a> of FParsec. A <code 369 class="fsharp"><span class="ci">CharStream</span></code> constructed directly from a string, char array or char pointer can always be safely 370 accessed concurrently using different <code class="fsharp"><span class="ci">Iterators</span></code> (provided that the <code 371 class="fsharp"><span class="ci">CharStream</span></code> is disposed by only one thread, of course). <span class="italic"><code 372 class="fsharp"><a href="#Iterator"><span class="ci">CharStream</span><span class="cm">.</span><span class="ci">Iterator</span></a></code> 373 instances are never thread safe.</span> 374 </p> 375 </dd> 376 </dl> 377 </div> 378 </div> 379 </div> 380 </div> 381 <div id="exceptions" class="section s3"> 382 <h2 class="title h3"><span class="section-number">2.8.3</span> I/O exceptions</h2> 383 <div class="intro i3"> 384 <p class="para _1"> 385 If you construct a <code class="fsharp"><span class="ci">CharStream</span></code> from a <code class="fsharp"><a 386 href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span class="ci">System</span><span class="cm">.</span><span 387 class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a></code> or a file path, any <code class="fsharp"><span 388 class="ci">CharStream</span></code> operation that requires reading chars from the underlying byte stream may throw one of the following 389 exceptions. 390 </p> 391 <p class="para _2"> 392 In general it is <em>not</em> safe to continue to use a <code class="fsharp"><span class="ci">CharStream</span></code> instance after one of 393 these exceptions was thrown. Calling <code class="fsharp"><a href="#members.Dispose"><span class="ci">Dispose</span></a><span 394 class="cp">()</span></code> on an <code class="fsharp"><span class="ci">CharStream</span></code> instance after the instance or one of its 395 iterators has thrown an exception is always safe. 396 </p> 397 <div class="para _3"> 398 <div class="dl multi-para"> 399 <dl class="dl multi-para"> 400 <dt class="_1"><code class="fsharp"><span class="ci">NotSupportedException</span></code></dt> 401 <dd class="_1"> 402 <p class="para _1"> 403 Seeking of the underlying byte stream is required, but the byte stream does not support seeking or the <code class="fsharp"><a 404 href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">Encoding</span></a></code>’s <code 405 class="fsharp"><a href="http://msdn.microsoft.com/en-us/library/system.text.decoder.aspx"><span class="ci">Decoder</span></a></code> is not 406 serializable. See also the remarks above on <a href="#remarks.non-sequential-access">non‐sequential access</a>. 407 </p> 408 </dd> 409 <dt class="_2"><code class="fsharp"><span class="ci">IOException</span></code></dt> 410 <dd class="_2"> 411 <p class="para _1">An I/O occured while reading data from the underlying byte stream.</p> 412 </dd> 413 <dt class="_3"><code class="fsharp"><span class="ci">ArgumentException</span></code></dt> 414 <dd class="_3"> 415 <p class="para _1"> 416 The underlying byte stream contains invalid bytes and the <code class="fsharp"><a 417 href="http://msdn.microsoft.com/en-us/library/system.text.encoding.aspx"><span class="ci">Encoding</span></a></code> was constructed with the 418 <code class="fsharp"><span class="ci">throwOnInvalidBytes</span></code> option. 419 </p> 420 </dd> 421 <dt class="_4"><code class="fsharp"><span class="ci">DecoderFallbackException</span></code></dt> 422 <dd class="_4"> 423 <p class="para _1">The underlying byte stream contains invalid bytes for which the decoder fallback threw this exception.</p> 424 <p class="para _2"> 425 The byte index of the invalid bytes in the stream is stored as a boxed <code class="fsharp"><span class="ci">System</span><span 426 class="cm">.</span><span class="ci">Int64</span></code> in the <code class="fsharp"><span class="cs"><span 427 class="left-delimiter">"</span>Stream.Position<span class="right-delimiter">"</span></span></code> entry of the <code class="fsharp"><a 428 href="http://msdn.microsoft.com/en-us/library/system.exception.data.aspx"><span class="ci">Data</span></a></code> member of the exception 429 instance. The precision of the index depends on the precision of the <code class="fsharp"><a 430 href="http://msdn.microsoft.com/en-us/library/system.text.decoderfallbackexception.aspx"><span 431 class="ci">DecoderFallbackException</span></a></code>’s <code class="fsharp"><a 432 href="http://msdn.microsoft.com/en-us/library/system.text.decoderfallbackexception.index.aspx"><span class="ci">Index</span></a></code> 433 member. If the underlying <code class="fsharp"><a href="http://msdn.microsoft.com/en-us/library/system.io.stream.aspx"><span 434 class="ci">System</span><span class="cm">.</span><span class="ci">IO</span><span class="cm">.</span><span class="ci">Stream</span></a></code> 435 is not seekable, the byte index only takes into account the bytes read by the <code class="fsharp"><span class="ci">CharStream</span></code>, 436 but not any bytes read before the <code class="fsharp"><span class="ci">CharStream</span></code> was constructed. 437 </p> 438 </dd> 439 </dl> 440 </div> 441 </div> 442 </div> 443 </div> 444 <div id="members" class="section s3"> 445 <h2 class="title h3"><span class="section-number">2.8.4</span> Members</h2> 446 <div class="intro i3"> 447 <div class="para _1"> 448 <div class="interface-members"> 449 <div class="interface-member _1" id="members.new_string"> 450 <div class="interface-member-code"> 451 <a class="interface-member-backlink" href="#interface.new_string:B:">⇧</a> 452<pre class="code fsharp"><span class="interface-member-marker"><span class="ck">new</span></span><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">string</span> <span class="cp">*</span> <span class="ci">index</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> 453 <span class="cr">-></span> <span class="ci">CharStream</span> 454</pre> 455 </div> 456 <div class="interface-member-description"> 457 <p class="para _1"> 458 Is equivalent to <code class="fsharp"><a href="#members.new_string_offset"><span class="ck">new</span> <span 459 class="ci">CharStream</span></a><span class="cp">(</span><span class="ci">chars</span><span class="cp">,</span> <span 460 class="ci">index</span><span class="cp">,</span> <span class="ci">length</span><span class="cp">,</span> <span class="cn">0L</span><span 461 class="cp">)</span></code>. 462 </p> 463 </div> 464 </div> 465 <div class="interface-member _2" id="members.new_string_offset"> 466 <div class="interface-member-code"> 467 <a class="interface-member-backlink" href="#interface.new_string_offset:B:">⇧</a> 468<pre class="code fsharp"><span class="interface-member-marker"><span class="ck">new</span></span><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">string</span> <span class="cp">*</span> <span class="ci">index</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">streamBeginIndex</span><span class="cp">:</span> <span class="ci">int64</span> 469 <span class="cr">-></span> <span class="ci">CharStream</span> 470</pre> 471 </div> 472 <div class="interface-member-description"> 473 <p class="para _1"> 474 Constructs a <code class="fsharp"><span class="ci">CharStream</span></code> from the chars in the string argument between the indices <code 475 class="fsharp"><span class="ci">index</span></code> (inclusive) and <code class="fsharp"><span class="ci">index</span> <span 476 class="co">+</span> <span class="ci">length</span></code> (exclusive). By directly referencing the chars in the string this constructor 477 avoids any copy of the string content. 478 </p> 479 <p class="para _2"> 480 The first char in the stream is assigned the index <code class="fsharp"><span class="ci">streamBeginIndex</span></code>. A positive <code 481 class="fsharp"><span class="ci">streamBeginIndex</span></code> allows you for example to create a substream of another <code 482 class="fsharp"><span class="ci">CharStream</span></code>, i.e. a <code class="fsharp"><span class="ci">CharStream</span></code> instance that 483 only contains a sub‐segment of another char stream but is accessible through the same char indices. 484 </p> 485 <div class="para _3"> 486 <p> 487 <code class="fsharp"><span class="ci">chars</span></code> must not be null. An <code class="fsharp"><a 488 href="http://msdn.microsoft.com/en-us/library/system.argumentoutofrangeexception.aspx"><span 489 class="ci">ArgumentOutOfRangeException</span></a></code> is thrown if the arguments do not satisfy the following conditions: 490 </p> 491 <ul class="l1"> 492 <li class="_1"> 493 <code class="fsharp"><span class="ci">index</span></code> ≥ 0, <code class="fsharp"><span class="ci">length</span></code> ≥ 0, <code 494 class="fsharp"><span class="ci">index</span></code> + <code class="fsharp"><span class="ci">length</span></code> ≤ <code 495 class="fsharp"><span class="ci">chars</span><span class="cm">.</span><span class="ci">Length</span></code> and 496 </li> 497 <li class="_2">0 ≤ <code class="fsharp"><span class="ci">streamBeginIndex</span></code> < 2<sup>60</sup>.</li> 498 </ul> 499 </div> 500 <div class="para _4"> 501 <div class="admonition"> 502 <div class="admonition-title">Important</div> 503 <div class="admonition-body"> 504 <p class="para _1"> 505 <span class="small"><span class="italic">This note does not apply to the <a href="#remarks.low-trust-version">Low‐Trust version</a> of 506 FParsec.</span></span><br /> The given string is “<a 507 href="http://msdn.microsoft.com/en-us/library/83y4ak54.aspx">pinned</a>” until the <code class="fsharp"><span 508 class="ci">CharStream</span></code> is disposed. Pinning the string prevents the GC from moving it around in memory during garbage 509 collection. On .NET (at least in versions up to and including 4.0) the pinning has no effect if the string is large enough to be allocated 510 on the Large Object Heap, i.e. has a length of about 42500 chars or more. However, pinning smaller strings does constrain the normal 511 operations of the GC. Thus, <strong>to minimize the negative impact on the GC, you should dispose <code class="fsharp"><span 512 class="ci">CharStream</span></code> instances constructed from small strings as soon as you’re done parsing it</strong>. If you keep a 513 large number of <code class="fsharp"><span class="ci">CharStream</span></code> instances constructed from small strings around for an 514 extended period of time, you risk fragmenting the heap. 515 </p> 516 </div> 517 </div> 518 </div> 519 </div> 520 </div> 521 <div class="interface-member _3" id="members.new_char-array"> 522 <div class="interface-member-code"> 523 <a class="interface-member-backlink" href="#interface.new_char-array:B:">⇧</a> 524<pre class="code fsharp"><span class="interface-member-marker"><span class="ck">new</span></span><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">char</span><span class="cp">[]</span> <span class="cp">*</span> <span class="ci">index</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> 525 <span class="cr">-></span> <span class="ci">CharStream</span> 526</pre> 527 </div> 528 <div class="interface-member-description"> 529 <p class="para _1"> 530 <span class="small"><span class="italic">This constructor is not available in the <a href="#remarks.low-trust-version">Low‐Trust version</a> 531 of FParsec.</span></span> 532 </p> 533 <p class="para _2"> 534 Is equivalent to <code class="fsharp"><a href="#members.new_char-array_offset"><span class="ck">new</span> <span 535 class="ci">CharStream</span></a><span class="cp">(</span><span class="ci">chars</span><span class="cp">,</span> <span 536 class="ci">index</span><span class="cp">,</span> <span class="ci">length</span><span class="cp">,</span> <span class="cn">0L</span><span 537 class="cp">)</span></code>. 538 </p> 539 </div> 540 </div> 541 <div class="interface-member _4" id="members.new_char-array_offset"> 542 <div class="interface-member-code"> 543 <a class="interface-member-backlink" href="#interface.new_char-array_offset:B:">⇧</a> 544<pre class="code fsharp"><span class="interface-member-marker"><span class="ck">new</span></span><span class="cp">:</span> <span class="ci">chars</span><span class="cp">:</span> <span class="ci">char</span><span class="cp">[]</span> <span class="cp">*</span> <span class="ci">index</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">length</span><span class="cp">:</span> <span class="ci">int</span> <span class="cp">*</span> <span class="ci">streamBeginIndex</span><span class="cp">:</span> <span class="ci">int64</span> 545 <span class="cr">-></span> <span class="ci">CharStream</span> 546</pre> 547 </div> 548 <div class="interface-member-description"> 549 <p class="para _1"> 550 <span class="small"><span class="italic">This constructor is not available in the <a href="#remarks.low-trust-version">Low‐Trust version</a> 551 of FParsec.</span></span> 552 </p> 553 <p class="para _2"> 554 Constructs a <code class="fsharp"><span class="ci">CharStream</span></code> from the chars in the char array argument between the indices 555 <code class="fsharp"><span class="ci">index</span></code> (inclusive) and <code class="fsharp"><span class="ci">index</span> <span 556 class="co">+</span> <span class="ci">length</span></code> (exclusive). By directly referencing the chars in the char array this constructor 557 avoids any copy of the char array content. 558 </p> 559 <p class="para _3"> 560 The first char in the stream is assigned the index <code class="fsharp"><span class="ci">streamBeginIndex</span></code>. A positive <code 561 class="fsharp"><span class="ci">streamBeginIndex</span></code> allows you for example to create a substream of another <code 562 class="fsharp"><span class="ci">CharStream</span></code>, i.e. a <code class="fsharp"><span class="ci">CharStream</span></code> instance that 563 only contains a sub‐segment of another char stream but is accessible through the same char indices. 564 </p> 565 <div class="para _4"> 566 <p> 567 <code class="fsharp"><span class="ci">chars</span></code> must not be null. An <code class="fsharp"><a 568 href="http://msdn.microsoft.com/en-us/library/system.argumentoutofrangeexception.aspx"><span 569 class="ci">ArgumentOutOfRangeException</span></a></code> is thrown if the arguments do not satisfy the following conditions: 570 </p> 571 <ul class="l1"> 572 <li class="_1"> 573 …
Large files files are truncated, but you can click here to view the full file