/thirdparty/breakpad/third_party/protobuf/protobuf/src/google/protobuf/io/tokenizer.cc
C++ | 694 lines | 339 code | 15 blank | 340 comment | 50 complexity | 5ac845cb643c37a1f20d81ddb0dc1733 MD5 | raw file
1// Protocol Buffers - Google's data interchange format 2// Copyright 2008 Google Inc. All rights reserved. 3// http://code.google.com/p/protobuf/ 4// 5// Redistribution and use in source and binary forms, with or without 6// modification, are permitted provided that the following conditions are 7// met: 8// 9// * Redistributions of source code must retain the above copyright 10// notice, this list of conditions and the following disclaimer. 11// * Redistributions in binary form must reproduce the above 12// copyright notice, this list of conditions and the following disclaimer 13// in the documentation and/or other materials provided with the 14// distribution. 15// * Neither the name of Google Inc. nor the names of its 16// contributors may be used to endorse or promote products derived from 17// this software without specific prior written permission. 18// 19// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 20// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 21// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 22// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 23// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 24// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 25// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 26// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 27// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 28// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 31// Author: kenton@google.com (Kenton Varda) 32// Based on original Protocol Buffers design by 33// Sanjay Ghemawat, Jeff Dean, and others. 34// 35// Here we have a hand-written lexer. At first you might ask yourself, 36// "Hand-written text processing? Is Kenton crazy?!" Well, first of all, 37// yes I am crazy, but that's beside the point. There are actually reasons 38// why I ended up writing this this way. 39// 40// The traditional approach to lexing is to use lex to generate a lexer for 41// you. Unfortunately, lex's output is ridiculously ugly and difficult to 42// integrate cleanly with C++ code, especially abstract code or code meant 43// as a library. Better parser-generators exist but would add dependencies 44// which most users won't already have, which we'd like to avoid. (GNU flex 45// has a C++ output option, but it's still ridiculously ugly, non-abstract, 46// and not library-friendly.) 47// 48// The next approach that any good software engineer should look at is to 49// use regular expressions. And, indeed, I did. I have code which 50// implements this same class using regular expressions. It's about 200 51// lines shorter. However: 52// - Rather than error messages telling you "This string has an invalid 53// escape sequence at line 5, column 45", you get error messages like 54// "Parse error on line 5". Giving more precise errors requires adding 55// a lot of code that ends up basically as complex as the hand-coded 56// version anyway. 57// - The regular expression to match a string literal looks like this: 58// kString = new RE("(\"([^\"\\\\]|" // non-escaped 59// "\\\\[abfnrtv?\"'\\\\0-7]|" // normal escape 60// "\\\\x[0-9a-fA-F])*\"|" // hex escape 61// "\'([^\'\\\\]|" // Also support single-quotes. 62// "\\\\[abfnrtv?\"'\\\\0-7]|" 63// "\\\\x[0-9a-fA-F])*\')"); 64// Verifying the correctness of this line noise is actually harder than 65// verifying the correctness of ConsumeString(), defined below. I'm not 66// even confident that the above is correct, after staring at it for some 67// time. 68// - PCRE is fast, but there's still more overhead involved than the code 69// below. 70// - Sadly, regular expressions are not part of the C standard library, so 71// using them would require depending on some other library. For the 72// open source release, this could be really annoying. Nobody likes 73// downloading one piece of software just to find that they need to 74// download something else to make it work, and in all likelihood 75// people downloading Protocol Buffers will already be doing so just 76// to make something else work. We could include a copy of PCRE with 77// our code, but that obligates us to keep it up-to-date and just seems 78// like a big waste just to save 200 lines of code. 79// 80// On a similar but unrelated note, I'm even scared to use ctype.h. 81// Apparently functions like isalpha() are locale-dependent. So, if we used 82// that, then if this code is being called from some program that doesn't 83// have its locale set to "C", it would behave strangely. We can't just set 84// the locale to "C" ourselves since we might break the calling program that 85// way, particularly if it is multi-threaded. WTF? Someone please let me 86// (Kenton) know if I'm missing something here... 87// 88// I'd love to hear about other alternatives, though, as this code isn't 89// exactly pretty. 90 91#include <google/protobuf/io/tokenizer.h> 92#include <google/protobuf/io/zero_copy_stream.h> 93#include <google/protobuf/stubs/strutil.h> 94 95namespace google { 96namespace protobuf { 97namespace io { 98namespace { 99 100// As mentioned above, I don't trust ctype.h due to the presence of "locales". 101// So, I have written replacement functions here. Someone please smack me if 102// this is a bad idea or if there is some way around this. 103// 104// These "character classes" are designed to be used in template methods. 105// For instance, Tokenizer::ConsumeZeroOrMore<Whitespace>() will eat 106// whitespace. 107 108// Note: No class is allowed to contain '\0', since this is used to mark end- 109// of-input and is handled specially. 110 111#define CHARACTER_CLASS(NAME, EXPRESSION) \ 112 class NAME { \ 113 public: \ 114 static inline bool InClass(char c) { \ 115 return EXPRESSION; \ 116 } \ 117 } 118 119CHARACTER_CLASS(Whitespace, c == ' ' || c == '\n' || c == '\t' || 120 c == '\r' || c == '\v' || c == '\f'); 121 122CHARACTER_CLASS(Unprintable, c < ' ' && c > '\0'); 123 124CHARACTER_CLASS(Digit, '0' <= c && c <= '9'); 125CHARACTER_CLASS(OctalDigit, '0' <= c && c <= '7'); 126CHARACTER_CLASS(HexDigit, ('0' <= c && c <= '9') || 127 ('a' <= c && c <= 'f') || 128 ('A' <= c && c <= 'F')); 129 130CHARACTER_CLASS(Letter, ('a' <= c && c <= 'z') || 131 ('A' <= c && c <= 'Z') || 132 (c == '_')); 133 134CHARACTER_CLASS(Alphanumeric, ('a' <= c && c <= 'z') || 135 ('A' <= c && c <= 'Z') || 136 ('0' <= c && c <= '9') || 137 (c == '_')); 138 139CHARACTER_CLASS(Escape, c == 'a' || c == 'b' || c == 'f' || c == 'n' || 140 c == 'r' || c == 't' || c == 'v' || c == '\\' || 141 c == '?' || c == '\'' || c == '\"'); 142 143#undef CHARACTER_CLASS 144 145// Given a char, interpret it as a numeric digit and return its value. 146// This supports any number base up to 36. 147inline int DigitValue(char digit) { 148 if ('0' <= digit && digit <= '9') return digit - '0'; 149 if ('a' <= digit && digit <= 'z') return digit - 'a' + 10; 150 if ('A' <= digit && digit <= 'Z') return digit - 'A' + 10; 151 return -1; 152} 153 154// Inline because it's only used in one place. 155inline char TranslateEscape(char c) { 156 switch (c) { 157 case 'a': return '\a'; 158 case 'b': return '\b'; 159 case 'f': return '\f'; 160 case 'n': return '\n'; 161 case 'r': return '\r'; 162 case 't': return '\t'; 163 case 'v': return '\v'; 164 case '\\': return '\\'; 165 case '?': return '\?'; // Trigraphs = :( 166 case '\'': return '\''; 167 case '"': return '\"'; 168 169 // We expect escape sequences to have been validated separately. 170 default: return '?'; 171 } 172} 173 174} // anonymous namespace 175 176ErrorCollector::~ErrorCollector() {} 177 178// =================================================================== 179 180Tokenizer::Tokenizer(ZeroCopyInputStream* input, 181 ErrorCollector* error_collector) 182 : input_(input), 183 error_collector_(error_collector), 184 buffer_(NULL), 185 buffer_size_(0), 186 buffer_pos_(0), 187 read_error_(false), 188 line_(0), 189 column_(0), 190 token_start_(-1), 191 allow_f_after_float_(false), 192 comment_style_(CPP_COMMENT_STYLE) { 193 194 current_.line = 0; 195 current_.column = 0; 196 current_.end_column = 0; 197 current_.type = TYPE_START; 198 199 Refresh(); 200} 201 202Tokenizer::~Tokenizer() { 203 // If we had any buffer left unread, return it to the underlying stream 204 // so that someone else can read it. 205 if (buffer_size_ > buffer_pos_) { 206 input_->BackUp(buffer_size_ - buffer_pos_); 207 } 208} 209 210// ------------------------------------------------------------------- 211// Internal helpers. 212 213void Tokenizer::NextChar() { 214 // Update our line and column counters based on the character being 215 // consumed. 216 if (current_char_ == '\n') { 217 ++line_; 218 column_ = 0; 219 } else if (current_char_ == '\t') { 220 column_ += kTabWidth - column_ % kTabWidth; 221 } else { 222 ++column_; 223 } 224 225 // Advance to the next character. 226 ++buffer_pos_; 227 if (buffer_pos_ < buffer_size_) { 228 current_char_ = buffer_[buffer_pos_]; 229 } else { 230 Refresh(); 231 } 232} 233 234void Tokenizer::Refresh() { 235 if (read_error_) { 236 current_char_ = '\0'; 237 return; 238 } 239 240 // If we're in a token, append the rest of the buffer to it. 241 if (token_start_ >= 0 && token_start_ < buffer_size_) { 242 current_.text.append(buffer_ + token_start_, buffer_size_ - token_start_); 243 token_start_ = 0; 244 } 245 246 const void* data = NULL; 247 buffer_ = NULL; 248 buffer_pos_ = 0; 249 do { 250 if (!input_->Next(&data, &buffer_size_)) { 251 // end of stream (or read error) 252 buffer_size_ = 0; 253 read_error_ = true; 254 current_char_ = '\0'; 255 return; 256 } 257 } while (buffer_size_ == 0); 258 259 buffer_ = static_cast<const char*>(data); 260 261 current_char_ = buffer_[0]; 262} 263 264inline void Tokenizer::StartToken() { 265 token_start_ = buffer_pos_; 266 current_.type = TYPE_START; // Just for the sake of initializing it. 267 current_.text.clear(); 268 current_.line = line_; 269 current_.column = column_; 270} 271 272inline void Tokenizer::EndToken() { 273 // Note: The if() is necessary because some STL implementations crash when 274 // you call string::append(NULL, 0), presumably because they are trying to 275 // be helpful by detecting the NULL pointer, even though there's nothing 276 // wrong with reading zero bytes from NULL. 277 if (buffer_pos_ != token_start_) { 278 current_.text.append(buffer_ + token_start_, buffer_pos_ - token_start_); 279 } 280 token_start_ = -1; 281 current_.end_column = column_; 282} 283 284// ------------------------------------------------------------------- 285// Helper methods that consume characters. 286 287template<typename CharacterClass> 288inline bool Tokenizer::LookingAt() { 289 return CharacterClass::InClass(current_char_); 290} 291 292template<typename CharacterClass> 293inline bool Tokenizer::TryConsumeOne() { 294 if (CharacterClass::InClass(current_char_)) { 295 NextChar(); 296 return true; 297 } else { 298 return false; 299 } 300} 301 302inline bool Tokenizer::TryConsume(char c) { 303 if (current_char_ == c) { 304 NextChar(); 305 return true; 306 } else { 307 return false; 308 } 309} 310 311template<typename CharacterClass> 312inline void Tokenizer::ConsumeZeroOrMore() { 313 while (CharacterClass::InClass(current_char_)) { 314 NextChar(); 315 } 316} 317 318template<typename CharacterClass> 319inline void Tokenizer::ConsumeOneOrMore(const char* error) { 320 if (!CharacterClass::InClass(current_char_)) { 321 AddError(error); 322 } else { 323 do { 324 NextChar(); 325 } while (CharacterClass::InClass(current_char_)); 326 } 327} 328 329// ------------------------------------------------------------------- 330// Methods that read whole patterns matching certain kinds of tokens 331// or comments. 332 333void Tokenizer::ConsumeString(char delimiter) { 334 while (true) { 335 switch (current_char_) { 336 case '\0': 337 case '\n': { 338 AddError("String literals cannot cross line boundaries."); 339 return; 340 } 341 342 case '\\': { 343 // An escape sequence. 344 NextChar(); 345 if (TryConsumeOne<Escape>()) { 346 // Valid escape sequence. 347 } else if (TryConsumeOne<OctalDigit>()) { 348 // Possibly followed by two more octal digits, but these will 349 // just be consumed by the main loop anyway so we don't need 350 // to do so explicitly here. 351 } else if (TryConsume('x') || TryConsume('X')) { 352 if (!TryConsumeOne<HexDigit>()) { 353 AddError("Expected hex digits for escape sequence."); 354 } 355 // Possibly followed by another hex digit, but again we don't care. 356 } else { 357 AddError("Invalid escape sequence in string literal."); 358 } 359 break; 360 } 361 362 default: { 363 if (current_char_ == delimiter) { 364 NextChar(); 365 return; 366 } 367 NextChar(); 368 break; 369 } 370 } 371 } 372} 373 374Tokenizer::TokenType Tokenizer::ConsumeNumber(bool started_with_zero, 375 bool started_with_dot) { 376 bool is_float = false; 377 378 if (started_with_zero && (TryConsume('x') || TryConsume('X'))) { 379 // A hex number (started with "0x"). 380 ConsumeOneOrMore<HexDigit>("\"0x\" must be followed by hex digits."); 381 382 } else if (started_with_zero && LookingAt<Digit>()) { 383 // An octal number (had a leading zero). 384 ConsumeZeroOrMore<OctalDigit>(); 385 if (LookingAt<Digit>()) { 386 AddError("Numbers starting with leading zero must be in octal."); 387 ConsumeZeroOrMore<Digit>(); 388 } 389 390 } else { 391 // A decimal number. 392 if (started_with_dot) { 393 is_float = true; 394 ConsumeZeroOrMore<Digit>(); 395 } else { 396 ConsumeZeroOrMore<Digit>(); 397 398 if (TryConsume('.')) { 399 is_float = true; 400 ConsumeZeroOrMore<Digit>(); 401 } 402 } 403 404 if (TryConsume('e') || TryConsume('E')) { 405 is_float = true; 406 TryConsume('-') || TryConsume('+'); 407 ConsumeOneOrMore<Digit>("\"e\" must be followed by exponent."); 408 } 409 410 if (allow_f_after_float_ && (TryConsume('f') || TryConsume('F'))) { 411 is_float = true; 412 } 413 } 414 415 if (LookingAt<Letter>()) { 416 AddError("Need space between number and identifier."); 417 } else if (current_char_ == '.') { 418 if (is_float) { 419 AddError( 420 "Already saw decimal point or exponent; can't have another one."); 421 } else { 422 AddError("Hex and octal numbers must be integers."); 423 } 424 } 425 426 return is_float ? TYPE_FLOAT : TYPE_INTEGER; 427} 428 429void Tokenizer::ConsumeLineComment() { 430 while (current_char_ != '\0' && current_char_ != '\n') { 431 NextChar(); 432 } 433 TryConsume('\n'); 434} 435 436void Tokenizer::ConsumeBlockComment() { 437 int start_line = line_; 438 int start_column = column_ - 2; 439 440 while (true) { 441 while (current_char_ != '\0' && 442 current_char_ != '*' && 443 current_char_ != '/') { 444 NextChar(); 445 } 446 447 if (TryConsume('*') && TryConsume('/')) { 448 // End of comment. 449 break; 450 } else if (TryConsume('/') && current_char_ == '*') { 451 // Note: We didn't consume the '*' because if there is a '/' after it 452 // we want to interpret that as the end of the comment. 453 AddError( 454 "\"/*\" inside block comment. Block comments cannot be nested."); 455 } else if (current_char_ == '\0') { 456 AddError("End-of-file inside block comment."); 457 error_collector_->AddError( 458 start_line, start_column, " Comment started here."); 459 break; 460 } 461 } 462} 463 464// ------------------------------------------------------------------- 465 466bool Tokenizer::Next() { 467 previous_ = current_; 468 469 // Did we skip any characters after the last token? 470 bool skipped_stuff = false; 471 472 while (!read_error_) { 473 if (TryConsumeOne<Whitespace>()) { 474 ConsumeZeroOrMore<Whitespace>(); 475 476 } else if (comment_style_ == CPP_COMMENT_STYLE && TryConsume('/')) { 477 // Starting a comment? 478 if (TryConsume('/')) { 479 ConsumeLineComment(); 480 } else if (TryConsume('*')) { 481 ConsumeBlockComment(); 482 } else { 483 // Oops, it was just a slash. Return it. 484 current_.type = TYPE_SYMBOL; 485 current_.text = "/"; 486 current_.line = line_; 487 current_.column = column_ - 1; 488 return true; 489 } 490 491 } else if (comment_style_ == SH_COMMENT_STYLE && TryConsume('#')) { 492 ConsumeLineComment(); 493 494 } else if (LookingAt<Unprintable>() || current_char_ == '\0') { 495 AddError("Invalid control characters encountered in text."); 496 NextChar(); 497 // Skip more unprintable characters, too. But, remember that '\0' is 498 // also what current_char_ is set to after EOF / read error. We have 499 // to be careful not to go into an infinite loop of trying to consume 500 // it, so make sure to check read_error_ explicitly before consuming 501 // '\0'. 502 while (TryConsumeOne<Unprintable>() || 503 (!read_error_ && TryConsume('\0'))) { 504 // Ignore. 505 } 506 507 } else { 508 // Reading some sort of token. 509 StartToken(); 510 511 if (TryConsumeOne<Letter>()) { 512 ConsumeZeroOrMore<Alphanumeric>(); 513 current_.type = TYPE_IDENTIFIER; 514 } else if (TryConsume('0')) { 515 current_.type = ConsumeNumber(true, false); 516 } else if (TryConsume('.')) { 517 // This could be the beginning of a floating-point number, or it could 518 // just be a '.' symbol. 519 520 if (TryConsumeOne<Digit>()) { 521 // It's a floating-point number. 522 if (previous_.type == TYPE_IDENTIFIER && !skipped_stuff) { 523 // We don't accept syntax like "blah.123". 524 error_collector_->AddError(line_, column_ - 2, 525 "Need space between identifier and decimal point."); 526 } 527 current_.type = ConsumeNumber(false, true); 528 } else { 529 current_.type = TYPE_SYMBOL; 530 } 531 } else if (TryConsumeOne<Digit>()) { 532 current_.type = ConsumeNumber(false, false); 533 } else if (TryConsume('\"')) { 534 ConsumeString('\"'); 535 current_.type = TYPE_STRING; 536 } else if (TryConsume('\'')) { 537 ConsumeString('\''); 538 current_.type = TYPE_STRING; 539 } else { 540 NextChar(); 541 current_.type = TYPE_SYMBOL; 542 } 543 544 EndToken(); 545 return true; 546 } 547 548 skipped_stuff = true; 549 } 550 551 // EOF 552 current_.type = TYPE_END; 553 current_.text.clear(); 554 current_.line = line_; 555 current_.column = column_; 556 current_.end_column = column_; 557 return false; 558} 559 560// ------------------------------------------------------------------- 561// Token-parsing helpers. Remember that these don't need to report 562// errors since any errors should already have been reported while 563// tokenizing. Also, these can assume that whatever text they 564// are given is text that the tokenizer actually parsed as a token 565// of the given type. 566 567bool Tokenizer::ParseInteger(const string& text, uint64 max_value, 568 uint64* output) { 569 // Sadly, we can't just use strtoul() since it is only 32-bit and strtoull() 570 // is non-standard. I hate the C standard library. :( 571 572// return strtoull(text.c_str(), NULL, 0); 573 574 const char* ptr = text.c_str(); 575 int base = 10; 576 if (ptr[0] == '0') { 577 if (ptr[1] == 'x' || ptr[1] == 'X') { 578 // This is hex. 579 base = 16; 580 ptr += 2; 581 } else { 582 // This is octal. 583 base = 8; 584 } 585 } 586 587 uint64 result = 0; 588 for (; *ptr != '\0'; ptr++) { 589 int digit = DigitValue(*ptr); 590 GOOGLE_LOG_IF(DFATAL, digit < 0 || digit >= base) 591 << " Tokenizer::ParseInteger() passed text that could not have been" 592 " tokenized as an integer: " << CEscape(text); 593 if (digit > max_value || result > (max_value - digit) / base) { 594 // Overflow. 595 return false; 596 } 597 result = result * base + digit; 598 } 599 600 *output = result; 601 return true; 602} 603 604double Tokenizer::ParseFloat(const string& text) { 605 const char* start = text.c_str(); 606 char* end; 607 double result = NoLocaleStrtod(start, &end); 608 609 // "1e" is not a valid float, but if the tokenizer reads it, it will 610 // report an error but still return it as a valid token. We need to 611 // accept anything the tokenizer could possibly return, error or not. 612 if (*end == 'e' || *end == 'E') { 613 ++end; 614 if (*end == '-' || *end == '+') ++end; 615 } 616 617 // If the Tokenizer had allow_f_after_float_ enabled, the float may be 618 // suffixed with the letter 'f'. 619 if (*end == 'f' || *end == 'F') { 620 ++end; 621 } 622 623 GOOGLE_LOG_IF(DFATAL, end - start != text.size() || *start == '-') 624 << " Tokenizer::ParseFloat() passed text that could not have been" 625 " tokenized as a float: " << CEscape(text); 626 return result; 627} 628 629void Tokenizer::ParseStringAppend(const string& text, string* output) { 630 // Reminder: text[0] is always the quote character. (If text is 631 // empty, it's invalid, so we'll just return.) 632 if (text.empty()) { 633 GOOGLE_LOG(DFATAL) 634 << " Tokenizer::ParseStringAppend() passed text that could not" 635 " have been tokenized as a string: " << CEscape(text); 636 return; 637 } 638 639 output->reserve(output->size() + text.size()); 640 641 // Loop through the string copying characters to "output" and 642 // interpreting escape sequences. Note that any invalid escape 643 // sequences or other errors were already reported while tokenizing. 644 // In this case we do not need to produce valid results. 645 for (const char* ptr = text.c_str() + 1; *ptr != '\0'; ptr++) { 646 if (*ptr == '\\' && ptr[1] != '\0') { 647 // An escape sequence. 648 ++ptr; 649 650 if (OctalDigit::InClass(*ptr)) { 651 // An octal escape. May one, two, or three digits. 652 int code = DigitValue(*ptr); 653 if (OctalDigit::InClass(ptr[1])) { 654 ++ptr; 655 code = code * 8 + DigitValue(*ptr); 656 } 657 if (OctalDigit::InClass(ptr[1])) { 658 ++ptr; 659 code = code * 8 + DigitValue(*ptr); 660 } 661 output->push_back(static_cast<char>(code)); 662 663 } else if (*ptr == 'x') { 664 // A hex escape. May zero, one, or two digits. (The zero case 665 // will have been caught as an error earlier.) 666 int code = 0; 667 if (HexDigit::InClass(ptr[1])) { 668 ++ptr; 669 code = DigitValue(*ptr); 670 } 671 if (HexDigit::InClass(ptr[1])) { 672 ++ptr; 673 code = code * 16 + DigitValue(*ptr); 674 } 675 output->push_back(static_cast<char>(code)); 676 677 } else { 678 // Some other escape code. 679 output->push_back(TranslateEscape(*ptr)); 680 } 681 682 } else if (*ptr == text[0]) { 683 // Ignore quote matching the starting quote. 684 } else { 685 output->push_back(*ptr); 686 } 687 } 688 689 return; 690} 691 692} // namespace io 693} // namespace protobuf 694} // namespace google