1# Macro expansion23Rust has a very powerful macro system.4In the previous chapter, we saw how5the parser sets aside macros to be expanded (using temporary [placeholders]).6This chapter is about the process of expanding those macros iteratively until7we have a complete [*Abstract Syntax Tree* (AST)][ast] for our crate with no8unexpanded macros (or a compile error).910[ast]: ./ast-validation.md11[placeholders]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/placeholders/index.html1213First, we discuss the algorithm that expands and integrates macro output into ASTs.14Next, we take a look at how hygiene data is collected.15Finally, we look at the specifics of expanding different types of macros.1617Many of the algorithms and data structures described below are in [`rustc_expand`],18with fundamental data structures in [`rustc_expand::base`][base].1920Also of note, `cfg` and `cfg_attr` are treated specially from other macros, and are21handled in [`rustc_expand::config`][cfg].2223[`rustc_expand`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/index.html24[base]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/index.html25[cfg]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/config/index.html2627## Expansion and AST Integration2829Firstly, expansion happens at the crate level.30Given a raw source code for31a crate, the compiler will produce a massive AST with all macros expanded, all32modules inlined, etc. The primary entry point for this process is the33[`MacroExpander::fully_expand_fragment`][fef] method.34With few exceptions, we35use this method on the whole crate (see ["Eager Expansion"](#eager-expansion)36below for more detailed discussion of edge case expansion issues).3738[`rustc_builtin_macros`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_builtin_macros/index.html39[reb]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/build/index.html4041At a high level, [`fully_expand_fragment`][fef] works in iterations.42We keep a43queue of unresolved macro invocations (i.e. macros we haven't found the44definition of yet).45We repeatedly try to pick a macro from the queue, resolve it, expand it, and integrate it back.46If we can't make progress in an iteration, this represents a compile error.47 Here is the [algorithm][original]:4849[fef]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/expand/struct.MacroExpander.html#method.fully_expand_fragment50[original]: https://github.com/rust-lang/rust/pull/53778#issuecomment-41922404951521. Initialize a `queue` of unresolved macros.532. Repeat until `queue` is empty (or we make no progress, which is an error):54   1. [Resolve](./name-resolution.md) imports in our partially built crate as55      much as possible.56   2. Collect as many macro [`Invocation`s][inv] as possible from our57      partially built crate (`fn`-like, attributes, derives) and add them to the queue.58   3. Dequeue the first element and attempt to resolve it.59   4. If it's resolved:60      1. Run the macro's expander function that consumes a [`TokenStream`] or61         AST and produces a [`TokenStream`] or [`AstFragment`] (depending on62         the macro kind).63         (A [`TokenStream`] is a collection of [`TokenTree`s][tt],64         each of which are a token (punctuation, identifier, or literal) or a65         delimited group (anything inside `()`/`[]`/`{}`)).66         - At this point, we know everything about the macro itself and can67           call [`set_expn_data`] to fill in its properties in the global68           data; that is the [hygiene] data associated with [`ExpnId`] (see69           [Hygiene][hybelow] below).70      2. Integrate that piece of AST into the currently-existing though71         partially-built AST.72         This is essentially where the "token-like mass"73         becomes a proper set-in-stone AST with side-tables.74         It happens as follows:75         - If the macro produces tokens (e.g. a proc macro), we parse into76           an AST, which may produce parse errors.77         - During expansion, we create [`SyntaxContext`]s (hierarchy 2) (see78           [Hygiene][hybelow] below).79         - These three passes happen one after another on every AST fragment80           freshly expanded from a macro:81           - [`NodeId`]s are assigned by [`InvocationCollector`].82             This also collects new macro calls from this new AST piece and83             adds them to the queue.84           - ["Def paths"][defpath] are created and [`DefId`]s are85             assigned to them by [`DefCollector`].86           - Names are put into modules (from the resolver's point of87             view) by [`BuildReducedGraphVisitor`].88      3. After expanding a single macro and integrating its output, continue89         to the next iteration of [`fully_expand_fragment`][fef].90   5. If it's not resolved:91      1. Put the macro back in the queue.92      2. Continue to next iteration...9394[`AstFragment`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/expand/enum.AstFragment.html95[`BuildReducedGraphVisitor`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/build_reduced_graph/struct.BuildReducedGraphVisitor.html96[`DefCollector`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/def_collector/struct.DefCollector.html97[`DefId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/def_id/struct.DefId.html98[`ExpnId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnId.html99[`InvocationCollector`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/expand/struct.InvocationCollector.html100[`NodeId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/node_id/struct.NodeId.html101[`set_expn_data`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.LocalExpnId.html#method.set_expn_data102[`SyntaxContext`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html103[`TokenStream`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/tokenstream/struct.TokenStream.html104[defpath]: hir.md#identifiers-in-the-hir105[hybelow]: #hygiene-and-hierarchies106[hygiene]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/index.html107[inv]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/expand/struct.Invocation.html108[tt]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/tokenstream/enum.TokenTree.html109110### Error Recovery111112If we make no progress in an iteration we have reached a compilation error113(e.g. an undefined macro). We attempt to recover from failures (i.e.114unresolved macros or imports) with the intent of generating diagnostics.115Failure recovery happens by expanding unresolved macros into116[`ExprKind::Err`][err] and allows compilation to continue past the first error117so that `rustc` can report more errors than just the original failure.118119[err]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/ast/enum.ExprKind.html#variant.Err120121### Name Resolution122123Notice that name resolution is involved here: we need to resolve imports and124macro names in the above algorithm.125This is done in [`rustc_resolve::macros`][mresolve], which resolves macro paths, validates126those resolutions, and reports various errors (e.g. "not found", "found, but127it's unstable", "expected x, found y").128However, we don't try to resolve other names yet.129This happens later, as we will see in the chapter: [Name Resolution](./name-resolution.md).130131[mresolve]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/macros/index.html132133### Eager Expansion134135_Eager expansion_ means we expand the arguments of a macro invocation before136the macro invocation itself.137This is implemented only for a few special138built-in macros that expect literals; expanding arguments first for some of139these macro results in a smoother user experience.140As an example, consider the following:141142```rust,ignore143macro bar($i: ident) { $i }144macro foo($i: ident) { $i }145146foo!(bar!(baz));147```148149A lazy-expansion would expand `foo!` first.150An eager-expansion would expand `bar!` first.151152Eager-expansion is not a generally available feature of Rust.153Implementing eager-expansion more generally would be challenging, so we implement it for a154few special built-in macros for the sake of user-experience.155The built-in macros are implemented in [`rustc_builtin_macros`], along with some other156early code generation facilities like injection of standard library imports or157generation of test harness.158There are some additional helpers for building AST fragments in [`rustc_expand::build`][reb].159Eager-expansion generally performs a subset of the things that lazy (normal) expansion does.160It is done by invoking [`fully_expand_fragment`][fef] on only part of a crate (as opposed161to the whole crate, like we normally do).162163### Other Data Structures164165Here are some other notable data structures involved in expansion and integration:166- [`ResolverExpand`] - a `trait` used to break crate dependencies.167  This allows the resolver services to be used in [`rustc_ast`], despite [`rustc_resolve`] and168  pretty much everything else depending on [`rustc_ast`].169- [`ExtCtxt`]/[`ExpansionData`] - holds various intermediate expansion infrastructure data.170- [`Annotatable`] - a piece of AST that can be an attribute target, almost the same171  thing as [`AstFragment`] except for types and patterns that can be produced by172  macros but cannot be annotated with attributes.173- [`MacResult`] - a "polymorphic" AST fragment, something that can turn into174  a different [`AstFragment`] depending on its [`AstFragmentKind`] (i.e. an item,175  expression, pattern, etc).176177[`AstFragment`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/expand/enum.AstFragment.html178[`rustc_ast`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/index.html179[`rustc_resolve`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/index.html180[`ResolverExpand`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/trait.ResolverExpand.html181[`ExtCtxt`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/struct.ExtCtxt.html182[`ExpansionData`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/struct.ExpansionData.html183[`Annotatable`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/enum.Annotatable.html184[`MacResult`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/trait.MacResult.html185[`AstFragmentKind`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/expand/enum.AstFragmentKind.html186187## Hygiene and Hierarchies188189If you have ever used the C/C++ preprocessor macros, you know that there are some190annoying and hard-to-debug gotchas!191For example, consider the following C code:192193```c194#define DEFINE_FOO struct Bar {int x;}; struct Foo {Bar bar;};195196// Then, somewhere else197struct Bar {198    ...199};200201DEFINE_FOO202```203204Most people avoid writing C like this – and for good reason: it doesn't compile.205The `struct Bar` defined by the macro clashes names with the `struct Bar` defined in the code.206Consider also the following example:207208```c209#define DO_FOO(x) {\210    int y = 0;\211    foo(x, y);\212    }213214// Then elsewhere215int y = 22;216DO_FOO(y);217```218219Do you see the problem?220We wanted to generate a call `foo(22, 0)`, but instead221we got `foo(0, 0)` because the macro defined its own `y`!222223These are both examples of _macro hygiene_ issues.224_Hygiene_ relates to how to handle names defined _within a macro_.225In particular, a hygienic macro system prevents errors due to names introduced within a macro.226Rust macros are hygienic in that they do not allow one to write the sorts of bugs above.227228At a high level, hygiene within the Rust compiler is accomplished by keeping229track of the context where a name is introduced and used.230We can then disambiguate names based on that context.231Future iterations of the macro system232will allow greater control to the macro author to use that context.233For example,234a macro author may want to introduce a new name to the context where the macro was called.235Alternately, the macro author may be defining a variable for use236only within the macro (i.e. it should not be visible outside the macro).237238[code_dir]: https://github.com/rust-lang/rust/tree/HEAD/compiler/rustc_expand/src/mbe239[code_mp]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/mbe/macro_parser240[code_mr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/mbe/macro_rules241[code_parse_int]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/mbe/macro_parser/struct.TtParser.html#method.parse_tt242[parsing]: ./the-parser.html243244The context is attached to AST nodes.245All AST nodes generated by macros have context attached.246Additionally, there may be other nodes that have context247attached, such as some desugared syntax (non-macro-expanded nodes are248considered to just have the "root" context, as described below).249Throughout the compiler, we use [`rustc_span::Span`s][span] to refer to code locations.250This struct also has hygiene information attached to it, as we will see later.251252[span]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html253254Because macros invocations and definitions can be nested, the syntax context of255a node must be a hierarchy.256For example, if we expand a macro and there is257another macro invocation or definition in the generated output, then the syntax258context should reflect the nesting.259260However, it turns out that there are actually a few types of context we may261want to track for different purposes.262Thus, there are not just one but _three_263expansion hierarchies that together comprise the hygiene information for a crate.264265All of these hierarchies need some sort of "macro ID" to identify individual266elements in the chain of expansions.267This ID is [`ExpnId`].268All macros receive an integer ID, assigned continuously starting from 0 as we discover new macro269calls.270All hierarchies start at [`ExpnId::root`][rootid], which is its own parent.271272The [`rustc_span::hygiene`][hy] crate contains all of the hygiene-related algorithms273(with the exception of some hacks in [`Resolver::resolve_crate_root`][hacks])274and structures related to hygiene and expansion that are kept in global data.275276The actual hierarchies are stored in [`HygieneData`][hd].277This is a global piece of data containing hygiene and expansion info that can be accessed from278any [`Ident`] without any context.279280281[`ExpnId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnId.html282[rootid]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnId.html#method.root283[hd]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.HygieneData.html284[hy]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/index.html285[hacks]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Resolver.html#method.resolve_crate_root286[`Ident`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/symbol/struct.Ident.html287288### The Expansion Order Hierarchy289290The first hierarchy tracks the order of expansions, i.e., when a macro291invocation is in the output of another macro.292293Here, the children in the hierarchy will be the "innermost" tokens.294The [`ExpnData`] struct itself contains a subset of properties from both macro295definition and macro call available through global data.296[`ExpnData::parent`][edp] tracks the child-to-parent link in this hierarchy.297298[`ExpnData`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnData.html299[edp]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnData.html#structfield.parent300301For example:302303```rust,ignore304macro_rules! foo { () => { println!(); } }305306fn main() { foo!(); }307```308309In this code, the AST nodes that are finally generated would have hierarchy310`root -> id(foo) -> id(println)`.311312### The Macro Definition Hierarchy313314The second hierarchy tracks the order of macro definitions, i.e., when we are315expanding one macro another macro definition is revealed in its output.316This one is a bit tricky and more complex than the other two hierarchies.317318[`SyntaxContext`][sc] represents a whole chain in this hierarchy via an ID.319[`SyntaxContextData`][scd] contains data associated with the given320[`SyntaxContext`][sc]; mostly it is a cache for results of filtering that chain in different ways.321 [`SyntaxContextData::parent`][scdp] is the child-to-parent322link here, and [`SyntaxContextData::outer_expns`][scdoe] are individual elements in the chain.323The "chaining-operator" is [`SyntaxContext::apply_mark`][am] in compiler code.324325A [`Span`][span], mentioned above, is actually just a compact representation of326a code location and [`SyntaxContext`][sc].327Likewise, an [`Ident`] is just an interned328[`Symbol`] + `Span` (i.e. an interned string + hygiene data).329330[`Symbol`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/symbol/struct.Symbol.html331[scd]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContextData.html332[scdp]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContextData.html#structfield.parent333[sc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html334[scdoe]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContextData.html#structfield.outer_expn335[am]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html#method.apply_mark336337For built-in macros, we use the context:338[`SyntaxContext::empty().apply_mark(expn_id)`], and such macros are339considered to be defined at the hierarchy root.340We do the same for `proc macro`s because we haven't implemented cross-crate hygiene yet.341342[`SyntaxContext::empty().apply_mark(expn_id)`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html#method.apply_mark343344If the token had context `X` before being produced by a macro then after being345produced by the macro it has context `X -> macro_id`.346Here are some examples:347348Example 0:349350```rust,ignore351macro m() { ident }352353m!();354```355356Here `ident` which initially has context [`SyntaxContext::root`][scr] has357context `ROOT -> id(m)` after it's produced by `m`.358359[scr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html#method.root360361Example 1:362363```rust,ignore364macro m() { macro n() { ident } }365366m!();367n!();368```369370In this example the `ident` has context `ROOT` initially, then `ROOT -> id(m)`371after the first expansion, then `ROOT -> id(m) -> id(n)`.372373Example 2:374375Note that these chains are not entirely determined by their last element, in376other words [`ExpnId`] is not isomorphic to [`SyntaxContext`][sc].377378```rust,ignore379macro m($i: ident) { macro n() { ($i, bar) } }380381m!(foo);382```383384After all expansions, `foo` has context `ROOT -> id(n)` and `bar` has context385`ROOT -> id(m) -> id(n)`.386387Currently this hierarchy for tracking macro definitions is subject to the388so-called ["context transplantation hack"][hack]. Modern (i.e. experimental)389macros have stronger hygiene than the legacy "Macros By Example" (MBE)390system which can result in weird interactions between the two.391The hack is intended to make things "just work" for now.392393[`ExpnId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnId.html394[hack]: https://github.com/rust-lang/rust/pull/51762#issuecomment-401400732395396### The Call-site Hierarchy397398The third and final hierarchy tracks the location of macro invocations.399400In this hierarchy [`ExpnData::call_site`][callsite] is the `child -> parent` link.401402[callsite]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnData.html#structfield.call_site403404Here is an example:405406```rust,ignore407macro bar($i: ident) { $i }408macro foo($i: ident) { $i }409410foo!(bar!(baz));411```412413For the `baz` AST node in the final output, the expansion-order hierarchy is414`ROOT -> id(foo) -> id(bar) -> baz`, while the call-site hierarchy is `ROOT -> baz`.415416### Macro Backtraces417418Macro backtraces are implemented in [`rustc_span`] using the hygiene machinery419in [`rustc_span::hygiene`][hy].420421[`rustc_span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/index.html422423## Producing Macro Output424425Above, we saw how the output of a macro is integrated into the AST for a crate,426and we also saw how the hygiene data for a crate is generated.427But how do we actually produce the output of a macro?428It depends on the type of macro.429430There are two types of macros in Rust:431  1. `macro_rules!` macros (a.k.a.432     "Macros By Example" (MBE)), and,433  2. procedural macros (proc macros); including custom derives.434435During the parsing phase, the normal Rust parser will set aside the contents of436macros and their invocations.437Later, macros are expanded using these portions of the code.438439Some important data structures/interfaces here:440- [`SyntaxExtension`] - a lowered macro representation, contains its expander441  function, which transforms a [`TokenStream`] or AST into another442  [`TokenStream`] or AST + some additional data like stability, or a list of443  unstable features allowed inside the macro.444- [`SyntaxExtensionKind`] - expander functions may have several different445  signatures (take one token stream, or two, or a piece of AST, etc).446  This is an `enum` that lists them.447- [`BangProcMacro`]/[`TTMacroExpander`]/[`AttrProcMacro`]/[`MultiItemModifier`] -448  `trait`s representing the expander function signatures.449450[`SyntaxExtension`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/struct.SyntaxExtension.html451[`SyntaxExtensionKind`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/enum.SyntaxExtensionKind.html452[`BangProcMacro`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/trait.BangProcMacro.html453[`TTMacroExpander`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/trait.TTMacroExpander.html454[`AttrProcMacro`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/trait.AttrProcMacro.html455[`MultiItemModifier`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/base/trait.MultiItemModifier.html456457## Macros By Example458459MBEs have their own parser distinct from the Rust parser.460When macros are expanded, we may invoke the MBE parser to parse and expand a macro.461 The MBE parser, in turn, may call the Rust parser when it needs to bind a462metavariable (e.g. `$my_expr`) while parsing the contents of a macro463invocation.464The code for macro expansion is in [`compiler/rustc_expand/src/mbe/`][code_dir].465466### Example467468```rust,ignore469macro_rules! printer {470    (print $mvar:ident) => {471        println!("{}", $mvar);472    };473    (print twice $mvar:ident) => {474        println!("{}", $mvar);475        println!("{}", $mvar);476    };477}478```479480Here `$mvar` is called a _metavariable_.481Unlike normal variables, rather than482binding to a value _at runtime_, a metavariable binds _at compile time_ to a tree of _tokens_.483A _token_ is a single "unit" of the grammar, such as an484identifier (e.g. `foo`) or punctuation (e.g. `=>`). There are also other485special tokens, such as `EOF`, which itself indicates that there are no more tokens.486There are token trees resulting from the paired parentheses-like487characters (`(`...`)`, `[`...`]`, and `{`...`}`) – they include the open and488close and all the tokens in between (Rust requires that parentheses-like characters be balanced).489Having macro expansion operate on token streams490rather than the raw bytes of a source-file abstracts away a lot of complexity.491The macro expander (and much of the rest of the compiler) doesn't consider492the exact line and column of some syntactic construct in the code; it considers493which constructs are used in the code.494Using tokens allows us to care about _what_ without worrying about _where_.495For more information about tokens, see the [Parsing][parsing] chapter of this book.496497```rust,ignore498printer!(print foo); // `foo` is a variable499```500501The process of expanding the macro invocation into the syntax tree502`println!("{}", foo)` and then expanding the syntax tree into a call to503`Display::fmt` is one common example of _macro expansion_.504505### The MBE parser506507There are two parts to MBE expansion done by the macro parser:508  1. parsing the definition, and,509  2. parsing the invocations.510511We think of the MBE parser as a nondeterministic finite automaton (NFA) based512regex parser since it uses an algorithm similar in spirit to the [Earley513parsing algorithm](https://en.wikipedia.org/wiki/Earley_parser).514The macro parser is defined in [`compiler/rustc_expand/src/mbe/macro_parser.rs`][code_mp].515516The interface of the macro parser is as follows (this is slightly simplified):517518```rust,ignore519fn parse_tt(520    &mut self,521    parser: &mut Cow<'_, Parser<'_>>,522    matcher: &[MatcherLoc]523) -> ParseResult524```525526We use these items in macro parser:527528- a `parser` variable is a reference to the state of a normal Rust parser,529  including the token stream and parsing session.530  The token stream is what we are about to ask the MBE parser to parse.531  We will consume the raw stream of532  tokens and output a binding of metavariables to corresponding token trees.533  The parsing session can be used to report parser errors.534- a `matcher` variable is a sequence of [`MatcherLoc`]s that we want to match the token stream535  against.536  They're converted from the original token trees in the macro's definition before matching.537538[`MatcherLoc`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/mbe/macro_parser/enum.MatcherLoc.html539540In the analogy of a regex parser, the token stream is the input and we are541matching it against the pattern defined by matcher.542Using our examples, the543token stream could be the stream of tokens containing the inside of the example544invocation `print foo`, while matcher might be the sequence of token (trees) `print $mvar:ident`.545546The output of the parser is a [`ParseResult`], which indicates which of three cases has occurred:547548- **Success**: the token stream matches the given matcher and we have produced a549  binding from metavariables to the corresponding token trees.550- **Failure**: the token stream does not match matcher and results in an error551  message such as "No rule expected token ...".552- **Error**: some fatal error has occurred _in the parser_.553  For example, this happens if there is more than one pattern match, since that indicates the554  macro is ambiguous.555556The full interface is defined [here][code_parse_int].557558The macro parser does pretty much exactly the same as a normal regex parser559with one exception: in order to parse different types of metavariables, such as560`ident`, `block`, `expr`, etc., the macro parser must call back to the normal561Rust parser.562563The code to parse macro definitions is in [`compiler/rustc_expand/src/mbe/macro_rules.rs`][code_mr].564For more information about the macro parser's implementation, see the comments in565[`compiler/rustc_expand/src/mbe/macro_parser.rs`][code_mp].566567Using our example, we would try to match the token stream `print foo` from the invocation against568the matchers `print $mvar:ident` and `print twice $mvar:ident` that we previously extracted from the569rules in the macro definition.570When the macro parser comes to a place in the current matcher where571it needs to match a _non-terminal_ (e.g. `$mvar:ident`), it calls back to the normal Rust parser to572get the contents of that non-terminal.573In this case, the Rust parser would look for an `ident`574token, which it finds (`foo`) and returns to the macro parser.575Then, the macro parser continues parsing.576577Note that exactly one of the matchers from the various rules should match the invocation; if there is578more than one match, the parse is ambiguous, while if there are no matches at all, there is a syntax579error.580581Assuming exactly one rule matches, macro expansion will then *transcribe* the right-hand side of the582rule, substituting the values of any matches it captured when matching against the left-hand side.583584## Procedural Macros585586Procedural macros are also expanded during parsing.587However, rather than having a parser in the compiler, proc macros are implemented as custom,588third-party crates.589The compiler will compile the proc macro crate and590specially annotated functions in them (i.e. the proc macro itself), passing591them a stream of tokens.592A proc macro can then transform the token stream and593output a new token stream, which is synthesized into the AST.594595The token stream type used by proc macros is _stable_, so `rustc` does not use it internally.596The compiler's (unstable) token stream is defined in597[`rustc_ast::tokenstream::TokenStream`][rustcts].598This is converted into the stable [`proc_macro::TokenStream`][stablets] and back in599[`rustc_expand::proc_macro`][pm] and [`rustc_expand::proc_macro_server`][pms].600Since the Rust ABI is currently unstable, we use the C ABI for this conversion.601602[tsmod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/tokenstream/index.html603[rustcts]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/tokenstream/struct.TokenStream.html604[stablets]: https://doc.rust-lang.org/proc_macro/struct.TokenStream.html605[pm]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/proc_macro/index.html606[pms]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/proc_macro_server/index.html607[`ParseResult`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/mbe/macro_parser/enum.ParseResult.html608609<!-- TODO(rylev): more here. [#1160](https://github.com/rust-lang/rustc-dev-guide/issues/1160) -->610611### Custom Derive612613Custom derives are a special type of proc macro.614615### Macros By Example and Macros 2.0616617There is an legacy and mostly undocumented effort to improve the MBE system618by giving it more hygiene-related features, better scoping and visibility619rules, etc. Internally this uses the same machinery as today's MBEs with some620additional syntactic sugar and are allowed to be in namespaces.621622<!-- TODO(rylev): more? [#1160](https://github.com/rust-lang/rustc-dev-guide/issues/1160) -->