1//! Mono Item Collection2//! ====================3//!4//! This module is responsible for discovering all items that will contribute5//! to code generation of the crate. The important part here is that it not only6//! needs to find syntax-level items (functions, structs, etc) but also all7//! their monomorphized instantiations. Every non-generic, non-const function8//! maps to one LLVM artifact. Every generic function can produce9//! from zero to N artifacts, depending on the sets of type arguments it10//! is instantiated with.11//! This also applies to generic items from other crates: A generic definition12//! in crate X might produce monomorphizations that are compiled into crate Y.13//! We also have to collect these here.14//!15//! The following kinds of "mono items" are handled here:16//!17//! - Functions18//! - Methods19//! - Closures20//! - Statics21//! - Drop glue22//!23//! The following things also result in LLVM artifacts, but are not collected24//! here, since we instantiate them locally on demand when needed in a given25//! codegen unit:26//!27//! - Constants28//! - VTables29//! - Object Shims30//!31//! The main entry point is `collect_crate_mono_items`, at the bottom of this file.32//!33//! General Algorithm34//! -----------------35//! Let's define some terms first:36//!37//! - A "mono item" is something that results in a function or global in38//!   the LLVM IR of a codegen unit. Mono items do not stand on their39//!   own, they can use other mono items. For example, if function40//!   `foo()` calls function `bar()` then the mono item for `foo()`41//!   uses the mono item for function `bar()`. In general, the42//!   definition for mono item A using a mono item B is that43//!   the LLVM artifact produced for A uses the LLVM artifact produced44//!   for B.45//!46//! - Mono items and the uses between them form a directed graph,47//!   where the mono items are the nodes and uses form the edges.48//!   Let's call this graph the "mono item graph".49//!50//! - The mono item graph for a program contains all mono items51//!   that are needed in order to produce the complete LLVM IR of the program.52//!53//! The purpose of the algorithm implemented in this module is to build the54//! mono item graph for the current crate. It runs in two phases:55//!56//! 1. Discover the roots of the graph by traversing the HIR of the crate.57//! 2. Starting from the roots, find uses by inspecting the MIR58//!    representation of the item corresponding to a given node, until no more59//!    new nodes are found.60//!61//! ### Discovering roots62//! The roots of the mono item graph correspond to the public non-generic63//! syntactic items in the source code. We find them by walking the HIR of the64//! crate, and whenever we hit upon a public function, method, or static item,65//! we create a mono item consisting of the items DefId and, since we only66//! consider non-generic items, an empty type-parameters set. (In eager67//! collection mode, during incremental compilation, all non-generic functions68//! are considered as roots, as well as when the `-Clink-dead-code` option is69//! specified. Functions marked `#[no_mangle]` and functions called by inlinable70//! functions also always act as roots.)71//!72//! ### Finding uses73//! Given a mono item node, we can discover uses by inspecting its MIR. We walk74//! the MIR to find other mono items used by each mono item. Since the mono75//! item we are currently at is always monomorphic, we also know the concrete76//! type arguments of its used mono items. The specific forms a use can take in77//! MIR are quite diverse. Here is an overview:78//!79//! #### Calling Functions/Methods80//! The most obvious way for one mono item to use another is a81//! function or method call (represented by a CALL terminator in MIR). But82//! calls are not the only thing that might introduce a use between two83//! function mono items, and as we will see below, they are just a84//! specialization of the form described next, and consequently will not get any85//! special treatment in the algorithm.86//!87//! #### Taking a reference to a function or method88//! A function does not need to actually be called in order to be used by89//! another function. It suffices to just take a reference in order to introduce90//! an edge. Consider the following example:91//!92//! ```93//! # use core::fmt::Display;94//! fn print_val<T: Display>(x: T) {95//!     println!("{}", x);96//! }97//!98//! fn call_fn(f: &dyn Fn(i32), x: i32) {99//!     f(x);100//! }101//!102//! fn main() {103//!     let print_i32 = print_val::<i32>;104//!     call_fn(&print_i32, 0);105//! }106//! ```107//! The MIR of none of these functions will contain an explicit call to108//! `print_val::<i32>`. Nonetheless, in order to mono this program, we need109//! an instance of this function. Thus, whenever we encounter a function or110//! method in operand position, we treat it as a use of the current111//! mono item. Calls are just a special case of that.112//!113//! #### Drop glue114//! Drop glue mono items are introduced by MIR drop-statements. The115//! generated mono item will have additional drop-glue item uses if the116//! type to be dropped contains nested values that also need to be dropped. It117//! might also have a function item use for the explicit `Drop::drop`118//! implementation of its type.119//!120//! #### Unsizing Casts121//! A subtle way of introducing use edges is by casting to a trait object.122//! Since the resulting wide-pointer contains a reference to a vtable, we need to123//! instantiate all dyn-compatible methods of the trait, as we need to store124//! pointers to these functions even if they never get called anywhere. This can125//! be seen as a special case of taking a function reference.126//!127//!128//! Interaction with Cross-Crate Inlining129//! -------------------------------------130//! The binary of a crate will not only contain machine code for the items131//! defined in the source code of that crate. It will also contain monomorphic132//! instantiations of any extern generic functions and of functions marked with133//! `#[inline]`.134//! The collection algorithm handles this more or less mono. If it is135//! about to create a mono item for something with an external `DefId`,136//! it will take a look if the MIR for that item is available, and if so just137//! proceed normally. If the MIR is not available, it assumes that the item is138//! just linked to and no node is created; which is exactly what we want, since139//! no machine code should be generated in the current crate for such an item.140//!141//! Eager and Lazy Collection Strategy142//! ----------------------------------143//! Mono item collection can be performed with one of two strategies:144//!145//! - Lazy strategy means that items will only be instantiated when actually146//!   used. The goal is to produce the least amount of machine code147//!   possible.148//!149//! - Eager strategy is meant to be used in conjunction with incremental compilation150//!   where a stable set of mono items is more important than a minimal151//!   one. Thus, eager strategy will instantiate drop-glue for every drop-able type152//!   in the crate, even if no drop call for that type exists (yet). It will153//!   also instantiate default implementations of trait methods, something that154//!   otherwise is only done on demand.155//!156//! Collection-time const evaluation and "mentioned" items157//! ------------------------------------------------------158//!159//! One important role of collection is to evaluate all constants that are used by all the items160//! which are being collected. Codegen can then rely on only encountering constants that evaluate161//! successfully, and if a constant fails to evaluate, the collector has much better context to be162//! able to show where this constant comes up.163//!164//! However, the exact set of "used" items (collected as described above), and therefore the exact165//! set of used constants, can depend on optimizations. Optimizing away dead code may optimize away166//! a function call that uses a failing constant, so an unoptimized build may fail where an167//! optimized build succeeds. This is undesirable.168//!169//! To avoid this, the collector has the concept of "mentioned" items. Some time during the MIR170//! pipeline, before any optimization-level-dependent optimizations, we compute a list of all items171//! that syntactically appear in the code. These are considered "mentioned", and even if they are in172//! dead code and get optimized away (which makes them no longer "used"), they are still173//! "mentioned". For every used item, the collector ensures that all mentioned items, recursively,174//! do not use a failing constant. This is reflected via the [`CollectionMode`], which determines175//! whether we are visiting a used item or merely a mentioned item.176//!177//! The collector and "mentioned items" gathering (which lives in `rustc_mir_transform::mentioned_items`)178//! need to stay in sync in the following sense:179//!180//! - For every item that the collector gather that could eventually lead to build failure (most181//!   likely due to containing a constant that fails to evaluate), a corresponding mentioned item182//!   must be added. This should use the exact same strategy as the ecollector to make sure they are183//!   in sync. However, while the collector works on monomorphized types, mentioned items are184//!   collected on generic MIR -- so any time the collector checks for a particular type (such as185//!   `ty::FnDef`), we have to just onconditionally add this as a mentioned item.186//! - In `visit_mentioned_item`, we then do with that mentioned item exactly what the collector187//!   would have done during regular MIR visiting. Basically you can think of the collector having188//!   two stages, a pre-monomorphization stage and a post-monomorphization stage (usually quite189//!   literally separated by a call to `self.monomorphize`); the pre-monomorphizationn stage is190//!   duplicated in mentioned items gathering and the post-monomorphization stage is duplicated in191//!   `visit_mentioned_item`.192//! - Finally, as a performance optimization, the collector should fill `used_mentioned_item` during193//!   its MIR traversal with exactly what mentioned item gathering would have added in the same194//!   situation. This detects mentioned items that have *not* been optimized away and hence don't195//!   need a dedicated traversal.196//!197//! Open Issues198//! -----------199//! Some things are not yet fully implemented in the current version of this200//! module.201//!202//! ### Const Fns203//! Ideally, no mono item should be generated for const fns unless there204//! is a call to them that cannot be evaluated at compile time. At the moment205//! this is not implemented however: a mono item will be produced206//! regardless of whether it is actually needed or not.207208use std::cell::OnceCell;209use std::ops::ControlFlow;210211use rustc_data_structures::fx::FxIndexMap;212use rustc_data_structures::sync::{Lock, par_for_each_in};213use rustc_data_structures::unord::{UnordMap, UnordSet};214use rustc_hir as hir;215use rustc_hir::attrs::InlineAttr;216use rustc_hir::def::DefKind;217use rustc_hir::def_id::{DefId, DefIdMap, LocalDefId};218use rustc_hir::lang_items::LangItem;219use rustc_hir::limit::Limit;220use rustc_middle::middle::codegen_fn_attrs::CodegenFnAttrFlags;221use rustc_middle::mir::interpret::{AllocId, ErrorHandled, GlobalAlloc, Scalar};222use rustc_middle::mir::visit::Visitor as MirVisitor;223use rustc_middle::mir::{self, Body, Location, MentionedItem, traversal};224use rustc_middle::mono::{CollectionMode, InstantiationMode, MonoItem, NormalizationErrorInMono};225use rustc_middle::query::TyCtxtAt;226use rustc_middle::ty::adjustment::{CustomCoerceUnsized, PointerCoercion};227use rustc_middle::ty::layout::ValidityRequirement;228use rustc_middle::ty::{229    self, GenericArgs, GenericParamDefKind, Instance, InstanceKind, Ty, TyCtxt, TypeFoldable,230    TypeVisitable, TypeVisitableExt, TypeVisitor, Unnormalized, VtblEntry,231};232use rustc_middle::util::Providers;233use rustc_middle::{bug, span_bug};234use rustc_session::config::{DebugInfo, EntryFnType};235use rustc_span::{DUMMY_SP, Span, Spanned, dummy_spanned, respan};236use tracing::{debug, instrument, trace};237238use crate::errors::{239    self, EncounteredErrorWhileInstantiating, EncounteredErrorWhileInstantiatingGlobalAsm,240    NoOptimizedMir, RecursionLimit,241};242243#[derive(PartialEq)]244pub(crate) enum MonoItemCollectionStrategy {245    Eager,246    Lazy,247}248249/// The state that is shared across the concurrent threads that are doing collection.250struct SharedState<'tcx> {251    /// Items that have been or are currently being recursively collected.252    visited: Lock<UnordSet<MonoItem<'tcx>>>,253    /// Items that have been or are currently being recursively treated as "mentioned", i.e., their254    /// consts are evaluated but nothing is added to the collection.255    mentioned: Lock<UnordSet<MonoItem<'tcx>>>,256    /// Which items are being used where, for better errors.257    usage_map: Lock<UsageMap<'tcx>>,258}259260pub(crate) struct UsageMap<'tcx> {261    // Maps every mono item to the mono items used by it.262    pub used_map: UnordMap<MonoItem<'tcx>, Vec<MonoItem<'tcx>>>,263264    // Maps each mono item with users to the mono items that use it.265    // Be careful: subsets `used_map`, so unused items are vacant.266    user_map: UnordMap<MonoItem<'tcx>, Vec<MonoItem<'tcx>>>,267}268269impl<'tcx> UsageMap<'tcx> {270    fn new() -> UsageMap<'tcx> {271        UsageMap { used_map: Default::default(), user_map: Default::default() }272    }273274    fn record_used<'a>(&mut self, user_item: MonoItem<'tcx>, used_items: &'a MonoItems<'tcx>)275    where276        'tcx: 'a,277    {278        for used_item in used_items.items() {279            self.user_map.entry(used_item).or_default().push(user_item);280        }281282        assert!(self.used_map.insert(user_item, used_items.items().collect()).is_none());283    }284285    pub(crate) fn get_user_items(&self, item: MonoItem<'tcx>) -> &[MonoItem<'tcx>] {286        self.user_map.get(&item).map(|items| items.as_slice()).unwrap_or(&[])287    }288289    /// Internally iterate over all inlined items used by `item`.290    pub(crate) fn for_each_inlined_used_item<F>(291        &self,292        tcx: TyCtxt<'tcx>,293        item: MonoItem<'tcx>,294        mut f: F,295    ) where296        F: FnMut(MonoItem<'tcx>),297    {298        let used_items = self.used_map.get(&item).unwrap();299        for used_item in used_items.iter() {300            let is_inlined = used_item.instantiation_mode(tcx) == InstantiationMode::LocalCopy;301            if is_inlined {302                f(*used_item);303            }304        }305    }306}307308struct MonoItems<'tcx> {309    // We want a set of MonoItem + Span where trying to re-insert a MonoItem with a different Span310    // is ignored. Map does that, but it looks odd.311    items: FxIndexMap<MonoItem<'tcx>, Span>,312}313314impl<'tcx> MonoItems<'tcx> {315    fn new() -> Self {316        Self { items: FxIndexMap::default() }317    }318319    fn is_empty(&self) -> bool {320        self.items.is_empty()321    }322323    fn push(&mut self, item: Spanned<MonoItem<'tcx>>) {324        // Insert only if the entry does not exist. A normal insert would stomp the first span that325        // got inserted.326        self.items.entry(item.node).or_insert(item.span);327    }328329    fn items(&self) -> impl Iterator<Item = MonoItem<'tcx>> {330        self.items.keys().cloned()331    }332}333334impl<'tcx> IntoIterator for MonoItems<'tcx> {335    type Item = Spanned<MonoItem<'tcx>>;336    type IntoIter = impl Iterator<Item = Spanned<MonoItem<'tcx>>>;337338    fn into_iter(self) -> Self::IntoIter {339        self.items.into_iter().map(|(item, span)| respan(span, item))340    }341}342343impl<'tcx> Extend<Spanned<MonoItem<'tcx>>> for MonoItems<'tcx> {344    fn extend<I>(&mut self, iter: I)345    where346        I: IntoIterator<Item = Spanned<MonoItem<'tcx>>>,347    {348        for item in iter {349            self.push(item)350        }351    }352}353354fn collect_items_root<'tcx>(355    tcx: TyCtxt<'tcx>,356    starting_item: Spanned<MonoItem<'tcx>>,357    state: &SharedState<'tcx>,358    recursion_limit: Limit,359) {360    if !state.visited.lock().insert(starting_item.node) {361        // We've been here already, no need to search again.362        return;363    }364    let mut recursion_depths = DefIdMap::default();365    collect_items_rec(366        tcx,367        starting_item,368        state,369        &mut recursion_depths,370        recursion_limit,371        CollectionMode::UsedItems,372    );373}374375/// Collect all monomorphized items reachable from `starting_point`, and emit a note diagnostic if a376/// post-monomorphization error is encountered during a collection step.377///378/// `mode` determined whether we are scanning for [used items][CollectionMode::UsedItems]379/// or [mentioned items][CollectionMode::MentionedItems].380#[instrument(skip(tcx, state, recursion_depths, recursion_limit), level = "debug")]381fn collect_items_rec<'tcx>(382    tcx: TyCtxt<'tcx>,383    starting_item: Spanned<MonoItem<'tcx>>,384    state: &SharedState<'tcx>,385    recursion_depths: &mut DefIdMap<usize>,386    recursion_limit: Limit,387    mode: CollectionMode,388) {389    let mut used_items = MonoItems::new();390    let mut mentioned_items = MonoItems::new();391    let recursion_depth_reset;392393    // Post-monomorphization errors MVP394    //395    // We can encounter errors while monomorphizing an item, but we don't have a good way of396    // showing a complete stack of spans ultimately leading to collecting the erroneous one yet.397    // (It's also currently unclear exactly which diagnostics and information would be interesting398    // to report in such cases)399    //400    // This leads to suboptimal error reporting: a post-monomorphization error (PME) will be401    // shown with just a spanned piece of code causing the error, without information on where402    // it was called from. This is especially obscure if the erroneous mono item is in a403    // dependency. See for example issue #85155, where, before minimization, a PME happened two404    // crates downstream from libcore's stdarch, without a way to know which dependency was the405    // cause.406    //407    // If such an error occurs in the current crate, its span will be enough to locate the408    // source. If the cause is in another crate, the goal here is to quickly locate which mono409    // item in the current crate is ultimately responsible for causing the error.410    //411    // To give at least _some_ context to the user: while collecting mono items, we check the412    // error count. If it has changed, a PME occurred, and we trigger some diagnostics about the413    // current step of mono items collection.414    //415    // FIXME: don't rely on global state, instead bubble up errors. Note: this is very hard to do.416    let error_count = tcx.dcx().err_count();417418    // In `mentioned_items` we collect items that were mentioned in this MIR but possibly do not419    // need to be monomorphized. This is done to ensure that optimizing away function calls does not420    // hide const-eval errors that those calls would otherwise have triggered.421    match starting_item.node {422        MonoItem::Static(def_id) => {423            recursion_depth_reset = None;424425            // Statics always get evaluated (which is possible because they can't be generic), so for426            // `MentionedItems` collection there's nothing to do here.427            if mode == CollectionMode::UsedItems {428                let instance = Instance::mono(tcx, def_id);429430                // Sanity check whether this ended up being collected accidentally431                debug_assert!(tcx.should_codegen_locally(instance));432433                let DefKind::Static { nested, .. } = tcx.def_kind(def_id) else { bug!() };434                // Nested statics have no type.435                if !nested {436                    let ty = instance.ty(tcx, ty::TypingEnv::fully_monomorphized());437                    visit_drop_use(tcx, ty, true, starting_item.span, &mut used_items);438                }439440                if let Ok(alloc) = tcx.eval_static_initializer(def_id) {441                    for &prov in alloc.inner().provenance().ptrs().values() {442                        collect_alloc(tcx, prov.alloc_id(), &mut used_items);443                    }444                }445446                if tcx.needs_thread_local_shim(def_id) {447                    used_items.push(respan(448                        starting_item.span,449                        MonoItem::Fn(Instance {450                            def: InstanceKind::ThreadLocalShim(def_id),451                            args: GenericArgs::empty(),452                        }),453                    ));454                }455            }456457            // mentioned_items stays empty since there's no codegen for statics. statics don't get458            // optimized, and if they did then the const-eval interpreter would have to worry about459            // mentioned_items.460        }461        MonoItem::Fn(instance) => {462            // Sanity check whether this ended up being collected accidentally463            debug_assert!(tcx.should_codegen_locally(instance));464465            // Keep track of the monomorphization recursion depth466            recursion_depth_reset = Some(check_recursion_limit(467                tcx,468                instance,469                starting_item.span,470                recursion_depths,471                recursion_limit,472            ));473474            rustc_data_structures::stack::ensure_sufficient_stack(|| {475                let Ok((used, mentioned)) = tcx.items_of_instance((instance, mode)) else {476                    // Normalization errors here are usually due to trait solving overflow.477                    // FIXME: I assume that there are few type errors at post-analysis stage, but not478                    // entirely sure.479                    // We have to emit the error outside of `items_of_instance` to access the480                    // span of the `starting_item`.481                    let def_id = instance.def_id();482                    let def_span = tcx.def_span(def_id);483                    let def_path_str = tcx.def_path_str(def_id);484                    tcx.dcx().emit_fatal(RecursionLimit {485                        span: starting_item.span,486                        instance,487                        def_span,488                        def_path_str,489                    });490                };491                used_items.extend(used.into_iter().copied());492                mentioned_items.extend(mentioned.into_iter().copied());493            });494        }495        MonoItem::GlobalAsm(item_id) => {496            assert!(497                mode == CollectionMode::UsedItems,498                "should never encounter global_asm when collecting mentioned items"499            );500            recursion_depth_reset = None;501502            let item = tcx.hir_item(item_id);503            if let hir::ItemKind::GlobalAsm { asm, .. } = item.kind {504                for (op, op_sp) in asm.operands {505                    match *op {506                        hir::InlineAsmOperand::Const { .. } => {507                            // Only constants which resolve to a plain integer508                            // are supported. Therefore the value should not509                            // depend on any other items.510                        }511                        hir::InlineAsmOperand::SymFn { expr } => {512                            let fn_ty = tcx.typeck(item_id.owner_id).expr_ty(expr);513                            visit_fn_use(tcx, fn_ty, false, *op_sp, &mut used_items);514                        }515                        hir::InlineAsmOperand::SymStatic { path: _, def_id } => {516                            let instance = Instance::mono(tcx, def_id);517                            if tcx.should_codegen_locally(instance) {518                                trace!("collecting static {:?}", def_id);519                                used_items.push(dummy_spanned(MonoItem::Static(def_id)));520                            }521                        }522                        hir::InlineAsmOperand::In { .. }523                        | hir::InlineAsmOperand::Out { .. }524                        | hir::InlineAsmOperand::InOut { .. }525                        | hir::InlineAsmOperand::SplitInOut { .. }526                        | hir::InlineAsmOperand::Label { .. } => {527                            span_bug!(*op_sp, "invalid operand type for global_asm!")528                        }529                    }530                }531            } else {532                span_bug!(item.span, "Mismatch between hir::Item type and MonoItem type")533            }534535            // mention_items stays empty as nothing gets optimized here.536        }537    };538539    // Check for PMEs and emit a diagnostic if one happened. To try to show relevant edges of the540    // mono item graph.541    if tcx.dcx().err_count() > error_count542        && starting_item.node.is_generic_fn()543        && starting_item.node.is_user_defined()544    {545        match starting_item.node {546            MonoItem::Fn(instance) => tcx.dcx().emit_note(EncounteredErrorWhileInstantiating {547                span: starting_item.span,548                kind: "fn",549                instance,550            }),551            MonoItem::Static(def_id) => tcx.dcx().emit_note(EncounteredErrorWhileInstantiating {552                span: starting_item.span,553                kind: "static",554                instance: Instance::new_raw(def_id, GenericArgs::empty()),555            }),556            MonoItem::GlobalAsm(_) => {557                tcx.dcx().emit_note(EncounteredErrorWhileInstantiatingGlobalAsm {558                    span: starting_item.span,559                })560            }561        }562    }563    // Only updating `usage_map` for used items as otherwise we may be inserting the same item564    // multiple times (if it is first 'mentioned' and then later actually used), and the usage map565    // logic does not like that.566    // This is part of the output of collection and hence only relevant for "used" items.567    // ("Mentioned" items are only considered internally during collection.)568    if mode == CollectionMode::UsedItems {569        state.usage_map.lock().record_used(starting_item.node, &used_items);570    }571572    {573        let mut visited = OnceCell::default();574        if mode == CollectionMode::UsedItems {575            used_items576                .items577                .retain(|k, _| visited.get_mut_or_init(|| state.visited.lock()).insert(*k));578        }579580        let mut mentioned = OnceCell::default();581        mentioned_items.items.retain(|k, _| {582            !visited.get_or_init(|| state.visited.lock()).contains(k)583                && mentioned.get_mut_or_init(|| state.mentioned.lock()).insert(*k)584        });585    }586    if mode == CollectionMode::MentionedItems {587        assert!(used_items.is_empty(), "'mentioned' collection should never encounter used items");588    } else {589        for used_item in used_items {590            collect_items_rec(591                tcx,592                used_item,593                state,594                recursion_depths,595                recursion_limit,596                CollectionMode::UsedItems,597            );598        }599    }600601    // Walk over mentioned items *after* used items, so that if an item is both mentioned and used then602    // the loop above has fully collected it, so this loop will skip it.603    for mentioned_item in mentioned_items {604        collect_items_rec(605            tcx,606            mentioned_item,607            state,608            recursion_depths,609            recursion_limit,610            CollectionMode::MentionedItems,611        );612    }613614    if let Some((def_id, depth)) = recursion_depth_reset {615        recursion_depths.insert(def_id, depth);616    }617}618619// Check whether we can normalize every type in the instantiated MIR body.620fn check_normalization_error<'tcx>(621    tcx: TyCtxt<'tcx>,622    instance: Instance<'tcx>,623    body: &Body<'tcx>,624) -> Result<(), NormalizationErrorInMono> {625    struct NormalizationChecker<'tcx> {626        tcx: TyCtxt<'tcx>,627        instance: Instance<'tcx>,628    }629    impl<'tcx> TypeVisitor<TyCtxt<'tcx>> for NormalizationChecker<'tcx> {630        type Result = ControlFlow<()>;631632        fn visit_ty(&mut self, t: Ty<'tcx>) -> Self::Result {633            match self.instance.try_instantiate_mir_and_normalize_erasing_regions(634                self.tcx,635                ty::TypingEnv::fully_monomorphized(),636                ty::EarlyBinder::bind(t),637            ) {638                Ok(_) => ControlFlow::Continue(()),639                Err(_) => ControlFlow::Break(()),640            }641        }642    }643644    let mut checker = NormalizationChecker { tcx, instance };645    if body.visit_with(&mut checker).is_break() { Err(NormalizationErrorInMono) } else { Ok(()) }646}647648fn check_recursion_limit<'tcx>(649    tcx: TyCtxt<'tcx>,650    instance: Instance<'tcx>,651    span: Span,652    recursion_depths: &mut DefIdMap<usize>,653    recursion_limit: Limit,654) -> (DefId, usize) {655    let def_id = instance.def_id();656    let recursion_depth = recursion_depths.get(&def_id).cloned().unwrap_or(0);657    debug!(" => recursion depth={}", recursion_depth);658659    let adjusted_recursion_depth = if tcx.is_lang_item(def_id, LangItem::DropInPlace) {660        // HACK: drop_in_place creates tight monomorphization loops. Give661        // it more margin.662        recursion_depth / 4663    } else {664        recursion_depth665    };666667    // Code that needs to instantiate the same function recursively668    // more than the recursion limit is assumed to be causing an669    // infinite expansion.670    if !recursion_limit.value_within_limit(adjusted_recursion_depth) {671        let def_span = tcx.def_span(def_id);672        let def_path_str = tcx.def_path_str(def_id);673        tcx.dcx().emit_fatal(RecursionLimit { span, instance, def_span, def_path_str });674    }675676    recursion_depths.insert(def_id, recursion_depth + 1);677678    (def_id, recursion_depth)679}680681struct MirUsedCollector<'a, 'tcx> {682    tcx: TyCtxt<'tcx>,683    body: &'a mir::Body<'tcx>,684    used_items: &'a mut MonoItems<'tcx>,685    /// See the comment in `collect_items_of_instance` for the purpose of this set.686    /// Note that this contains *not-monomorphized* items!687    used_mentioned_items: &'a mut UnordSet<MentionedItem<'tcx>>,688    instance: Instance<'tcx>,689}690691impl<'a, 'tcx> MirUsedCollector<'a, 'tcx> {692    fn monomorphize<T>(&self, value: T) -> T693    where694        T: TypeFoldable<TyCtxt<'tcx>>,695    {696        trace!("monomorphize: self.instance={:?}", self.instance);697        self.instance.instantiate_mir_and_normalize_erasing_regions(698            self.tcx,699            ty::TypingEnv::fully_monomorphized(),700            ty::EarlyBinder::bind(value),701        )702    }703704    /// Evaluates a *not yet monomorphized* constant.705    fn eval_constant(&mut self, constant: &mir::ConstOperand<'tcx>) -> Option<mir::ConstValue> {706        let const_ = self.monomorphize(constant.const_);707        // Evaluate the constant. This makes const eval failure a collection-time error (rather than708        // a codegen-time error). rustc stops after collection if there was an error, so this709        // ensures codegen never has to worry about failing consts.710        // (codegen relies on this and ICEs will happen if this is violated.)711        match const_.eval(self.tcx, ty::TypingEnv::fully_monomorphized(), constant.span) {712            Ok(v) => Some(v),713            Err(ErrorHandled::TooGeneric(..)) => span_bug!(714                constant.span,715                "collection encountered polymorphic constant: {:?}",716                const_717            ),718            Err(err @ ErrorHandled::Reported(..)) => {719                err.emit_note(self.tcx);720                return None;721            }722        }723    }724}725726impl<'a, 'tcx> MirVisitor<'tcx> for MirUsedCollector<'a, 'tcx> {727    fn visit_rvalue(&mut self, rvalue: &mir::Rvalue<'tcx>, location: Location) {728        debug!("visiting rvalue {:?}", *rvalue);729730        let span = self.body.source_info(location).span;731732        match *rvalue {733            // When doing an cast from a regular pointer to a wide pointer, we734            // have to instantiate all methods of the trait being cast to, so we735            // can build the appropriate vtable.736            mir::Rvalue::Cast(737                mir::CastKind::PointerCoercion(PointerCoercion::Unsize, _),738                ref operand,739                target_ty,740            ) => {741                let source_ty = operand.ty(self.body, self.tcx);742                // *Before* monomorphizing, record that we already handled this mention.743                self.used_mentioned_items744                    .insert(MentionedItem::UnsizeCast { source_ty, target_ty });745                let target_ty = self.monomorphize(target_ty);746                let source_ty = self.monomorphize(source_ty);747                let (source_ty, target_ty) =748                    find_tails_for_unsizing(self.tcx.at(span), source_ty, target_ty);749                // This could also be a different Unsize instruction, like750                // from a fixed sized array to a slice. But we are only751                // interested in things that produce a vtable.752                if target_ty.is_trait() && !source_ty.is_trait() {753                    create_mono_items_for_vtable_methods(754                        self.tcx,755                        target_ty,756                        source_ty,757                        span,758                        self.used_items,759                    );760                }761            }762            mir::Rvalue::Cast(763                mir::CastKind::PointerCoercion(PointerCoercion::ReifyFnPointer(_), _),764                ref operand,765                _,766            ) => {767                let fn_ty = operand.ty(self.body, self.tcx);768                // *Before* monomorphizing, record that we already handled this mention.769                self.used_mentioned_items.insert(MentionedItem::Fn(fn_ty));770                let fn_ty = self.monomorphize(fn_ty);771                visit_fn_use(self.tcx, fn_ty, false, span, self.used_items);772            }773            mir::Rvalue::Cast(774                mir::CastKind::PointerCoercion(PointerCoercion::ClosureFnPointer(_), _),775                ref operand,776                _,777            ) => {778                let source_ty = operand.ty(self.body, self.tcx);779                // *Before* monomorphizing, record that we already handled this mention.780                self.used_mentioned_items.insert(MentionedItem::Closure(source_ty));781                let source_ty = self.monomorphize(source_ty);782                if let ty::Closure(def_id, args) = *source_ty.kind() {783                    let instance =784                        Instance::resolve_closure(self.tcx, def_id, args, ty::ClosureKind::FnOnce);785                    if self.tcx.should_codegen_locally(instance) {786                        self.used_items.push(create_fn_mono_item(self.tcx, instance, span));787                    }788                } else {789                    bug!()790                }791            }792            mir::Rvalue::ThreadLocalRef(def_id) => {793                assert!(self.tcx.is_thread_local_static(def_id));794                let instance = Instance::mono(self.tcx, def_id);795                if self.tcx.should_codegen_locally(instance) {796                    trace!("collecting thread-local static {:?}", def_id);797                    self.used_items.push(respan(span, MonoItem::Static(def_id)));798                }799            }800            _ => { /* not interesting */ }801        }802803        self.super_rvalue(rvalue, location);804    }805806    /// This does not walk the MIR of the constant as that is not needed for codegen, all we need is807    /// to ensure that the constant evaluates successfully and walk the result.808    #[instrument(skip(self), level = "debug")]809    fn visit_const_operand(&mut self, constant: &mir::ConstOperand<'tcx>, _location: Location) {810        // No `super_constant` as we don't care about `visit_ty`/`visit_ty_const`.811        let Some(val) = self.eval_constant(constant) else { return };812        collect_const_value(self.tcx, val, self.used_items);813    }814815    fn visit_terminator(&mut self, terminator: &mir::Terminator<'tcx>, location: Location) {816        debug!("visiting terminator {:?} @ {:?}", terminator, location);817        let source = self.body.source_info(location).span;818819        let tcx = self.tcx;820        let push_mono_lang_item = |this: &mut Self, lang_item: LangItem| {821            let instance = Instance::mono(tcx, tcx.require_lang_item(lang_item, source));822            if tcx.should_codegen_locally(instance) {823                this.used_items.push(create_fn_mono_item(tcx, instance, source));824            }825        };826827        match terminator.kind {828            mir::TerminatorKind::Call { ref func, .. }829            | mir::TerminatorKind::TailCall { ref func, .. } => {830                let callee_ty = func.ty(self.body, tcx);831                // *Before* monomorphizing, record that we already handled this mention.832                self.used_mentioned_items.insert(MentionedItem::Fn(callee_ty));833                let callee_ty = self.monomorphize(callee_ty);834835                // HACK(explicit_tail_calls): collect tail calls to `#[track_caller]` functions as indirect,836                // because we later call them as such, to prevent issues with ABI incompatibility.837                // Ideally we'd replace such tail calls with normal call + return, but this requires838                // post-mono MIR optimizations, which we don't yet have.839                let force_indirect_call =840                    if matches!(terminator.kind, mir::TerminatorKind::TailCall { .. })841                        && let &ty::FnDef(def_id, args) = callee_ty.kind()842                        && let instance = ty::Instance::expect_resolve(843                            self.tcx,844                            ty::TypingEnv::fully_monomorphized(),845                            def_id,846                            args,847                            source,848                        )849                        && instance.def.requires_caller_location(self.tcx)850                    {851                        true852                    } else {853                        false854                    };855856                visit_fn_use(857                    self.tcx,858                    callee_ty,859                    !force_indirect_call,860                    source,861                    &mut self.used_items,862                )863            }864            mir::TerminatorKind::Drop { ref place, .. } => {865                let ty = place.ty(self.body, self.tcx).ty;866                // *Before* monomorphizing, record that we already handled this mention.867                self.used_mentioned_items.insert(MentionedItem::Drop(ty));868                let ty = self.monomorphize(ty);869                visit_drop_use(self.tcx, ty, true, source, self.used_items);870            }871            mir::TerminatorKind::InlineAsm { ref operands, .. } => {872                for op in operands {873                    match *op {874                        mir::InlineAsmOperand::SymFn { ref value } => {875                            let fn_ty = value.const_.ty();876                            // *Before* monomorphizing, record that we already handled this mention.877                            self.used_mentioned_items.insert(MentionedItem::Fn(fn_ty));878                            let fn_ty = self.monomorphize(fn_ty);879                            visit_fn_use(self.tcx, fn_ty, false, source, self.used_items);880                        }881                        mir::InlineAsmOperand::SymStatic { def_id } => {882                            let instance = Instance::mono(self.tcx, def_id);883                            if self.tcx.should_codegen_locally(instance) {884                                trace!("collecting asm sym static {:?}", def_id);885                                self.used_items.push(respan(source, MonoItem::Static(def_id)));886                            }887                        }888                        _ => {}889                    }890                }891            }892            mir::TerminatorKind::Assert { ref msg, .. } => match &**msg {893                mir::AssertKind::BoundsCheck { .. } => {894                    push_mono_lang_item(self, LangItem::PanicBoundsCheck);895                }896                mir::AssertKind::MisalignedPointerDereference { .. } => {897                    push_mono_lang_item(self, LangItem::PanicMisalignedPointerDereference);898                }899                mir::AssertKind::NullPointerDereference => {900                    push_mono_lang_item(self, LangItem::PanicNullPointerDereference);901                }902                mir::AssertKind::InvalidEnumConstruction(_) => {903                    push_mono_lang_item(self, LangItem::PanicInvalidEnumConstruction);904                }905                _ => {906                    push_mono_lang_item(self, msg.panic_function());907                }908            },909            mir::TerminatorKind::UnwindTerminate(reason) => {910                push_mono_lang_item(self, reason.lang_item());911            }912            mir::TerminatorKind::Goto { .. }913            | mir::TerminatorKind::SwitchInt { .. }914            | mir::TerminatorKind::UnwindResume915            | mir::TerminatorKind::Return916            | mir::TerminatorKind::Unreachable => {}917            mir::TerminatorKind::CoroutineDrop918            | mir::TerminatorKind::Yield { .. }919            | mir::TerminatorKind::FalseEdge { .. }920            | mir::TerminatorKind::FalseUnwind { .. } => bug!(),921        }922923        if let Some(mir::UnwindAction::Terminate(reason)) = terminator.unwind() {924            push_mono_lang_item(self, reason.lang_item());925        }926927        self.super_terminator(terminator, location);928    }929}930931fn visit_drop_use<'tcx>(932    tcx: TyCtxt<'tcx>,933    ty: Ty<'tcx>,934    is_direct_call: bool,935    source: Span,936    output: &mut MonoItems<'tcx>,937) {938    let instance = Instance::resolve_drop_in_place(tcx, ty);939    visit_instance_use(tcx, instance, is_direct_call, source, output);940}941942/// For every call of this function in the visitor, make sure there is a matching call in the943/// `mentioned_items` pass!944fn visit_fn_use<'tcx>(945    tcx: TyCtxt<'tcx>,946    ty: Ty<'tcx>,947    is_direct_call: bool,948    source: Span,949    output: &mut MonoItems<'tcx>,950) {951    if let ty::FnDef(def_id, args) = *ty.kind() {952        let instance = if is_direct_call {953            ty::Instance::expect_resolve(954                tcx,955                ty::TypingEnv::fully_monomorphized(),956                def_id,957                args,958                source,959            )960        } else {961            match ty::Instance::resolve_for_fn_ptr(962                tcx,963                ty::TypingEnv::fully_monomorphized(),964                def_id,965                args,966            ) {967                Some(instance) => instance,968                _ => bug!("failed to resolve instance for {ty}"),969            }970        };971        visit_instance_use(tcx, instance, is_direct_call, source, output);972    }973}974975fn visit_instance_use<'tcx>(976    tcx: TyCtxt<'tcx>,977    instance: ty::Instance<'tcx>,978    is_direct_call: bool,979    source: Span,980    output: &mut MonoItems<'tcx>,981) {982    debug!("visit_item_use({:?}, is_direct_call={:?})", instance, is_direct_call);983    if !tcx.should_codegen_locally(instance) {984        return;985    }986    if let Some(intrinsic) = tcx.intrinsic(instance.def_id()) {987        if let Some(_requirement) = ValidityRequirement::from_intrinsic(intrinsic.name) {988            // The intrinsics assert_inhabited, assert_zero_valid, and assert_mem_uninitialized_valid will989            // be lowered in codegen to nothing or a call to panic_nounwind. So if we encounter any990            // of those intrinsics, we need to include a mono item for panic_nounwind, else we may try to991            // codegen a call to that function without generating code for the function itself.992            let def_id = tcx.require_lang_item(LangItem::PanicNounwind, source);993            let panic_instance = Instance::mono(tcx, def_id);994            if tcx.should_codegen_locally(panic_instance) {995                output.push(create_fn_mono_item(tcx, panic_instance, source));996            }997        } else if !intrinsic.must_be_overridden998            && !tcx.sess.replaced_intrinsics.contains(&intrinsic.name)999        {1000            // Codegen the fallback body of intrinsics with fallback bodies.1001            // We have to skip this otherwise as there's no body to codegen.1002            // We also skip intrinsics the backend handles, to reduce monomorphizations.1003            let instance = ty::Instance::new_raw(instance.def_id(), instance.args);1004            if tcx.should_codegen_locally(instance) {1005                output.push(create_fn_mono_item(tcx, instance, source));1006            }1007        }1008    }10091010    match instance.def {1011        ty::InstanceKind::Virtual(..) | ty::InstanceKind::Intrinsic(_) => {1012            if !is_direct_call {1013                bug!("{:?} being reified", instance);1014            }1015        }1016        ty::InstanceKind::ThreadLocalShim(..) => {1017            bug!("{:?} being reified", instance);1018        }1019        ty::InstanceKind::DropGlue(_, None) => {1020            // Don't need to emit noop drop glue if we are calling directly.1021            //1022            // Note that we also optimize away the call to visit_instance_use in vtable construction1023            // (see create_mono_items_for_vtable_methods).1024            if !is_direct_call {1025                output.push(create_fn_mono_item(tcx, instance, source));1026            }1027        }1028        ty::InstanceKind::DropGlue(_, Some(_))1029        | ty::InstanceKind::FutureDropPollShim(..)1030        | ty::InstanceKind::AsyncDropGlue(_, _)1031        | ty::InstanceKind::AsyncDropGlueCtorShim(_, _)1032        | ty::InstanceKind::VTableShim(..)1033        | ty::InstanceKind::ReifyShim(..)1034        | ty::InstanceKind::ClosureOnceShim { .. }1035        | ty::InstanceKind::ConstructCoroutineInClosureShim { .. }1036        | ty::InstanceKind::Item(..)1037        | ty::InstanceKind::FnPtrShim(..)1038        | ty::InstanceKind::CloneShim(..)1039        | ty::InstanceKind::FnPtrAddrShim(..) => {1040            output.push(create_fn_mono_item(tcx, instance, source));1041        }1042    }1043}10441045/// Returns `true` if we should codegen an instance in the local crate, or returns `false` if we1046/// can just link to the upstream crate and therefore don't need a mono item.1047fn should_codegen_locally<'tcx>(tcx: TyCtxt<'tcx>, instance: Instance<'tcx>) -> bool {1048    let Some(def_id) = instance.def.def_id_if_not_guaranteed_local_codegen() else {1049        return true;1050    };10511052    if tcx.is_foreign_item(def_id) {1053        // Foreign items are always linked against, there's no way of instantiating them.1054        return false;1055    }10561057    if tcx.def_kind(def_id).has_codegen_attrs()1058        && matches!(tcx.codegen_fn_attrs(def_id).inline, InlineAttr::Force { .. })1059    {1060        // `#[rustc_force_inline]` items should never be codegened. This should be caught by1061        // the MIR validator.1062        tcx.dcx().delayed_bug("attempt to codegen `#[rustc_force_inline]` item");1063    }10641065    if def_id.is_local() {1066        // Local items cannot be referred to locally without monomorphizing them locally.1067        return true;1068    }10691070    if tcx.is_reachable_non_generic(def_id) || instance.upstream_monomorphization(tcx).is_some() {1071        // We can link to the item in question, no instance needed in this crate.1072        return false;1073    }10741075    if let DefKind::Static { .. } = tcx.def_kind(def_id) {1076        // We cannot monomorphize statics from upstream crates.1077        return false;1078    }10791080    // See comment in should_encode_mir in rustc_metadata for why we don't report1081    // an error for constructors.1082    if !tcx.is_mir_available(def_id) && !matches!(tcx.def_kind(def_id), DefKind::Ctor(..)) {1083        tcx.dcx().emit_fatal(NoOptimizedMir {1084            span: tcx.def_span(def_id),1085            crate_name: tcx.crate_name(def_id.krate),1086            instance: instance.to_string(),1087        });1088    }10891090    true1091}10921093/// For a given pair of source and target type that occur in an unsizing coercion,1094/// this function finds the pair of types that determines the vtable linking1095/// them.1096///1097/// For example, the source type might be `&SomeStruct` and the target type1098/// might be `&dyn SomeTrait` in a cast like:1099///1100/// ```rust,ignore (not real code)1101/// let src: &SomeStruct = ...;1102/// let target = src as &dyn SomeTrait;1103/// ```1104///1105/// Then the output of this function would be (SomeStruct, SomeTrait) since for1106/// constructing the `target` wide-pointer we need the vtable for that pair.1107///1108/// Things can get more complicated though because there's also the case where1109/// the unsized type occurs as a field:1110///1111/// ```rust1112/// struct ComplexStruct<T: ?Sized> {1113///    a: u32,1114///    b: f64,1115///    c: T1116/// }1117/// ```1118///1119/// In this case, if `T` is sized, `&ComplexStruct<T>` is a thin pointer. If `T`1120/// is unsized, `&SomeStruct` is a wide pointer, and the vtable it points to is1121/// for the pair of `T` (which is a trait) and the concrete type that `T` was1122/// originally coerced from:1123///1124/// ```rust,ignore (not real code)1125/// let src: &ComplexStruct<SomeStruct> = ...;1126/// let target = src as &ComplexStruct<dyn SomeTrait>;1127/// ```1128///1129/// Again, we want this `find_vtable_types_for_unsizing()` to provide the pair1130/// `(SomeStruct, SomeTrait)`.1131///1132/// Finally, there is also the case of custom unsizing coercions, e.g., for1133/// smart pointers such as `Rc` and `Arc`.1134fn find_tails_for_unsizing<'tcx>(1135    tcx: TyCtxtAt<'tcx>,1136    source_ty: Ty<'tcx>,1137    target_ty: Ty<'tcx>,1138) -> (Ty<'tcx>, Ty<'tcx>) {1139    let typing_env = ty::TypingEnv::fully_monomorphized();1140    debug_assert!(!source_ty.has_param(), "{source_ty} should be fully monomorphic");1141    debug_assert!(!target_ty.has_param(), "{target_ty} should be fully monomorphic");11421143    match (source_ty.kind(), target_ty.kind()) {1144        (&ty::Pat(source, _), &ty::Pat(target, _)) => find_tails_for_unsizing(tcx, source, target),1145        (1146            &ty::Ref(_, source_pointee, _),1147            &ty::Ref(_, target_pointee, _) | &ty::RawPtr(target_pointee, _),1148        )1149        | (&ty::RawPtr(source_pointee, _), &ty::RawPtr(target_pointee, _)) => {1150            tcx.struct_lockstep_tails_for_codegen(source_pointee, target_pointee, typing_env)1151        }11521153        // `Box<T>` could go through the ADT code below, b/c it'll unpeel to `Unique<T>`,1154        // and eventually bottom out in a raw ref, but we can micro-optimize it here.1155        (_, _)1156            if let Some(source_boxed) = source_ty.boxed_ty()1157                && let Some(target_boxed) = target_ty.boxed_ty() =>1158        {1159            tcx.struct_lockstep_tails_for_codegen(source_boxed, target_boxed, typing_env)1160        }11611162        (&ty::Adt(source_adt_def, source_args), &ty::Adt(target_adt_def, target_args)) => {1163            assert_eq!(source_adt_def, target_adt_def);1164            let CustomCoerceUnsized::Struct(coerce_index) =1165                match crate::custom_coerce_unsize_info(tcx, source_ty, target_ty) {1166                    Ok(ccu) => ccu,1167                    Err(e) => {1168                        let e = Ty::new_error(tcx.tcx, e);1169                        return (e, e);1170                    }1171                };1172            let coerce_field = &source_adt_def.non_enum_variant().fields[coerce_index];1173            // We're getting a possibly unnormalized type, so normalize it.1174            let source_field = tcx.normalize_erasing_regions(1175                typing_env,1176                Unnormalized::new_wip(coerce_field.ty(*tcx, source_args)),1177            );1178            let target_field = tcx.normalize_erasing_regions(1179                typing_env,1180                Unnormalized::new_wip(coerce_field.ty(*tcx, target_args)),1181            );1182            find_tails_for_unsizing(tcx, source_field, target_field)1183        }11841185        _ => bug!(1186            "find_vtable_types_for_unsizing: invalid coercion {:?} -> {:?}",1187            source_ty,1188            target_ty1189        ),1190    }1191}11921193#[instrument(skip(tcx), level = "debug", ret)]1194fn create_fn_mono_item<'tcx>(1195    tcx: TyCtxt<'tcx>,1196    instance: Instance<'tcx>,1197    source: Span,1198) -> Spanned<MonoItem<'tcx>> {1199    let def_id = instance.def_id();1200    if tcx.sess.opts.unstable_opts.profile_closures1201        && def_id.is_local()1202        && tcx.is_closure_like(def_id)1203    {1204        crate::util::dump_closure_profile(tcx, instance);1205    }12061207    respan(source, MonoItem::Fn(instance))1208}12091210/// Creates a `MonoItem` for each method that is referenced by the vtable for1211/// the given trait/impl pair.1212fn create_mono_items_for_vtable_methods<'tcx>(1213    tcx: TyCtxt<'tcx>,1214    trait_ty: Ty<'tcx>,1215    impl_ty: Ty<'tcx>,1216    source: Span,1217    output: &mut MonoItems<'tcx>,1218) {1219    assert!(!trait_ty.has_escaping_bound_vars() && !impl_ty.has_escaping_bound_vars());12201221    let ty::Dynamic(trait_ty, ..) = trait_ty.kind() else {1222        bug!("create_mono_items_for_vtable_methods: {trait_ty:?} not a trait type");1223    };1224    if let Some(principal) = trait_ty.principal() {1225        let trait_ref =1226            tcx.instantiate_bound_regions_with_erased(principal.with_self_ty(tcx, impl_ty));1227        assert!(!trait_ref.has_escaping_bound_vars());12281229        // Walk all methods of the trait, including those of its supertraits1230        let entries = tcx.vtable_entries(trait_ref);1231        debug!(?entries);1232        let methods = entries1233            .iter()1234            .filter_map(|entry| match entry {1235                VtblEntry::MetadataDropInPlace1236                | VtblEntry::MetadataSize1237                | VtblEntry::MetadataAlign1238                | VtblEntry::Vacant => None,1239                VtblEntry::TraitVPtr(_) => {1240                    // all super trait items already covered, so skip them.1241                    None1242                }1243                VtblEntry::Method(instance) => {1244                    Some(*instance).filter(|instance| tcx.should_codegen_locally(*instance))1245                }1246            })1247            .map(|item| create_fn_mono_item(tcx, item, source));1248        output.extend(methods);1249    }12501251    // Also add the destructor, if it's necessary.1252    //1253    // This matches the check in vtable_allocation_provider in middle/ty/vtable.rs,1254    // if we don't need drop we're not adding an actual pointer to the vtable.1255    if impl_ty.needs_drop(tcx, ty::TypingEnv::fully_monomorphized()) {1256        visit_drop_use(tcx, impl_ty, false, source, output);1257    }1258}12591260/// Scans the CTFE alloc in order to find function pointers and statics that must be monomorphized.1261fn collect_alloc<'tcx>(tcx: TyCtxt<'tcx>, alloc_id: AllocId, output: &mut MonoItems<'tcx>) {1262    match tcx.global_alloc(alloc_id) {1263        GlobalAlloc::Static(def_id) => {1264            assert!(!tcx.is_thread_local_static(def_id));1265            let instance = Instance::mono(tcx, def_id);1266            if tcx.should_codegen_locally(instance) {1267                trace!("collecting static {:?}", def_id);1268                output.push(dummy_spanned(MonoItem::Static(def_id)));1269            }1270        }1271        GlobalAlloc::Memory(alloc) => {1272            trace!("collecting {:?} with {:#?}", alloc_id, alloc);1273            let ptrs = alloc.inner().provenance().ptrs();1274            // avoid `ensure_sufficient_stack` in the common case of "no pointers"1275            if !ptrs.is_empty() {1276                rustc_data_structures::stack::ensure_sufficient_stack(move || {1277                    for &prov in ptrs.values() {1278                        collect_alloc(tcx, prov.alloc_id(), output);1279                    }1280                });1281            }1282        }1283        GlobalAlloc::Function { instance, .. } => {1284            if tcx.should_codegen_locally(instance) {1285                trace!("collecting {:?} with {:#?}", alloc_id, instance);1286                output.push(create_fn_mono_item(tcx, instance, DUMMY_SP));1287            }1288        }1289        GlobalAlloc::VTable(ty, dyn_ty) => {1290            let alloc_id = tcx.vtable_allocation((1291                ty,1292                dyn_ty1293                    .principal()1294                    .map(|principal| tcx.instantiate_bound_regions_with_erased(principal)),1295            ));1296            collect_alloc(tcx, alloc_id, output)1297        }1298        GlobalAlloc::TypeId { .. } => {}1299    }1300}13011302/// Scans the MIR in order to find function calls, closures, and drop-glue.1303///1304/// Anything that's found is added to `output`. Furthermore the "mentioned items" of the MIR are returned.1305#[instrument(skip(tcx), level = "debug")]1306fn collect_items_of_instance<'tcx>(1307    tcx: TyCtxt<'tcx>,1308    instance: Instance<'tcx>,1309    mode: CollectionMode,1310) -> Result<(MonoItems<'tcx>, MonoItems<'tcx>), NormalizationErrorInMono> {1311    // This item is getting monomorphized, do mono-time checks.1312    let body = tcx.instance_mir(instance.def);1313    // Plenty of code paths later assume that everything can be normalized. So we have to check1314    // normalization first.1315    // We choose to emit the error outside to provide helpful diagnostics.1316    check_normalization_error(tcx, instance, body)?;1317    tcx.ensure_ok().check_mono_item(instance);13181319    // Naively, in "used" collection mode, all functions get added to *both* `used_items` and1320    // `mentioned_items`. Mentioned items processing will then notice that they have already been1321    // visited, but at that point each mentioned item has been monomorphized, added to the1322    // `mentioned_items` worklist, and checked in the global set of visited items. To remove that1323    // overhead, we have a special optimization that avoids adding items to `mentioned_items` when1324    // they are already added in `used_items`. We could just scan `used_items`, but that's a linear1325    // scan and not very efficient. Furthermore we can only do that *after* monomorphizing the1326    // mentioned item. So instead we collect all pre-monomorphized `MentionedItem` that were already1327    // added to `used_items` in a hash set, which can efficiently query in the1328    // `body.mentioned_items` loop below without even having to monomorphize the item.1329    let mut used_items = MonoItems::new();1330    let mut mentioned_items = MonoItems::new();1331    let mut used_mentioned_items = Default::default();1332    let mut collector = MirUsedCollector {1333        tcx,1334        body,1335        used_items: &mut used_items,1336        used_mentioned_items: &mut used_mentioned_items,1337        instance,1338    };13391340    if mode == CollectionMode::UsedItems {1341        if tcx.sess.opts.debuginfo == DebugInfo::Full {1342            for var_debug_info in &body.var_debug_info {1343                collector.visit_var_debug_info(var_debug_info);1344            }1345        }1346        for (bb, data) in traversal::mono_reachable(body, tcx, instance) {1347            collector.visit_basic_block_data(bb, data)1348        }1349    }13501351    // Always visit all `required_consts`, so that we evaluate them and abort compilation if any of1352    // them errors.1353    for const_op in body.required_consts() {1354        if let Some(val) = collector.eval_constant(const_op) {1355            collect_const_value(tcx, val, &mut mentioned_items);1356        }1357    }13581359    // Always gather mentioned items. We try to avoid processing items that we have already added to1360    // `used_items` above.1361    for item in body.mentioned_items() {1362        if !collector.used_mentioned_items.contains(&item.node) {1363            let item_mono = collector.monomorphize(item.node);1364            visit_mentioned_item(tcx, &item_mono, item.span, &mut mentioned_items);1365        }1366    }13671368    Ok((used_items, mentioned_items))1369}13701371fn items_of_instance<'tcx>(1372    tcx: TyCtxt<'tcx>,1373    (instance, mode): (Instance<'tcx>, CollectionMode),1374) -> Result<1375    (&'tcx [Spanned<MonoItem<'tcx>>], &'tcx [Spanned<MonoItem<'tcx>>]),1376    NormalizationErrorInMono,1377> {1378    let (used_items, mentioned_items) = collect_items_of_instance(tcx, instance, mode)?;13791380    let used_items = tcx.arena.alloc_from_iter(used_items);1381    let mentioned_items = tcx.arena.alloc_from_iter(mentioned_items);13821383    Ok((used_items, mentioned_items))1384}13851386/// `item` must be already monomorphized.1387#[instrument(skip(tcx, span, output), level = "debug")]1388fn visit_mentioned_item<'tcx>(1389    tcx: TyCtxt<'tcx>,1390    item: &MentionedItem<'tcx>,1391    span: Span,1392    output: &mut MonoItems<'tcx>,1393) {1394    match *item {1395        MentionedItem::Fn(ty) => {1396            if let ty::FnDef(def_id, args) = *ty.kind() {1397                let instance = Instance::expect_resolve(1398                    tcx,1399                    ty::TypingEnv::fully_monomorphized(),1400                    def_id,1401                    args,1402                    span,1403                );1404                // `visit_instance_use` was written for "used" item collection but works just as well1405                // for "mentioned" item collection.1406                // We can set `is_direct_call`; that just means we'll skip a bunch of shims that anyway1407                // can't have their own failing constants.1408                visit_instance_use(tcx, instance, /*is_direct_call*/ true, span, output);1409            }1410        }1411        MentionedItem::Drop(ty) => {1412            visit_drop_use(tcx, ty, /*is_direct_call*/ true, span, output);1413        }1414        MentionedItem::UnsizeCast { source_ty, target_ty } => {1415            let (source_ty, target_ty) =1416                find_tails_for_unsizing(tcx.at(span), source_ty, target_ty);1417            // This could also be a different Unsize instruction, like1418            // from a fixed sized array to a slice. But we are only1419            // interested in things that produce a vtable.1420            if target_ty.is_trait() && !source_ty.is_trait() {1421                create_mono_items_for_vtable_methods(tcx, target_ty, source_ty, span, output);1422            }1423        }1424        MentionedItem::Closure(source_ty) => {1425            if let ty::Closure(def_id, args) = *source_ty.kind() {1426                let instance =1427                    Instance::resolve_closure(tcx, def_id, args, ty::ClosureKind::FnOnce);1428                if tcx.should_codegen_locally(instance) {1429                    output.push(create_fn_mono_item(tcx, instance, span));1430                }1431            } else {1432                bug!()1433            }1434        }1435    }1436}14371438#[instrument(skip(tcx, output), level = "debug")]1439fn collect_const_value<'tcx>(1440    tcx: TyCtxt<'tcx>,1441    value: mir::ConstValue,1442    output: &mut MonoItems<'tcx>,1443) {1444    match value {1445        mir::ConstValue::Scalar(Scalar::Ptr(ptr, _size)) => {1446            collect_alloc(tcx, ptr.provenance.alloc_id(), output)1447        }1448        mir::ConstValue::Indirect { alloc_id, .. }1449        | mir::ConstValue::Slice { alloc_id, meta: _ } => collect_alloc(tcx, alloc_id, output),1450        _ => {}1451    }1452}14531454//=-----------------------------------------------------------------------------1455// Root Collection1456//=-----------------------------------------------------------------------------14571458// Find all non-generic items by walking the HIR. These items serve as roots to1459// start monomorphizing from.1460#[instrument(skip(tcx, mode), level = "debug")]1461fn collect_roots(tcx: TyCtxt<'_>, mode: MonoItemCollectionStrategy) -> Vec<MonoItem<'_>> {1462    debug!("collecting roots");1463    let mut roots = MonoItems::new();14641465    {1466        let entry_fn = tcx.entry_fn(());14671468        debug!("collect_roots: entry_fn = {:?}", entry_fn);14691470        let mut collector = RootCollector { tcx, strategy: mode, entry_fn, output: &mut roots };14711472        let crate_items = tcx.hir_crate_items(());14731474        for id in crate_items.free_items() {1475            collector.process_item(id);1476        }14771478        for id in crate_items.impl_items() {1479            collector.process_impl_item(id);1480        }14811482        for id in crate_items.nested_bodies() {1483            collector.process_nested_body(id);1484        }14851486        collector.push_extra_entry_roots();1487    }14881489    // We can only codegen items that are instantiable - items all of1490    // whose predicates hold. Luckily, items that aren't instantiable1491    // can't actually be used, so we can just skip codegenning them.1492    roots1493        .into_iter()1494        .filter_map(|Spanned { node: mono_item, .. }| {1495            mono_item.is_instantiable(tcx).then_some(mono_item)1496        })1497        .collect()1498}14991500struct RootCollector<'a, 'tcx> {1501    tcx: TyCtxt<'tcx>,1502    strategy: MonoItemCollectionStrategy,1503    output: &'a mut MonoItems<'tcx>,1504    entry_fn: Option<(DefId, EntryFnType)>,1505}15061507impl<'v> RootCollector<'_, 'v> {1508    fn process_item(&mut self, id: hir::ItemId) {1509        match self.tcx.def_kind(id.owner_id) {1510            DefKind::Enum | DefKind::Struct | DefKind::Union => {1511                if self.strategy == MonoItemCollectionStrategy::Eager1512                    && !self.tcx.generics_of(id.owner_id).requires_monomorphization(self.tcx)1513                {1514                    debug!("RootCollector: ADT drop-glue for `{id:?}`",);1515                    let id_args =1516                        ty::GenericArgs::for_item(self.tcx, id.owner_id.to_def_id(), |param, _| {1517                            match param.kind {1518                                GenericParamDefKind::Lifetime => {1519                                    self.tcx.lifetimes.re_erased.into()1520                                }1521                                GenericParamDefKind::Type { .. }1522                                | GenericParamDefKind::Const { .. } => {1523                                    unreachable!(1524                                        "`own_requires_monomorphization` check means that \1525                                we should have no type/const params"1526                                    )1527                                }1528                            }1529                        });15301531                    // This type is impossible to instantiate, so we should not try to1532                    // generate a `drop_in_place` instance for it.1533                    if self.tcx.instantiate_and_check_impossible_predicates((1534                        id.owner_id.to_def_id(),1535                        id_args,1536                    )) {1537                        return;1538                    }15391540                    let ty = self1541                        .tcx1542                        .type_of(id.owner_id.to_def_id())1543                        .instantiate(self.tcx, id_args)1544                        .skip_norm_wip();1545                    assert!(!ty.has_non_region_param());1546                    visit_drop_use(self.tcx, ty, true, DUMMY_SP, self.output);1547                }1548            }1549            DefKind::GlobalAsm => {1550                debug!(1551                    "RootCollector: ItemKind::GlobalAsm({})",1552                    self.tcx.def_path_str(id.owner_id)1553                );1554                self.output.push(dummy_spanned(MonoItem::GlobalAsm(id)));1555            }1556            DefKind::Static { .. } => {1557                let def_id = id.owner_id.to_def_id();1558                debug!("RootCollector: ItemKind::Static({})", self.tcx.def_path_str(def_id));1559                self.output.push(dummy_spanned(MonoItem::Static(def_id)));1560            }1561            DefKind::Const { .. } => {1562                // Const items only generate mono items if they are actually used somewhere.1563                // Just declaring them is insufficient.15641565                // If we're collecting items eagerly, then recurse into all constants.1566                // Otherwise the value is only collected when explicitly mentioned in other items.1567                if self.strategy == MonoItemCollectionStrategy::Eager {1568                    let def_id = id.owner_id.to_def_id();1569                    // Type Consts don't have bodies to evaluate1570                    // nor do they make sense as a static.1571                    if self.tcx.is_type_const(def_id) {1572                        // FIXME(mgca): Is this actually what we want? We may want to1573                        // normalize to a ValTree then convert to a const allocation and1574                        // collect that?1575                        return;1576                    }1577                    if self.tcx.generics_of(id.owner_id).own_requires_monomorphization() {1578                        return;1579                    }1580                    let Ok(val) = self.tcx.const_eval_poly(def_id) else {1581                        return;1582                    };1583                    collect_const_value(self.tcx, val, self.output);1584                }1585            }1586            DefKind::Impl { of_trait: true } => {1587                if self.strategy == MonoItemCollectionStrategy::Eager {1588                    create_mono_items_for_default_impls(self.tcx, id, self.output);1589                }1590            }1591            DefKind::Fn => {1592                self.push_if_root(id.owner_id.def_id);1593            }1594            _ => {}1595        }1596    }15971598    fn process_impl_item(&mut self, id: hir::ImplItemId) {1599        if self.tcx.def_kind(id.owner_id) == DefKind::AssocFn {1600            self.push_if_root(id.owner_id.def_id);1601        }1602    }16031604    fn process_nested_body(&mut self, def_id: LocalDefId) {1605        match self.tcx.def_kind(def_id) {1606            DefKind::Closure => {1607                // for 'pub async fn foo(..)' also trying to monomorphize foo::{closure}1608                let is_pub_fn_coroutine =1609                    match *self.tcx.type_of(def_id).instantiate_identity().skip_norm_wip().kind() {1610                        ty::Coroutine(cor_id, _args) => {1611                            let tcx = self.tcx;1612                            let parent_id = tcx.parent(cor_id);1613                            tcx.def_kind(parent_id) == DefKind::Fn1614                                && tcx.asyncness(parent_id).is_async()1615                                && tcx.visibility(parent_id).is_public()1616                        }1617                        ty::Closure(..) | ty::CoroutineClosure(..) => false,1618                        _ => unreachable!(),1619                    };1620                if (self.strategy == MonoItemCollectionStrategy::Eager || is_pub_fn_coroutine)1621                    && !self1622                        .tcx1623                        .generics_of(self.tcx.typeck_root_def_id_local(def_id))1624                        .requires_monomorphization(self.tcx)1625                {1626                    let instance = match *self1627                        .tcx1628                        .type_of(def_id)1629                        .instantiate_identity()1630                        .skip_norm_wip()1631                        .kind()1632                    {1633                        ty::Closure(def_id, args)1634                        | ty::Coroutine(def_id, args)1635                        | ty::CoroutineClosure(def_id, args) => {1636                            Instance::new_raw(def_id, self.tcx.erase_and_anonymize_regions(args))1637                        }1638                        _ => unreachable!(),1639                    };1640                    let Ok(instance) = self.tcx.try_normalize_erasing_regions(1641                        ty::TypingEnv::fully_monomorphized(),1642                        Unnormalized::new_wip(instance),1643                    ) else {1644                        // Don't ICE on an impossible-to-normalize closure.1645                        return;1646                    };1647                    let mono_item = create_fn_mono_item(self.tcx, instance, DUMMY_SP);1648                    if mono_item.node.is_instantiable(self.tcx) {1649                        self.output.push(mono_item);1650                    }1651                }1652            }1653            _ => {}1654        }1655    }16561657    fn is_root(&self, def_id: LocalDefId) -> bool {1658        !self.tcx.generics_of(def_id).requires_monomorphization(self.tcx)1659            && match self.strategy {1660                MonoItemCollectionStrategy::Eager => {1661                    !matches!(self.tcx.codegen_fn_attrs(def_id).inline, InlineAttr::Force { .. })1662                }1663                MonoItemCollectionStrategy::Lazy => {1664                    self.entry_fn.and_then(|(id, _)| id.as_local()) == Some(def_id)1665                        || self.tcx.is_reachable_non_generic(def_id)1666                        || {1667                            let flags = self.tcx.codegen_fn_attrs(def_id).flags;1668                            flags.intersects(1669                                CodegenFnAttrFlags::RUSTC_STD_INTERNAL_SYMBOL1670                                    | CodegenFnAttrFlags::EXTERNALLY_IMPLEMENTABLE_ITEM,1671                            )1672                        }1673                }1674            }1675    }16761677    /// If `def_id` represents a root, pushes it onto the list of1678    /// outputs. (Note that all roots must be monomorphic.)1679    #[instrument(skip(self), level = "debug")]1680    fn push_if_root(&mut self, def_id: LocalDefId) {1681        if self.is_root(def_id) {1682            debug!("found root");16831684            let instance = Instance::mono(self.tcx, def_id.to_def_id());1685            self.output.push(create_fn_mono_item(self.tcx, instance, DUMMY_SP));1686        }1687    }16881689    /// As a special case, when/if we encounter the1690    /// `main()` function, we also have to generate a1691    /// monomorphized copy of the start lang item based on1692    /// the return type of `main`. This is not needed when1693    /// the user writes their own `start` manually.1694    fn push_extra_entry_roots(&mut self) {1695        let Some((main_def_id, EntryFnType::Main { .. })) = self.entry_fn else {1696            return;1697        };16981699        let main_instance = Instance::mono(self.tcx, main_def_id);1700        if self.tcx.should_codegen_locally(main_instance) {1701            self.output.push(create_fn_mono_item(1702                self.tcx,1703                main_instance,1704                self.tcx.def_span(main_def_id),1705            ));1706        }17071708        let Some(start_def_id) = self.tcx.lang_items().start_fn() else {1709            self.tcx.dcx().emit_fatal(errors::StartNotFound);1710        };1711        let main_ret_ty = self.tcx.fn_sig(main_def_id).no_bound_vars().unwrap().output();17121713        // Given that `main()` has no arguments,1714        // then its return type cannot have1715        // late-bound regions, since late-bound1716        // regions must appear in the argument1717        // listing.1718        let main_ret_ty = self.tcx.normalize_erasing_regions(1719            ty::TypingEnv::fully_monomorphized(),1720            Unnormalized::new_wip(main_ret_ty.no_bound_vars().unwrap()),1721        );17221723        let start_instance = Instance::expect_resolve(1724            self.tcx,1725            ty::TypingEnv::fully_monomorphized(),1726            start_def_id,1727            self.tcx.mk_args(&[main_ret_ty.into()]),1728            DUMMY_SP,1729        );17301731        self.output.push(create_fn_mono_item(self.tcx, start_instance, DUMMY_SP));1732    }1733}17341735#[instrument(level = "debug", skip(tcx, output))]1736fn create_mono_items_for_default_impls<'tcx>(1737    tcx: TyCtxt<'tcx>,1738    item: hir::ItemId,1739    output: &mut MonoItems<'tcx>,1740) {1741    let impl_ = tcx.impl_trait_header(item.owner_id);17421743    if impl_.polarity == ty::ImplPolarity::Negative {1744        return;1745    }17461747    if tcx.generics_of(item.owner_id).own_requires_monomorphization() {1748        return;1749    }17501751    // Lifetimes never affect trait selection, so we are allowed to eagerly1752    // instantiate an instance of an impl method if the impl (and method,1753    // which we check below) is only parameterized over lifetime. In that case,1754    // we use the ReErased, which has no lifetime information associated with1755    // it, to validate whether or not the impl is legal to instantiate at all.1756    let only_region_params = |param: &ty::GenericParamDef, _: &_| match param.kind {1757        GenericParamDefKind::Lifetime => tcx.lifetimes.re_erased.into(),1758        GenericParamDefKind::Type { .. } | GenericParamDefKind::Const { .. } => {1759            unreachable!(1760                "`own_requires_monomorphization` check means that \1761                we should have no type/const params"1762            )1763        }1764    };1765    let impl_args = GenericArgs::for_item(tcx, item.owner_id.to_def_id(), only_region_params);1766    let trait_ref = impl_.trait_ref.instantiate(tcx, impl_args).skip_norm_wip();17671768    // Unlike 'lazy' monomorphization that begins by collecting items transitively1769    // called by `main` or other global items, when eagerly monomorphizing impl1770    // items, we never actually check that the predicates of this impl are satisfied1771    // in a empty param env (i.e. with no assumptions).1772    //1773    // Even though this impl has no type or const generic parameters, because we don't1774    // consider higher-ranked predicates such as `for<'a> &'a mut [u8]: Copy` to1775    // be trivially false. We must now check that the impl has no impossible-to-satisfy1776    // predicates.1777    if tcx.instantiate_and_check_impossible_predicates((item.owner_id.to_def_id(), impl_args)) {1778        return;1779    }17801781    let typing_env = ty::TypingEnv::fully_monomorphized();1782    let trait_ref = tcx.normalize_erasing_regions(typing_env, Unnormalized::new_wip(trait_ref));1783    let overridden_methods = tcx.impl_item_implementor_ids(item.owner_id);1784    for method in tcx.provided_trait_methods(trait_ref.def_id) {1785        if overridden_methods.contains_key(&method.def_id) {1786            continue;1787        }17881789        if tcx.generics_of(method.def_id).own_requires_monomorphization() {1790            continue;1791        }17921793        // As mentioned above, the method is legal to eagerly instantiate if it1794        // only has lifetime generic parameters. This is validated by calling1795        // `own_requires_monomorphization` on both the impl and method.1796        let args = trait_ref.args.extend_to(tcx, method.def_id, only_region_params);1797        let instance = ty::Instance::expect_resolve(tcx, typing_env, method.def_id, args, DUMMY_SP);17981799        let mono_item = create_fn_mono_item(tcx, instance, DUMMY_SP);1800        if mono_item.node.is_instantiable(tcx) && tcx.should_codegen_locally(instance) {1801            output.push(mono_item);1802        }1803    }1804}18051806//=-----------------------------------------------------------------------------1807// Top-level entry point, tying it all together1808//=-----------------------------------------------------------------------------18091810#[instrument(skip(tcx, strategy), level = "debug")]1811pub(crate) fn collect_crate_mono_items<'tcx>(1812    tcx: TyCtxt<'tcx>,1813    strategy: MonoItemCollectionStrategy,1814) -> (Vec<MonoItem<'tcx>>, UsageMap<'tcx>) {1815    let _prof_timer = tcx.prof.generic_activity("monomorphization_collector");18161817    let roots = tcx1818        .sess1819        .time("monomorphization_collector_root_collections", || collect_roots(tcx, strategy));18201821    debug!("building mono item graph, beginning at roots");18221823    let state = SharedState {1824        visited: Lock::new(UnordSet::default()),1825        mentioned: Lock::new(UnordSet::default()),1826        usage_map: Lock::new(UsageMap::new()),1827    };1828    let recursion_limit = tcx.recursion_limit();18291830    tcx.sess.time("monomorphization_collector_graph_walk", || {1831        par_for_each_in(roots, |root| {1832            collect_items_root(tcx, dummy_spanned(*root), &state, recursion_limit);1833        });1834    });18351836    // The set of MonoItems was created in an inherently indeterministic order because1837    // of parallelism. We sort it here to ensure that the output is deterministic.1838    let mono_items = tcx.with_stable_hashing_context(move |mut hcx| {1839        state.visited.into_inner().into_sorted(&mut hcx, true)1840    });18411842    (mono_items, state.usage_map.into_inner())1843}18441845pub(crate) fn provide(providers: &mut Providers) {1846    providers.hooks.should_codegen_locally = should_codegen_locally;1847    providers.queries.items_of_instance = items_of_instance;1848}