1//! Mono Item Collection2//! ====================3//!4//! This module is responsible for discovering all items that will contribute5//! to code generation of the crate. The important part here is that it not only6//! needs to find syntax-level items (functions, structs, etc) but also all7//! their monomorphized instantiations. Every non-generic, non-const function8//! maps to one LLVM artifact. Every generic function can produce9//! from zero to N artifacts, depending on the sets of type arguments it10//! is instantiated with.11//! This also applies to generic items from other crates: A generic definition12//! in crate X might produce monomorphizations that are compiled into crate Y.13//! We also have to collect these here.14//!15//! The following kinds of "mono items" are handled here:16//!17//! - Functions18//! - Methods19//! - Closures20//! - Statics21//! - Drop glue22//!23//! The following things also result in LLVM artifacts, but are not collected24//! here, since we instantiate them locally on demand when needed in a given25//! codegen unit:26//!27//! - Constants28//! - VTables29//! - Object Shims30//!31//! The main entry point is `collect_crate_mono_items`, at the bottom of this file.32//!33//! General Algorithm34//! -----------------35//! Let's define some terms first:36//!37//! - A "mono item" is something that results in a function or global in38//!   the LLVM IR of a codegen unit. Mono items do not stand on their39//!   own, they can use other mono items. For example, if function40//!   `foo()` calls function `bar()` then the mono item for `foo()`41//!   uses the mono item for function `bar()`. In general, the42//!   definition for mono item A using a mono item B is that43//!   the LLVM artifact produced for A uses the LLVM artifact produced44//!   for B.45//!46//! - Mono items and the uses between them form a directed graph,47//!   where the mono items are the nodes and uses form the edges.48//!   Let's call this graph the "mono item graph".49//!50//! - The mono item graph for a program contains all mono items51//!   that are needed in order to produce the complete LLVM IR of the program.52//!53//! The purpose of the algorithm implemented in this module is to build the54//! mono item graph for the current crate. It runs in two phases:55//!56//! 1. Discover the roots of the graph by traversing the HIR of the crate.57//! 2. Starting from the roots, find uses by inspecting the MIR58//!    representation of the item corresponding to a given node, until no more59//!    new nodes are found.60//!61//! ### Discovering roots62//! The roots of the mono item graph correspond to the public non-generic63//! syntactic items in the source code. We find them by walking the HIR of the64//! crate, and whenever we hit upon a public function, method, or static item,65//! we create a mono item consisting of the items DefId and, since we only66//! consider non-generic items, an empty type-parameters set. (In eager67//! collection mode, during incremental compilation, all non-generic functions68//! are considered as roots, as well as when the `-Clink-dead-code` option is69//! specified. Functions marked `#[no_mangle]` and functions called by inlinable70//! functions also always act as roots.)71//!72//! ### Finding uses73//! Given a mono item node, we can discover uses by inspecting its MIR. We walk74//! the MIR to find other mono items used by each mono item. Since the mono75//! item we are currently at is always monomorphic, we also know the concrete76//! type arguments of its used mono items. The specific forms a use can take in77//! MIR are quite diverse. Here is an overview:78//!79//! #### Calling Functions/Methods80//! The most obvious way for one mono item to use another is a81//! function or method call (represented by a CALL terminator in MIR). But82//! calls are not the only thing that might introduce a use between two83//! function mono items, and as we will see below, they are just a84//! specialization of the form described next, and consequently will not get any85//! special treatment in the algorithm.86//!87//! #### Taking a reference to a function or method88//! A function does not need to actually be called in order to be used by89//! another function. It suffices to just take a reference in order to introduce90//! an edge. Consider the following example:91//!92//! ```93//! # use core::fmt::Display;94//! fn print_val<T: Display>(x: T) {95//!     println!("{}", x);96//! }97//!98//! fn call_fn(f: &dyn Fn(i32), x: i32) {99//!     f(x);100//! }101//!102//! fn main() {103//!     let print_i32 = print_val::<i32>;104//!     call_fn(&print_i32, 0);105//! }106//! ```107//! The MIR of none of these functions will contain an explicit call to108//! `print_val::<i32>`. Nonetheless, in order to mono this program, we need109//! an instance of this function. Thus, whenever we encounter a function or110//! method in operand position, we treat it as a use of the current111//! mono item. Calls are just a special case of that.112//!113//! #### Drop glue114//! Drop glue mono items are introduced by MIR drop-statements. The115//! generated mono item will have additional drop-glue item uses if the116//! type to be dropped contains nested values that also need to be dropped. It117//! might also have a function item use for the explicit `Drop::drop`118//! implementation of its type.119//!120//! #### Unsizing Casts121//! A subtle way of introducing use edges is by casting to a trait object.122//! Since the resulting wide-pointer contains a reference to a vtable, we need to123//! instantiate all dyn-compatible methods of the trait, as we need to store124//! pointers to these functions even if they never get called anywhere. This can125//! be seen as a special case of taking a function reference.126//!127//!128//! Interaction with Cross-Crate Inlining129//! -------------------------------------130//! The binary of a crate will not only contain machine code for the items131//! defined in the source code of that crate. It will also contain monomorphic132//! instantiations of any extern generic functions and of functions marked with133//! `#[inline]`.134//! The collection algorithm handles this more or less mono. If it is135//! about to create a mono item for something with an external `DefId`,136//! it will take a look if the MIR for that item is available, and if so just137//! proceed normally. If the MIR is not available, it assumes that the item is138//! just linked to and no node is created; which is exactly what we want, since139//! no machine code should be generated in the current crate for such an item.140//!141//! Eager and Lazy Collection Strategy142//! ----------------------------------143//! Mono item collection can be performed with one of two strategies:144//!145//! - Lazy strategy means that items will only be instantiated when actually146//!   used. The goal is to produce the least amount of machine code147//!   possible.148//!149//! - Eager strategy is meant to be used in conjunction with incremental compilation150//!   where a stable set of mono items is more important than a minimal151//!   one. Thus, eager strategy will instantiate drop-glue for every drop-able type152//!   in the crate, even if no drop call for that type exists (yet). It will153//!   also instantiate default implementations of trait methods, something that154//!   otherwise is only done on demand.155//!156//! Collection-time const evaluation and "mentioned" items157//! ------------------------------------------------------158//!159//! One important role of collection is to evaluate all constants that are used by all the items160//! which are being collected. Codegen can then rely on only encountering constants that evaluate161//! successfully, and if a constant fails to evaluate, the collector has much better context to be162//! able to show where this constant comes up.163//!164//! However, the exact set of "used" items (collected as described above), and therefore the exact165//! set of used constants, can depend on optimizations. Optimizing away dead code may optimize away166//! a function call that uses a failing constant, so an unoptimized build may fail where an167//! optimized build succeeds. This is undesirable.168//!169//! To avoid this, the collector has the concept of "mentioned" items. Some time during the MIR170//! pipeline, before any optimization-level-dependent optimizations, we compute a list of all items171//! that syntactically appear in the code. These are considered "mentioned", and even if they are in172//! dead code and get optimized away (which makes them no longer "used"), they are still173//! "mentioned". For every used item, the collector ensures that all mentioned items, recursively,174//! do not use a failing constant. This is reflected via the [`CollectionMode`], which determines175//! whether we are visiting a used item or merely a mentioned item.176//!177//! The collector and "mentioned items" gathering (which lives in `rustc_mir_transform::mentioned_items`)178//! need to stay in sync in the following sense:179//!180//! - For every item that the collector gather that could eventually lead to build failure (most181//!   likely due to containing a constant that fails to evaluate), a corresponding mentioned item182//!   must be added. This should use the exact same strategy as the ecollector to make sure they are183//!   in sync. However, while the collector works on monomorphized types, mentioned items are184//!   collected on generic MIR -- so any time the collector checks for a particular type (such as185//!   `ty::FnDef`), we have to just onconditionally add this as a mentioned item.186//! - In `visit_mentioned_item`, we then do with that mentioned item exactly what the collector187//!   would have done during regular MIR visiting. Basically you can think of the collector having188//!   two stages, a pre-monomorphization stage and a post-monomorphization stage (usually quite189//!   literally separated by a call to `self.monomorphize`); the pre-monomorphizationn stage is190//!   duplicated in mentioned items gathering and the post-monomorphization stage is duplicated in191//!   `visit_mentioned_item`.192//! - Finally, as a performance optimization, the collector should fill `used_mentioned_item` during193//!   its MIR traversal with exactly what mentioned item gathering would have added in the same194//!   situation. This detects mentioned items that have *not* been optimized away and hence don't195//!   need a dedicated traversal.196//!197//! Open Issues198//! -----------199//! Some things are not yet fully implemented in the current version of this200//! module.201//!202//! ### Const Fns203//! Ideally, no mono item should be generated for const fns unless there204//! is a call to them that cannot be evaluated at compile time. At the moment205//! this is not implemented however: a mono item will be produced206//! regardless of whether it is actually needed or not.207208use std::cell::OnceCell;209use std::ops::ControlFlow;210211use rustc_attr_ir::InlineAttr;212use rustc_attr_ir::lang_items::LangItem;213use rustc_data_structures::fx::FxIndexMap;214use rustc_data_structures::sync::{Lock, par_for_each_in};215use rustc_data_structures::unord::{UnordMap, UnordSet};216use rustc_hir as hir;217use rustc_hir::def::DefKind;218use rustc_hir::def_id::{DefId, DefIdMap, LocalDefId};219use rustc_middle::middle::codegen_fn_attrs::CodegenFnAttrFlags;220use rustc_middle::mir::interpret::{AllocId, ErrorHandled, GlobalAlloc, Scalar};221use rustc_middle::mir::visit::Visitor as MirVisitor;222use rustc_middle::mir::{self, Body, Location, MentionedItem, traversal};223use rustc_middle::mono::{CollectionMode, InstantiationMode, MonoItem, NormalizationErrorInMono};224use rustc_middle::query::TyCtxtAt;225use rustc_middle::ty::adjustment::{CustomCoerceUnsized, PointerCoercion};226use rustc_middle::ty::layout::ValidityRequirement;227use rustc_middle::ty::{228    self, GenericArgs, GenericParamDefKind, Instance, InstanceKind, ShimKind, Ty, TyCtxt,229    TypeFoldable, TypeVisitable, TypeVisitableExt, TypeVisitor, Unnormalized, VtblEntry,230};231use rustc_middle::util::Providers;232use rustc_session::config::{DebugInfo, EntryFnType, Offload};233use rustc_span::{DUMMY_SP, Span, Spanned, Symbol, bug, dummy_spanned, respan, span_bug};234use rustc_structures::Limit;235use tracing::{debug, instrument, trace};236237use crate::diagnostics::{238    self, EncounteredErrorWhileInstantiating, EncounteredErrorWhileInstantiatingGlobalAsm,239    NoOptimizedMir, RecursionLimit,240};241242#[derive(PartialEq)]243pub(crate) enum MonoItemCollectionStrategy {244    Eager,245    Lazy,246}247248/// The state that is shared across the concurrent threads that are doing collection.249struct SharedState<'tcx> {250    /// Items that have been or are currently being recursively collected.251    visited: Lock<UnordSet<MonoItem<'tcx>>>,252    /// Items that have been or are currently being recursively treated as "mentioned", i.e., their253    /// consts are evaluated but nothing is added to the collection.254    mentioned: Lock<UnordSet<MonoItem<'tcx>>>,255    /// Which items are being used where, for better errors.256    usage_map: Lock<UsageMap<'tcx>>,257}258259pub(crate) struct UsageMap<'tcx> {260    // Maps every mono item to the mono items used by it.261    pub used_map: UnordMap<MonoItem<'tcx>, Vec<MonoItem<'tcx>>>,262263    // Maps each mono item with users to the mono items that use it.264    // Be careful: subsets `used_map`, so unused items are vacant.265    user_map: UnordMap<MonoItem<'tcx>, Vec<MonoItem<'tcx>>>,266}267268impl<'tcx> UsageMap<'tcx> {269    fn new() -> UsageMap<'tcx> {270        UsageMap { used_map: Default::default(), user_map: Default::default() }271    }272273    fn record_used<'a>(&mut self, user_item: MonoItem<'tcx>, used_items: &'a MonoItems<'tcx>)274    where275        'tcx: 'a,276    {277        for used_item in used_items.items() {278            self.user_map.entry(used_item).or_default().push(user_item);279        }280281        assert!(self.used_map.insert(user_item, used_items.items().collect()).is_none());282    }283284    pub(crate) fn get_user_items(&self, item: MonoItem<'tcx>) -> &[MonoItem<'tcx>] {285        self.user_map.get(&item).map(|items| items.as_slice()).unwrap_or(&[])286    }287288    /// Internally iterate over all inlined items used by `item`.289    pub(crate) fn for_each_inlined_used_item<F>(290        &self,291        tcx: TyCtxt<'tcx>,292        item: MonoItem<'tcx>,293        mut f: F,294    ) where295        F: FnMut(MonoItem<'tcx>),296    {297        let used_items = self.used_map.get(&item).unwrap();298        for used_item in used_items.iter() {299            let is_inlined = used_item.instantiation_mode(tcx) == InstantiationMode::LocalCopy;300            if is_inlined {301                f(*used_item);302            }303        }304    }305}306307struct MonoItems<'tcx> {308    // We want a set of MonoItem + Span where trying to re-insert a MonoItem with a different Span309    // is ignored. Map does that, but it looks odd.310    items: FxIndexMap<MonoItem<'tcx>, Span>,311}312313impl<'tcx> MonoItems<'tcx> {314    fn new() -> Self {315        Self { items: FxIndexMap::default() }316    }317318    fn is_empty(&self) -> bool {319        self.items.is_empty()320    }321322    fn push(&mut self, item: Spanned<MonoItem<'tcx>>) {323        // Insert only if the entry does not exist. A normal insert would stomp the first span that324        // got inserted.325        self.items.entry(item.node).or_insert(item.span);326    }327328    fn items(&self) -> impl Iterator<Item = MonoItem<'tcx>> {329        self.items.keys().cloned()330    }331}332333impl<'tcx> IntoIterator for MonoItems<'tcx> {334    type Item = Spanned<MonoItem<'tcx>>;335    type IntoIter = impl Iterator<Item = Spanned<MonoItem<'tcx>>>;336337    fn into_iter(self) -> Self::IntoIter {338        self.items.into_iter().map(|(item, span)| respan(span, item))339    }340}341342impl<'tcx> Extend<Spanned<MonoItem<'tcx>>> for MonoItems<'tcx> {343    fn extend<I>(&mut self, iter: I)344    where345        I: IntoIterator<Item = Spanned<MonoItem<'tcx>>>,346    {347        let iter = iter.into_iter();348        self.items.reserve(iter.size_hint().0);349        for item in iter {350            self.push(item)351        }352    }353}354355fn collect_items_root<'tcx>(356    tcx: TyCtxt<'tcx>,357    starting_item: Spanned<MonoItem<'tcx>>,358    state: &SharedState<'tcx>,359    recursion_limit: Limit,360) {361    if !state.visited.lock().insert(starting_item.node) {362        // We've been here already, no need to search again.363        return;364    }365    let mut recursion_depths = DefIdMap::default();366    collect_items_rec(367        tcx,368        starting_item,369        state,370        &mut recursion_depths,371        recursion_limit,372        CollectionMode::UsedItems,373    );374}375376/// Collect all monomorphized items reachable from `starting_point`, and emit a note diagnostic if a377/// post-monomorphization error is encountered during a collection step.378///379/// `mode` determined whether we are scanning for [used items][CollectionMode::UsedItems]380/// or [mentioned items][CollectionMode::MentionedItems].381#[instrument(skip(tcx, state, recursion_depths, recursion_limit), level = "debug")]382fn collect_items_rec<'tcx>(383    tcx: TyCtxt<'tcx>,384    starting_item: Spanned<MonoItem<'tcx>>,385    state: &SharedState<'tcx>,386    recursion_depths: &mut DefIdMap<usize>,387    recursion_limit: Limit,388    mode: CollectionMode,389) {390    let mut used_items = MonoItems::new();391    let mut mentioned_items = MonoItems::new();392    let recursion_depth_reset;393394    // Post-monomorphization errors MVP395    //396    // We can encounter errors while monomorphizing an item, but we don't have a good way of397    // showing a complete stack of spans ultimately leading to collecting the erroneous one yet.398    // (It's also currently unclear exactly which diagnostics and information would be interesting399    // to report in such cases)400    //401    // This leads to suboptimal error reporting: a post-monomorphization error (PME) will be402    // shown with just a spanned piece of code causing the error, without information on where403    // it was called from. This is especially obscure if the erroneous mono item is in a404    // dependency. See for example issue #85155, where, before minimization, a PME happened two405    // crates downstream from libcore's stdarch, without a way to know which dependency was the406    // cause.407    //408    // If such an error occurs in the current crate, its span will be enough to locate the409    // source. If the cause is in another crate, the goal here is to quickly locate which mono410    // item in the current crate is ultimately responsible for causing the error.411    //412    // To give at least _some_ context to the user: while collecting mono items, we check the413    // error count. If it has changed, a PME occurred, and we trigger some diagnostics about the414    // current step of mono items collection.415    //416    // FIXME: don't rely on global state, instead bubble up errors. Note: this is very hard to do.417    let error_count = tcx.dcx().err_count_on_current_thread();418419    // In `mentioned_items` we collect items that were mentioned in this MIR but possibly do not420    // need to be monomorphized. This is done to ensure that optimizing away function calls does not421    // hide const-eval errors that those calls would otherwise have triggered.422    match starting_item.node {423        MonoItem::Static(def_id) => {424            recursion_depth_reset = None;425426            // Statics always get evaluated (which is possible because they can't be generic), so for427            // `MentionedItems` collection there's nothing to do here.428            if mode == CollectionMode::UsedItems {429                let instance = Instance::mono(tcx, def_id);430431                // Sanity check whether this ended up being collected accidentally432                debug_assert!(tcx.should_codegen_locally(instance));433434                let DefKind::Static { nested, .. } = tcx.def_kind(def_id) else { bug!() };435                // Nested statics have no type.436                if !nested {437                    let ty = instance.ty(tcx, ty::TypingEnv::fully_monomorphized());438                    visit_drop_use(tcx, ty, true, starting_item.span, &mut used_items);439                }440441                if let Ok(alloc) = tcx.eval_static_initializer(def_id) {442                    for &prov in alloc.inner().provenance().ptrs().values() {443                        collect_alloc(tcx, prov.alloc_id(), &mut used_items);444                    }445                }446447                if tcx.needs_thread_local_shim(def_id) {448                    used_items.push(respan(449                        starting_item.span,450                        MonoItem::Fn(Instance {451                            def: InstanceKind::Shim(ShimKind::ThreadLocal(def_id)),452                            args: GenericArgs::empty(),453                        }),454                    ));455                }456            }457458            // mentioned_items stays empty since there's no codegen for statics. statics don't get459            // optimized, and if they did then the const-eval interpreter would have to worry about460            // mentioned_items.461        }462        MonoItem::Fn(instance) => {463            // Sanity check whether this ended up being collected accidentally464            debug_assert!(tcx.should_codegen_locally(instance));465466            // Keep track of the monomorphization recursion depth467            recursion_depth_reset = Some(check_recursion_limit(468                tcx,469                instance,470                starting_item.span,471                recursion_depths,472                recursion_limit,473            ));474475            let Ok((used, mentioned)) = tcx.items_of_instance((instance, mode)) else {476                // Normalization errors here are usually due to trait solving overflow.477                // FIXME: I assume that there are few type errors at post-analysis stage, but not478                // entirely sure.479                // We have to emit the error outside of `items_of_instance` to access the480                // span of the `starting_item`.481                let def_id = instance.def_id();482                let def_span = tcx.def_span(def_id);483                let def_path_str = tcx.def_path_str(def_id);484                tcx.dcx().emit_fatal(RecursionLimit {485                    span: starting_item.span,486                    instance,487                    def_span,488                    def_path_str,489                });490            };491            used_items.extend(used.into_iter().copied());492            mentioned_items.extend(mentioned.into_iter().copied());493        }494        MonoItem::GlobalAsm(item_id) => {495            assert!(496                mode == CollectionMode::UsedItems,497                "should never encounter global_asm when collecting mentioned items"498            );499            recursion_depth_reset = None;500501            let item = tcx.hir_item(item_id);502            if let hir::ItemKind::GlobalAsm { asm, .. } = item.kind {503                for (op, op_sp) in asm.operands {504                    match *op {505                        hir::InlineAsmOperand::Const { anon_const } => {506                            match tcx.const_eval_poly(anon_const.def_id.to_def_id()) {507                                Ok(val) => {508                                    collect_const_value(tcx, val, &mut used_items);509                                }510                                Err(ErrorHandled::TooGeneric(..)) => {511                                    span_bug!(*op_sp, "asm const cannot be resolved; too generic")512                                }513                                Err(ErrorHandled::Reported(..)) => {514                                    continue;515                                }516                            }517                        }518                        hir::InlineAsmOperand::SymFn { expr } => {519                            let fn_ty = tcx.typeck(item_id.owner_id).expr_ty(expr);520                            visit_fn_use(tcx, fn_ty, false, *op_sp, &mut used_items);521                        }522                        hir::InlineAsmOperand::SymStatic { path: _, def_id } => {523                            let instance = Instance::mono(tcx, def_id);524                            if tcx.should_codegen_locally(instance) {525                                trace!("collecting static {:?}", def_id);526                                used_items.push(dummy_spanned(MonoItem::Static(def_id)));527                            }528                        }529                        hir::InlineAsmOperand::In { .. }530                        | hir::InlineAsmOperand::Out { .. }531                        | hir::InlineAsmOperand::InOut { .. }532                        | hir::InlineAsmOperand::SplitInOut { .. }533                        | hir::InlineAsmOperand::Label { .. } => {534                            span_bug!(*op_sp, "invalid operand type for global_asm!")535                        }536                    }537                }538            } else {539                span_bug!(item.span, "Mismatch between hir::Item type and MonoItem type")540            }541542            // mention_items stays empty as nothing gets optimized here.543        }544    };545546    // Check for PMEs and emit a diagnostic if one happened. To try to show relevant edges of the547    // mono item graph.548    if tcx.dcx().err_count_on_current_thread() > error_count549        && starting_item.node.is_generic_fn()550        && starting_item.node.is_user_defined()551    {552        match starting_item.node {553            MonoItem::Fn(instance) => tcx.dcx().emit_note(EncounteredErrorWhileInstantiating {554                span: starting_item.span,555                kind: "fn",556                instance,557            }),558            MonoItem::Static(def_id) => tcx.dcx().emit_note(EncounteredErrorWhileInstantiating {559                span: starting_item.span,560                kind: "static",561                instance: Instance::new_raw(def_id, GenericArgs::empty()),562            }),563            MonoItem::GlobalAsm(_) => {564                tcx.dcx().emit_note(EncounteredErrorWhileInstantiatingGlobalAsm {565                    span: starting_item.span,566                })567            }568        }569    }570    // Only updating `usage_map` for used items as otherwise we may be inserting the same item571    // multiple times (if it is first 'mentioned' and then later actually used), and the usage map572    // logic does not like that.573    // This is part of the output of collection and hence only relevant for "used" items.574    // ("Mentioned" items are only considered internally during collection.)575    if mode == CollectionMode::UsedItems {576        state.usage_map.lock().record_used(starting_item.node, &used_items);577    }578579    {580        let mut visited = OnceCell::default();581        if mode == CollectionMode::UsedItems {582            used_items583                .items584                .retain(|k, _| visited.get_mut_or_init(|| state.visited.lock()).insert(*k));585        }586587        let mut mentioned = OnceCell::default();588        mentioned_items.items.retain(|k, _| {589            !visited.get_or_init(|| state.visited.lock()).contains(k)590                && mentioned.get_mut_or_init(|| state.mentioned.lock()).insert(*k)591        });592    }593    if mode == CollectionMode::MentionedItems {594        assert!(used_items.is_empty(), "'mentioned' collection should never encounter used items");595    } else {596        for used_item in used_items {597            collect_items_rec(598                tcx,599                used_item,600                state,601                recursion_depths,602                recursion_limit,603                CollectionMode::UsedItems,604            );605        }606    }607608    // Walk over mentioned items *after* used items, so that if an item is both mentioned and used then609    // the loop above has fully collected it, so this loop will skip it.610    for mentioned_item in mentioned_items {611        collect_items_rec(612            tcx,613            mentioned_item,614            state,615            recursion_depths,616            recursion_limit,617            CollectionMode::MentionedItems,618        );619    }620621    if let Some((def_id, depth)) = recursion_depth_reset {622        recursion_depths.insert(def_id, depth);623    }624}625626// Check whether we can normalize every type in the instantiated MIR body.627fn check_normalization_error<'tcx>(628    tcx: TyCtxt<'tcx>,629    instance: Instance<'tcx>,630    body: &Body<'tcx>,631) -> Result<(), NormalizationErrorInMono> {632    struct NormalizationChecker<'tcx> {633        tcx: TyCtxt<'tcx>,634        instance: Instance<'tcx>,635    }636    impl<'tcx> TypeVisitor<TyCtxt<'tcx>> for NormalizationChecker<'tcx> {637        type Result = ControlFlow<()>;638639        fn visit_ty(&mut self, t: Ty<'tcx>) -> Self::Result {640            match self.instance.try_instantiate_mir_and_normalize_erasing_regions(641                self.tcx,642                ty::TypingEnv::fully_monomorphized(),643                ty::EarlyBinder::bind(self.tcx, t),644            ) {645                Ok(_) => ControlFlow::Continue(()),646                Err(_) => ControlFlow::Break(()),647            }648        }649    }650651    let mut checker = NormalizationChecker { tcx, instance };652    if body.visit_with(&mut checker).is_break() { Err(NormalizationErrorInMono) } else { Ok(()) }653}654655fn check_recursion_limit<'tcx>(656    tcx: TyCtxt<'tcx>,657    instance: Instance<'tcx>,658    span: Span,659    recursion_depths: &mut DefIdMap<usize>,660    recursion_limit: Limit,661) -> (DefId, usize) {662    let def_id = instance.def_id();663    let recursion_depth = recursion_depths.get(&def_id).cloned().unwrap_or(0);664    debug!(" => recursion depth={}", recursion_depth);665666    let adjusted_recursion_depth = if tcx.is_lang_item(def_id, LangItem::DropGlue) {667        // HACK: `drop_glue` creates tight monomorphization loops. Give668        // it more margin.669        recursion_depth / 4670    } else {671        recursion_depth672    };673674    // Code that needs to instantiate the same function recursively675    // more than the recursion limit is assumed to be causing an676    // infinite expansion.677    if !recursion_limit.value_within_limit(adjusted_recursion_depth) {678        let def_span = tcx.def_span(def_id);679        let def_path_str = tcx.def_path_str(def_id);680        tcx.dcx().emit_fatal(RecursionLimit { span, instance, def_span, def_path_str });681    }682683    recursion_depths.insert(def_id, recursion_depth + 1);684685    (def_id, recursion_depth)686}687688struct MirUsedCollector<'a, 'tcx> {689    tcx: TyCtxt<'tcx>,690    body: &'a mir::Body<'tcx>,691    used_items: &'a mut MonoItems<'tcx>,692    /// See the comment in `collect_items_of_instance` for the purpose of this set.693    /// Note that this contains *not-monomorphized* items!694    used_mentioned_items: &'a mut UnordSet<MentionedItem<'tcx>>,695    instance: Instance<'tcx>,696}697698impl<'a, 'tcx> MirUsedCollector<'a, 'tcx> {699    fn monomorphize<T>(&self, value: T) -> T700    where701        T: TypeFoldable<TyCtxt<'tcx>>,702    {703        trace!("monomorphize: self.instance={:?}", self.instance);704        self.instance.instantiate_mir_and_normalize_erasing_regions(705            self.tcx,706            ty::TypingEnv::fully_monomorphized(),707            ty::EarlyBinder::bind(self.tcx, value),708        )709    }710711    /// Evaluates a *not yet monomorphized* constant.712    fn eval_constant(&mut self, constant: &mir::ConstOperand<'tcx>) -> Option<mir::ConstValue> {713        let const_ = self.monomorphize(constant.const_);714        // Evaluate the constant. This makes const eval failure a collection-time error (rather than715        // a codegen-time error). rustc stops after collection if there was an error, so this716        // ensures codegen never has to worry about failing consts.717        // (codegen relies on this and ICEs will happen if this is violated.)718        match const_.eval(self.tcx, ty::TypingEnv::fully_monomorphized(), constant.span) {719            Ok(v) => Some(v),720            Err(ErrorHandled::TooGeneric(..)) => span_bug!(721                constant.span,722                "collection encountered polymorphic constant: {:?}",723                const_724            ),725            Err(err @ ErrorHandled::Reported(..)) => {726                err.emit_note(self.tcx);727                return None;728            }729        }730    }731}732733impl<'a, 'tcx> MirVisitor<'tcx> for MirUsedCollector<'a, 'tcx> {734    fn visit_rvalue(&mut self, rvalue: &mir::Rvalue<'tcx>, location: Location) {735        debug!("visiting rvalue {:?}", *rvalue);736737        let span = self.body.source_info(location).span;738739        match *rvalue {740            // When doing an cast from a regular pointer to a wide pointer, we741            // have to instantiate all methods of the trait being cast to, so we742            // can build the appropriate vtable.743            mir::Rvalue::Cast(744                mir::CastKind::PointerCoercion(PointerCoercion::Unsize, _),745                ref operand,746                target_ty,747            ) => {748                let source_ty = operand.ty(self.body, self.tcx);749                // *Before* monomorphizing, record that we already handled this mention.750                self.used_mentioned_items751                    .insert(MentionedItem::UnsizeCast { source_ty, target_ty });752                let target_ty = self.monomorphize(target_ty);753                let source_ty = self.monomorphize(source_ty);754                let (source_ty, target_ty) =755                    find_tails_for_unsizing(self.tcx.at(span), source_ty, target_ty);756                // This could also be a different Unsize instruction, like757                // from a fixed sized array to a slice. But we are only758                // interested in things that produce a vtable.759                if target_ty.is_trait() && !source_ty.is_trait() {760                    create_mono_items_for_vtable_methods(761                        self.tcx,762                        target_ty,763                        source_ty,764                        span,765                        self.used_items,766                    );767                }768            }769            mir::Rvalue::Cast(770                mir::CastKind::PointerCoercion(PointerCoercion::ReifyFnPointer(_), _),771                ref operand,772                _,773            ) => {774                let fn_ty = operand.ty(self.body, self.tcx);775                // *Before* monomorphizing, record that we already handled this mention.776                self.used_mentioned_items.insert(MentionedItem::Fn(fn_ty));777                let fn_ty = self.monomorphize(fn_ty);778                visit_fn_use(self.tcx, fn_ty, false, span, self.used_items);779            }780            mir::Rvalue::Cast(781                mir::CastKind::PointerCoercion(PointerCoercion::ClosureFnPointer(_), _),782                ref operand,783                _,784            ) => {785                let source_ty = operand.ty(self.body, self.tcx);786                // *Before* monomorphizing, record that we already handled this mention.787                self.used_mentioned_items.insert(MentionedItem::Closure(source_ty));788                let source_ty = self.monomorphize(source_ty);789                if let ty::Closure(def_id, args) = *source_ty.kind() {790                    let instance =791                        Instance::resolve_closure(self.tcx, def_id, args, ty::ClosureKind::FnOnce);792                    if self.tcx.should_codegen_locally(instance) {793                        self.used_items.push(create_fn_mono_item(self.tcx, instance, span));794                    }795                } else {796                    bug!()797                }798            }799            mir::Rvalue::ThreadLocalRef(def_id) => {800                assert!(self.tcx.is_thread_local_static(def_id));801                let instance = Instance::mono(self.tcx, def_id);802                if self.tcx.should_codegen_locally(instance) {803                    trace!("collecting thread-local static {:?}", def_id);804                    self.used_items.push(respan(span, MonoItem::Static(def_id)));805                }806            }807            _ => { /* not interesting */ }808        }809810        self.super_rvalue(rvalue, location);811    }812813    /// This does not walk the MIR of the constant as that is not needed for codegen, all we need is814    /// to ensure that the constant evaluates successfully and walk the result.815    #[instrument(skip(self), level = "debug")]816    fn visit_const_operand(&mut self, constant: &mir::ConstOperand<'tcx>, _location: Location) {817        // No `super_constant` as we don't care about `visit_ty`/`visit_ty_const`.818        let Some(val) = self.eval_constant(constant) else { return };819        collect_const_value(self.tcx, val, self.used_items);820    }821822    fn visit_terminator(&mut self, terminator: &mir::Terminator<'tcx>, location: Location) {823        debug!("visiting terminator {:?} @ {:?}", terminator, location);824        let source = self.body.source_info(location).span;825826        let tcx = self.tcx;827        let push_mono_lang_item = |this: &mut Self, lang_item: LangItem| {828            let instance = Instance::mono(tcx, tcx.require_lang_item(lang_item, source));829            if tcx.should_codegen_locally(instance) {830                this.used_items.push(create_fn_mono_item(tcx, instance, source));831            }832        };833834        match terminator.kind {835            mir::TerminatorKind::Call { ref func, ref args, .. }836            | mir::TerminatorKind::TailCall { ref func, ref args, .. } => {837                let callee_ty = func.ty(self.body, tcx);838                // *Before* monomorphizing, record that we already handled this mention.839                self.used_mentioned_items.insert(MentionedItem::Fn(callee_ty));840                let callee_ty = self.monomorphize(callee_ty);841842                // HACK(explicit_tail_calls): collect tail calls to `#[track_caller]` functions as indirect,843                // because we later call them as such, to prevent issues with ABI incompatibility.844                // Ideally we'd replace such tail calls with normal call + return, but this requires845                // post-mono MIR optimizations, which we don't yet have.846                let force_indirect_call =847                    if matches!(terminator.kind, mir::TerminatorKind::TailCall { .. })848                        && let &ty::FnDef(def_id, args) = callee_ty.kind()849                        && let instance = ty::Instance::expect_resolve(850                            self.tcx,851                            ty::TypingEnv::fully_monomorphized(),852                            def_id,853                            args.no_bound_vars().unwrap(),854                            source,855                        )856                        && instance.def.requires_caller_location(self.tcx)857                    {858                        true859                    } else {860                        false861                    };862863                visit_fn_use(864                    self.tcx,865                    callee_ty,866                    !force_indirect_call,867                    source,868                    &mut self.used_items,869                );870871                if let ty::FnDef(def_id, _) = *callee_ty.kind()872                    && self.tcx.is_intrinsic(def_id, rustc_span::sym::offload)873                    && let Some(kernel) = args.first()874                {875                    let kernel_ty = kernel.node.ty(self.body, self.tcx);876                    let kernel_ty = self.monomorphize(kernel_ty);877                    visit_fn_use(self.tcx, kernel_ty, false, source, &mut self.used_items);878                }879            }880            mir::TerminatorKind::Drop { ref place, .. } => {881                let ty = place.ty(self.body, self.tcx).ty;882                // *Before* monomorphizing, record that we already handled this mention.883                self.used_mentioned_items.insert(MentionedItem::Drop(ty));884                let ty = self.monomorphize(ty);885                visit_drop_use(self.tcx, ty, true, source, self.used_items);886            }887            mir::TerminatorKind::InlineAsm { ref operands, .. } => {888                for op in operands {889                    match *op {890                        mir::InlineAsmOperand::SymFn { ref value } => {891                            let fn_ty = value.const_.ty();892                            // *Before* monomorphizing, record that we already handled this mention.893                            self.used_mentioned_items.insert(MentionedItem::Fn(fn_ty));894                            let fn_ty = self.monomorphize(fn_ty);895                            visit_fn_use(self.tcx, fn_ty, false, source, self.used_items);896                        }897                        mir::InlineAsmOperand::SymStatic { def_id } => {898                            let instance = Instance::mono(self.tcx, def_id);899                            if self.tcx.should_codegen_locally(instance) {900                                trace!("collecting asm sym static {:?}", def_id);901                                self.used_items.push(respan(source, MonoItem::Static(def_id)));902                            }903                        }904                        _ => {}905                    }906                }907            }908            mir::TerminatorKind::Assert { ref msg, .. } => match &**msg {909                mir::AssertKind::BoundsCheck { .. } => {910                    push_mono_lang_item(self, LangItem::PanicBoundsCheck);911                }912                mir::AssertKind::MisalignedPointerDereference { .. } => {913                    push_mono_lang_item(self, LangItem::PanicMisalignedPointerDereference);914                }915                mir::AssertKind::NullPointerDereference => {916                    push_mono_lang_item(self, LangItem::PanicNullPointerDereference);917                }918                mir::AssertKind::NullReferenceConstructed => {919                    push_mono_lang_item(self, LangItem::PanicNullReferenceConstructed);920                }921                mir::AssertKind::InvalidEnumConstruction(_) => {922                    push_mono_lang_item(self, LangItem::PanicInvalidEnumConstruction);923                }924                _ => {925                    push_mono_lang_item(self, msg.panic_function());926                }927            },928            mir::TerminatorKind::UnwindTerminate(reason) => {929                push_mono_lang_item(self, reason.lang_item());930            }931            mir::TerminatorKind::Goto { .. }932            | mir::TerminatorKind::SwitchInt { .. }933            | mir::TerminatorKind::UnwindResume934            | mir::TerminatorKind::Return935            | mir::TerminatorKind::Unreachable => {}936            mir::TerminatorKind::CoroutineDrop937            | mir::TerminatorKind::Yield { .. }938            | mir::TerminatorKind::FalseEdge { .. }939            | mir::TerminatorKind::FalseUnwind { .. } => bug!(),940        }941942        if let Some(mir::UnwindAction::Terminate(reason)) = terminator.unwind() {943            push_mono_lang_item(self, reason.lang_item());944        }945946        self.super_terminator(terminator, location);947    }948}949950fn visit_drop_use<'tcx>(951    tcx: TyCtxt<'tcx>,952    ty: Ty<'tcx>,953    is_direct_call: bool,954    source: Span,955    output: &mut MonoItems<'tcx>,956) {957    let instance = Instance::resolve_drop_glue(tcx, ty);958    visit_instance_use(tcx, instance, is_direct_call, source, output);959}960961/// For every call of this function in the visitor, make sure there is a matching call in the962/// `mentioned_items` pass!963fn visit_fn_use<'tcx>(964    tcx: TyCtxt<'tcx>,965    ty: Ty<'tcx>,966    is_direct_call: bool,967    source: Span,968    output: &mut MonoItems<'tcx>,969) {970    if let ty::FnDef(def_id, args) = *ty.kind() {971        let args = args.no_bound_vars().unwrap();972        let instance = if is_direct_call {973            ty::Instance::expect_resolve(974                tcx,975                ty::TypingEnv::fully_monomorphized(),976                def_id,977                args,978                source,979            )980        } else {981            match ty::Instance::resolve_for_fn_ptr(982                tcx,983                ty::TypingEnv::fully_monomorphized(),984                def_id,985                args,986            ) {987                Some(instance) => instance,988                _ => bug!("failed to resolve instance for {ty}"),989            }990        };991        visit_instance_use(tcx, instance, is_direct_call, source, output);992    }993}994995fn visit_instance_use<'tcx>(996    tcx: TyCtxt<'tcx>,997    instance: ty::Instance<'tcx>,998    is_direct_call: bool,999    source: Span,1000    output: &mut MonoItems<'tcx>,1001) {1002    debug!("visit_item_use({:?}, is_direct_call={:?})", instance, is_direct_call);1003    if !tcx.should_codegen_locally(instance) {1004        return;1005    }1006    if let Some(intrinsic) = tcx.intrinsic(instance.def_id()) {1007        if let Some(_requirement) = ValidityRequirement::from_intrinsic(intrinsic.name) {1008            // The intrinsics assert_inhabited, assert_zero_valid, and assert_mem_uninitialized_valid will1009            // be lowered in codegen to nothing or a call to panic_nounwind. So if we encounter any1010            // of those intrinsics, we need to include a mono item for panic_nounwind, else we may try to1011            // codegen a call to that function without generating code for the function itself.1012            let def_id = tcx.require_lang_item(LangItem::PanicNounwind, source);1013            let panic_instance = Instance::mono(tcx, def_id);1014            if tcx.should_codegen_locally(panic_instance) {1015                output.push(create_fn_mono_item(tcx, panic_instance, source));1016            }1017        } else if !intrinsic.must_be_overridden1018            && (tcx.sess.opts.unstable_opts.force_intrinsic_fallback1019                || !tcx.sess.replaced_intrinsics.contains(&intrinsic.name))1020        {1021            // Codegen the fallback body of intrinsics with fallback bodies.1022            // We have to skip this otherwise as there's no body to codegen.1023            //1024            // We also skip `replaced_intrinsics` which are always replaced by the backend and hence1025            // monomorphizing the fallback body would be pointless.1026            //1027            // However, when -Zforce-intrinsic-fallback is set (e.g. to test the fallback1028            // implementations) we ignore the optimization hint and do monomorphize1029            // the fallback body.1030            let instance = ty::Instance::new_raw(instance.def_id(), instance.args);1031            if tcx.should_codegen_locally(instance) {1032                output.push(create_fn_mono_item(tcx, instance, source));1033            }1034        }1035    }10361037    match instance.def {1038        ty::InstanceKind::Virtual(..)1039        | ty::InstanceKind::Intrinsic(_)1040        | ty::InstanceKind::LlvmIntrinsic(_) => {1041            if !is_direct_call {1042                bug!("{:?} being reified", instance);1043            }1044        }1045        ty::InstanceKind::Shim(ty::ShimKind::ThreadLocal(..)) => {1046            bug!("{:?} being reified", instance);1047        }1048        ty::InstanceKind::Shim(ty::ShimKind::DropGlue(_, None)) => {1049            // Don't need to emit noop drop glue if we are calling directly.1050            //1051            // Note that we also optimize away the call to visit_instance_use in vtable construction1052            // (see create_mono_items_for_vtable_methods).1053            if !is_direct_call {1054                output.push(create_fn_mono_item(tcx, instance, source));1055            }1056        }1057        ty::InstanceKind::Item(..)1058        | ty::InstanceKind::Shim(ty::ShimKind::DropGlue(_, Some(_)))1059        | ty::InstanceKind::Shim(ty::ShimKind::FutureDropPoll(..))1060        | ty::InstanceKind::Shim(ty::ShimKind::AsyncDropGlue(_, _))1061        | ty::InstanceKind::Shim(ty::ShimKind::AsyncDropGlueCtor(_, _))1062        | ty::InstanceKind::Shim(ty::ShimKind::VTable(..))1063        | ty::InstanceKind::Shim(ty::ShimKind::Reify(..))1064        | ty::InstanceKind::Shim(ty::ShimKind::ClosureOnce { .. })1065        | ty::InstanceKind::Shim(ty::ShimKind::ConstructCoroutineInClosure { .. })1066        | ty::InstanceKind::Shim(ty::ShimKind::FnPtr(..))1067        | ty::InstanceKind::Shim(ty::ShimKind::Clone(..))1068        | ty::InstanceKind::Shim(ty::ShimKind::FnPtrAsPtr(..))1069        | ty::InstanceKind::Shim(ty::ShimKind::FnPtrFromPtr(..)) => {1070            output.push(create_fn_mono_item(tcx, instance, source));1071        }1072    }1073}10741075/// Returns `true` if we should codegen an instance in the local crate, or returns `false` if we1076/// can just link to the upstream crate and therefore don't need a mono item.1077fn should_codegen_locally<'tcx>(tcx: TyCtxt<'tcx>, instance: Instance<'tcx>) -> bool {1078    let Some(def_id) = instance.def.def_id_if_not_guaranteed_local_codegen() else {1079        return true;1080    };10811082    if tcx.is_foreign_item(def_id) {1083        // Foreign items are always linked against, there's no way of instantiating them.1084        return false;1085    }10861087    if tcx.def_kind(def_id).has_codegen_attrs()1088        && matches!(tcx.codegen_fn_attrs(def_id).inline, InlineAttr::Force { .. })1089    {1090        // `#[rustc_force_inline]` items should never be codegened. This should be caught by1091        // the MIR validator.1092        tcx.dcx().delayed_bug("attempt to codegen `#[rustc_force_inline]` item");1093    }10941095    if def_id.is_local() {1096        // Local items cannot be referred to locally without monomorphizing them locally.1097        return true;1098    }10991100    if tcx.is_reachable_non_generic(def_id) || instance.upstream_monomorphization(tcx).is_some() {1101        // We can link to the item in question, no instance needed in this crate.1102        return false;1103    }11041105    if let DefKind::Static { .. } = tcx.def_kind(def_id) {1106        // We cannot monomorphize statics from upstream crates.1107        return false;1108    }11091110    // See comment in should_encode_mir in rustc_metadata for why we don't report1111    // an error for constructors.1112    if !tcx.is_mir_available(def_id) && !matches!(tcx.def_kind(def_id), DefKind::Ctor(..)) {1113        tcx.dcx().emit_fatal(NoOptimizedMir {1114            span: tcx.def_span(def_id),1115            crate_name: tcx.crate_name(def_id.krate),1116            instance: instance.to_string(),1117        });1118    }11191120    true1121}11221123/// For a given pair of source and target type that occur in an unsizing coercion,1124/// this function finds the pair of types that determines the vtable linking1125/// them.1126///1127/// For example, the source type might be `&SomeStruct` and the target type1128/// might be `&dyn SomeTrait` in a cast like:1129///1130/// ```rust,ignore (not real code)1131/// let src: &SomeStruct = ...;1132/// let target = src as &dyn SomeTrait;1133/// ```1134///1135/// Then the output of this function would be (SomeStruct, SomeTrait) since for1136/// constructing the `target` wide-pointer we need the vtable for that pair.1137///1138/// Things can get more complicated though because there's also the case where1139/// the unsized type occurs as a field:1140///1141/// ```rust1142/// struct ComplexStruct<T: ?Sized> {1143///    a: u32,1144///    b: f64,1145///    c: T1146/// }1147/// ```1148///1149/// In this case, if `T` is sized, `&ComplexStruct<T>` is a thin pointer. If `T`1150/// is unsized, `&SomeStruct` is a wide pointer, and the vtable it points to is1151/// for the pair of `T` (which is a trait) and the concrete type that `T` was1152/// originally coerced from:1153///1154/// ```rust,ignore (not real code)1155/// let src: &ComplexStruct<SomeStruct> = ...;1156/// let target = src as &ComplexStruct<dyn SomeTrait>;1157/// ```1158///1159/// Again, we want this `find_vtable_types_for_unsizing()` to provide the pair1160/// `(SomeStruct, SomeTrait)`.1161///1162/// Finally, there is also the case of custom unsizing coercions, e.g., for1163/// smart pointers such as `Rc` and `Arc`.1164fn find_tails_for_unsizing<'tcx>(1165    tcx: TyCtxtAt<'tcx>,1166    source_ty: Ty<'tcx>,1167    target_ty: Ty<'tcx>,1168) -> (Ty<'tcx>, Ty<'tcx>) {1169    let typing_env = ty::TypingEnv::fully_monomorphized();1170    debug_assert!(!source_ty.has_param(), "{source_ty} should be fully monomorphic");1171    debug_assert!(!target_ty.has_param(), "{target_ty} should be fully monomorphic");11721173    match (source_ty.kind(), target_ty.kind()) {1174        (&ty::Pat(source, _), &ty::Pat(target, _)) => find_tails_for_unsizing(tcx, source, target),1175        (1176            &ty::Ref(_, source_pointee, _),1177            &ty::Ref(_, target_pointee, _) | &ty::RawPtr(target_pointee, _),1178        )1179        | (&ty::RawPtr(source_pointee, _), &ty::RawPtr(target_pointee, _)) => {1180            tcx.struct_lockstep_tails_for_codegen(source_pointee, target_pointee, typing_env)1181        }11821183        // `Box<T>` could go through the ADT code below, b/c it'll unpeel to `Unique<T>`,1184        // and eventually bottom out in a raw ref, but we can micro-optimize it here.1185        (_, _)1186            if let Some(source_boxed) = source_ty.boxed_ty()1187                && let Some(target_boxed) = target_ty.boxed_ty() =>1188        {1189            tcx.struct_lockstep_tails_for_codegen(source_boxed, target_boxed, typing_env)1190        }11911192        (&ty::Adt(source_adt_def, source_args), &ty::Adt(target_adt_def, target_args)) => {1193            assert_eq!(source_adt_def, target_adt_def);1194            let CustomCoerceUnsized::Struct(coerce_index) =1195                match crate::custom_coerce_unsize_info(tcx, source_ty, target_ty) {1196                    Ok(ccu) => ccu,1197                    Err(e) => {1198                        let e = Ty::new_error(tcx.tcx, e);1199                        return (e, e);1200                    }1201                };1202            let coerce_field = &source_adt_def.non_enum_variant().fields[coerce_index];1203            // We're getting a possibly unnormalized type, so normalize it.1204            let source_field =1205                tcx.normalize_erasing_regions(typing_env, coerce_field.ty(*tcx, source_args));1206            let target_field =1207                tcx.normalize_erasing_regions(typing_env, coerce_field.ty(*tcx, target_args));1208            find_tails_for_unsizing(tcx, source_field, target_field)1209        }12101211        _ => bug!(1212            "find_vtable_types_for_unsizing: invalid coercion {:?} -> {:?}",1213            source_ty,1214            target_ty1215        ),1216    }1217}12181219#[instrument(skip(tcx), level = "debug", ret)]1220fn create_fn_mono_item<'tcx>(1221    tcx: TyCtxt<'tcx>,1222    instance: Instance<'tcx>,1223    source: Span,1224) -> Spanned<MonoItem<'tcx>> {1225    let def_id = instance.def_id();1226    if tcx.sess.opts.unstable_opts.profile_closures1227        && def_id.is_local()1228        && tcx.is_closure_like(def_id)1229    {1230        crate::util::dump_closure_profile(tcx, instance);1231    }12321233    respan(source, MonoItem::Fn(instance))1234}12351236/// Creates a `MonoItem` for each method that is referenced by the vtable for1237/// the given trait/impl pair.1238fn create_mono_items_for_vtable_methods<'tcx>(1239    tcx: TyCtxt<'tcx>,1240    trait_ty: Ty<'tcx>,1241    impl_ty: Ty<'tcx>,1242    source: Span,1243    output: &mut MonoItems<'tcx>,1244) {1245    assert!(!trait_ty.has_escaping_bound_vars() && !impl_ty.has_escaping_bound_vars());12461247    let ty::Dynamic(trait_ty, ..) = trait_ty.kind() else {1248        bug!("create_mono_items_for_vtable_methods: {trait_ty:?} not a trait type");1249    };1250    if let Some(principal) = trait_ty.principal() {1251        let trait_ref =1252            tcx.instantiate_bound_regions_with_erased(principal.with_self_ty(tcx, impl_ty));1253        assert!(!trait_ref.has_escaping_bound_vars());12541255        // Walk all methods of the trait, including those of its supertraits1256        let entries = tcx.vtable_entries(trait_ref);1257        debug!(?entries);1258        let methods = entries1259            .iter()1260            .filter_map(|entry| match entry {1261                VtblEntry::MetadataDropInPlace1262                | VtblEntry::MetadataSize1263                | VtblEntry::MetadataAlign1264                | VtblEntry::Vacant => None,1265                VtblEntry::TraitVPtr(_) => {1266                    // all super trait items already covered, so skip them.1267                    None1268                }1269                VtblEntry::Method(instance) => {1270                    Some(*instance).filter(|instance| tcx.should_codegen_locally(*instance))1271                }1272            })1273            .map(|item| create_fn_mono_item(tcx, item, source));1274        output.extend(methods);1275    }12761277    // Also add the destructor, if it's necessary.1278    //1279    // This matches the check in vtable_allocation_provider in middle/ty/vtable.rs,1280    // if we don't need drop we're not adding an actual pointer to the vtable.1281    if impl_ty.needs_drop(tcx, ty::TypingEnv::fully_monomorphized()) {1282        visit_drop_use(tcx, impl_ty, false, source, output);1283    }1284}12851286/// Scans the CTFE alloc in order to find function pointers and statics that must be monomorphized.1287fn collect_alloc<'tcx>(tcx: TyCtxt<'tcx>, alloc_id: AllocId, output: &mut MonoItems<'tcx>) {1288    match tcx.global_alloc(alloc_id) {1289        GlobalAlloc::Static(def_id) => {1290            assert!(!tcx.is_thread_local_static(def_id));1291            let instance = Instance::mono(tcx, def_id);1292            if tcx.should_codegen_locally(instance) {1293                trace!("collecting static {:?}", def_id);1294                output.push(dummy_spanned(MonoItem::Static(def_id)));1295            }1296        }1297        GlobalAlloc::Memory(alloc) => {1298            trace!("collecting {:?} with {:#?}", alloc_id, alloc);1299            let ptrs = alloc.inner().provenance().ptrs();1300            if !ptrs.is_empty() {1301                for &prov in ptrs.values() {1302                    collect_alloc(tcx, prov.alloc_id(), output);1303                }1304            }1305        }1306        GlobalAlloc::Function { instance, .. } => {1307            if tcx.should_codegen_locally(instance) {1308                trace!("collecting {:?} with {:#?}", alloc_id, instance);1309                output.push(create_fn_mono_item(tcx, instance, DUMMY_SP));1310            }1311        }1312        GlobalAlloc::VTable(ty, dyn_ty) => {1313            let alloc_id = tcx.vtable_allocation((1314                ty,1315                dyn_ty1316                    .principal()1317                    .map(|principal| tcx.instantiate_bound_regions_with_erased(principal)),1318            ));1319            collect_alloc(tcx, alloc_id, output)1320        }1321        GlobalAlloc::TypeId { .. } => {}1322    }1323}13241325/// Scans the MIR in order to find function calls, closures, and drop-glue.1326///1327/// Anything that's found is added to `output`. Furthermore the "mentioned items" of the MIR are returned.1328#[instrument(skip(tcx), level = "debug")]1329fn collect_items_of_instance<'tcx>(1330    tcx: TyCtxt<'tcx>,1331    instance: Instance<'tcx>,1332    mode: CollectionMode,1333) -> Result<(MonoItems<'tcx>, MonoItems<'tcx>), NormalizationErrorInMono> {1334    // This item is getting monomorphized, do mono-time checks.1335    let body = tcx.instance_mir(instance.def);1336    // Plenty of code paths later assume that everything can be normalized. So we have to check1337    // normalization first.1338    // We choose to emit the error outside to provide helpful diagnostics.1339    check_normalization_error(tcx, instance, body)?;1340    tcx.ensure_ok().check_mono_item(instance);13411342    // Naively, in "used" collection mode, all functions get added to *both* `used_items` and1343    // `mentioned_items`. Mentioned items processing will then notice that they have already been1344    // visited, but at that point each mentioned item has been monomorphized, added to the1345    // `mentioned_items` worklist, and checked in the global set of visited items. To remove that1346    // overhead, we have a special optimization that avoids adding items to `mentioned_items` when1347    // they are already added in `used_items`. We could just scan `used_items`, but that's a linear1348    // scan and not very efficient. Furthermore we can only do that *after* monomorphizing the1349    // mentioned item. So instead we collect all pre-monomorphized `MentionedItem` that were already1350    // added to `used_items` in a hash set, which can efficiently query in the1351    // `body.mentioned_items` loop below without even having to monomorphize the item.1352    let mut used_items = MonoItems::new();1353    let mut mentioned_items = MonoItems::new();1354    let mut used_mentioned_items = Default::default();1355    let mut collector = MirUsedCollector {1356        tcx,1357        body,1358        used_items: &mut used_items,1359        used_mentioned_items: &mut used_mentioned_items,1360        instance,1361    };13621363    if mode == CollectionMode::UsedItems {1364        if tcx.sess.opts.debuginfo == DebugInfo::Full {1365            for var_debug_info in &body.var_debug_info {1366                collector.visit_var_debug_info(var_debug_info);1367            }1368        }1369        for (bb, data) in traversal::mono_reachable(body, tcx, instance) {1370            collector.visit_basic_block_data(bb, data)1371        }1372    }13731374    // Always visit all `required_consts`, so that we evaluate them and abort compilation if any of1375    // them errors.1376    for const_op in body.required_consts() {1377        if let Some(val) = collector.eval_constant(const_op) {1378            collect_const_value(tcx, val, &mut mentioned_items);1379        }1380    }13811382    // Always gather mentioned items. We try to avoid processing items that we have already added to1383    // `used_items` above.1384    for item in body.mentioned_items() {1385        if !collector.used_mentioned_items.contains(&item.node) {1386            let item_mono = collector.monomorphize(item.node);1387            visit_mentioned_item(tcx, &item_mono, item.span, &mut mentioned_items);1388        }1389    }13901391    Ok((used_items, mentioned_items))1392}13931394fn items_of_instance<'tcx>(1395    tcx: TyCtxt<'tcx>,1396    (instance, mode): (Instance<'tcx>, CollectionMode),1397) -> Result<1398    (&'tcx [Spanned<MonoItem<'tcx>>], &'tcx [Spanned<MonoItem<'tcx>>]),1399    NormalizationErrorInMono,1400> {1401    let (used_items, mentioned_items) = collect_items_of_instance(tcx, instance, mode)?;14021403    let used_items = tcx.arena.alloc_from_iter(used_items);1404    let mentioned_items = tcx.arena.alloc_from_iter(mentioned_items);14051406    Ok((used_items, mentioned_items))1407}14081409/// `item` must be already monomorphized.1410#[instrument(skip(tcx, span, output), level = "debug")]1411fn visit_mentioned_item<'tcx>(1412    tcx: TyCtxt<'tcx>,1413    item: &MentionedItem<'tcx>,1414    span: Span,1415    output: &mut MonoItems<'tcx>,1416) {1417    match *item {1418        MentionedItem::Fn(ty) => {1419            if let ty::FnDef(def_id, args) = *ty.kind() {1420                let args = args.no_bound_vars().unwrap();1421                let instance = Instance::expect_resolve(1422                    tcx,1423                    ty::TypingEnv::fully_monomorphized(),1424                    def_id,1425                    args,1426                    span,1427                );1428                // `visit_instance_use` was written for "used" item collection but works just as well1429                // for "mentioned" item collection.1430                // We can set `is_direct_call`; that just means we'll skip a bunch of shims that anyway1431                // can't have their own failing constants.1432                visit_instance_use(tcx, instance, /*is_direct_call*/ true, span, output);1433            }1434        }1435        MentionedItem::Drop(ty) => {1436            visit_drop_use(tcx, ty, /*is_direct_call*/ true, span, output);1437        }1438        MentionedItem::UnsizeCast { source_ty, target_ty } => {1439            let (source_ty, target_ty) =1440                find_tails_for_unsizing(tcx.at(span), source_ty, target_ty);1441            // This could also be a different Unsize instruction, like1442            // from a fixed sized array to a slice. But we are only1443            // interested in things that produce a vtable.1444            if target_ty.is_trait() && !source_ty.is_trait() {1445                create_mono_items_for_vtable_methods(tcx, target_ty, source_ty, span, output);1446            }1447        }1448        MentionedItem::Closure(source_ty) => {1449            if let ty::Closure(def_id, args) = *source_ty.kind() {1450                let instance =1451                    Instance::resolve_closure(tcx, def_id, args, ty::ClosureKind::FnOnce);1452                if tcx.should_codegen_locally(instance) {1453                    output.push(create_fn_mono_item(tcx, instance, span));1454                }1455            } else {1456                bug!()1457            }1458        }1459    }1460}14611462#[instrument(skip(tcx, output), level = "debug")]1463fn collect_const_value<'tcx>(1464    tcx: TyCtxt<'tcx>,1465    value: mir::ConstValue,1466    output: &mut MonoItems<'tcx>,1467) {1468    match value {1469        mir::ConstValue::Scalar(Scalar::Ptr(ptr, _size)) => {1470            collect_alloc(tcx, ptr.provenance.alloc_id(), output)1471        }1472        mir::ConstValue::Indirect { alloc_id, .. }1473        | mir::ConstValue::Slice { alloc_id, meta: _ } => collect_alloc(tcx, alloc_id, output),1474        _ => {}1475    }1476}14771478//=-----------------------------------------------------------------------------1479// Root Collection1480//=-----------------------------------------------------------------------------14811482// Find all non-generic items by walking the HIR. These items serve as roots to1483// start monomorphizing from.1484#[instrument(skip(tcx, mode), level = "debug")]1485fn collect_roots(tcx: TyCtxt<'_>, mode: MonoItemCollectionStrategy) -> Vec<MonoItem<'_>> {1486    debug!("collecting roots");1487    let mut roots = MonoItems::new();14881489    // Read the manifest and add the recorded kernel instantiations as roots so they are codegened.1490    if let Some(manifest_path) = tcx.sess.opts.unstable_opts.offload.iter().find_map(|o| {1491        if let rustc_session::config::Offload::Device(p) = o1492            && !p.is_empty()1493        {1494            Some(p)1495        } else {1496            None1497        }1498    }) {1499        tcx.sess.file_depinfo.borrow_mut().insert(Symbol::intern(manifest_path));1500        match crate::offload::manifest::read_manifest(std::path::Path::new(manifest_path), tcx) {1501            Ok(instances) => {1502                for instance in instances {1503                    if instance.def_id().is_local() {1504                        roots.push(dummy_spanned(MonoItem::Fn(instance)));1505                    }1506                }1507            }1508            Err(e) => {1509                tcx.dcx().emit_err(crate::diagnostics::OffloadManifestReadError {1510                    path: manifest_path.clone(),1511                    err: e.to_string(),1512                });1513            }1514        }1515    }15161517    {1518        let entry_fn = tcx.entry_fn(());15191520        debug!("collect_roots: entry_fn = {:?}", entry_fn);15211522        let mut collector = RootCollector { tcx, strategy: mode, entry_fn, output: &mut roots };15231524        let crate_items = tcx.hir_crate_items(());15251526        for id in crate_items.free_items() {1527            collector.process_item(id);1528        }15291530        for id in crate_items.impl_items() {1531            collector.process_impl_item(id);1532        }15331534        for id in crate_items.nested_bodies() {1535            collector.process_nested_body(id);1536        }15371538        collector.push_extra_entry_roots();1539    }15401541    let is_host_metadata = tcx1542        .sess1543        .opts1544        .unstable_opts1545        .offload1546        .iter()1547        .any(|o| matches!(o, rustc_session::config::Offload::HostMetadata(_)));1548    if is_host_metadata {1549        let crate_items = tcx.hir_crate_items(());1550        for id in crate_items.free_items() {1551            if !matches!(tcx.def_kind(id.owner_id), DefKind::Fn | DefKind::AssocFn) {1552                continue;1553            }1554            let def_id = id.owner_id.to_def_id();1555            if !tcx.generics_of(def_id).requires_monomorphization(tcx)1556                && tcx.codegen_fn_attrs(def_id).flags.intersects(CodegenFnAttrFlags::OFFLOAD_KERNEL)1557            {1558                roots.push(dummy_spanned(MonoItem::Fn(Instance::mono(tcx, def_id))));1559            }1560        }1561        for id in crate_items.impl_items() {1562            if !matches!(tcx.def_kind(id.owner_id), DefKind::Fn | DefKind::AssocFn) {1563                continue;1564            }1565            let def_id = id.owner_id.to_def_id();1566            if !tcx.generics_of(def_id).requires_monomorphization(tcx)1567                && tcx.codegen_fn_attrs(def_id).flags.intersects(CodegenFnAttrFlags::OFFLOAD_KERNEL)1568            {1569                roots.push(dummy_spanned(MonoItem::Fn(Instance::mono(tcx, def_id))));1570            }1571        }1572        for id in crate_items.trait_items() {1573            if !matches!(tcx.def_kind(id.owner_id), DefKind::Fn | DefKind::AssocFn) {1574                continue;1575            }1576            let def_id = id.owner_id.to_def_id();1577            if !tcx.generics_of(def_id).requires_monomorphization(tcx)1578                && tcx.codegen_fn_attrs(def_id).flags.intersects(CodegenFnAttrFlags::OFFLOAD_KERNEL)1579            {1580                roots.push(dummy_spanned(MonoItem::Fn(Instance::mono(tcx, def_id))));1581            }1582        }1583    }15841585    // We can only codegen items that are instantiable - items all of1586    // whose predicates hold. Luckily, items that aren't instantiable1587    // can't actually be used, so we can just skip codegenning them.1588    roots1589        .into_iter()1590        .filter_map(|Spanned { node: mono_item, .. }| {1591            mono_item.is_instantiable(tcx).then_some(mono_item)1592        })1593        .collect()1594}15951596struct RootCollector<'a, 'tcx> {1597    tcx: TyCtxt<'tcx>,1598    strategy: MonoItemCollectionStrategy,1599    output: &'a mut MonoItems<'tcx>,1600    entry_fn: Option<(DefId, EntryFnType)>,1601}16021603impl<'v> RootCollector<'_, 'v> {1604    fn process_item(&mut self, id: hir::ItemId) {1605        match self.tcx.def_kind(id.owner_id) {1606            DefKind::Enum | DefKind::Struct | DefKind::Union => {1607                if self.strategy == MonoItemCollectionStrategy::Eager1608                    && !self.tcx.generics_of(id.owner_id).requires_monomorphization(self.tcx)1609                {1610                    debug!("RootCollector: ADT drop-glue for `{id:?}`",);1611                    let id_args =1612                        ty::GenericArgs::for_item(self.tcx, id.owner_id.to_def_id(), |param, _| {1613                            match param.kind {1614                                GenericParamDefKind::Lifetime => {1615                                    self.tcx.lifetimes.re_erased.into()1616                                }1617                                GenericParamDefKind::Type { .. }1618                                | GenericParamDefKind::Const { .. } => {1619                                    unreachable!(1620                                        "`own_requires_monomorphization` check means that \1621                                we should have no type/const params"1622                                    )1623                                }1624                            }1625                        });16261627                    // This type is impossible to instantiate, so we should not try to1628                    // generate a `drop_glue` instance for it.1629                    if self.tcx.instantiate_and_check_impossible_clauses((1630                        id.owner_id.to_def_id(),1631                        id_args,1632                    )) {1633                        return;1634                    }16351636                    let ty = self1637                        .tcx1638                        .type_of(id.owner_id.to_def_id())1639                        .instantiate(self.tcx, id_args)1640                        .skip_norm_wip();1641                    assert!(!ty.has_non_region_param());1642                    visit_drop_use(self.tcx, ty, true, DUMMY_SP, self.output);1643                }1644            }1645            DefKind::GlobalAsm => {1646                debug!(1647                    "RootCollector: ItemKind::GlobalAsm({})",1648                    self.tcx.def_path_str(id.owner_id)1649                );1650                self.output.push(dummy_spanned(MonoItem::GlobalAsm(id)));1651            }1652            DefKind::Static { .. } => {1653                let def_id = id.owner_id.to_def_id();1654                debug!("RootCollector: ItemKind::Static({})", self.tcx.def_path_str(def_id));1655                self.output.push(dummy_spanned(MonoItem::Static(def_id)));1656            }1657            DefKind::Const => {1658                // Const items only generate mono items if they are actually used somewhere.1659                // Just declaring them is insufficient.16601661                // If we're collecting items eagerly, then recurse into all constants.1662                // Otherwise the value is only collected when explicitly mentioned in other items.1663                if self.strategy == MonoItemCollectionStrategy::Eager {1664                    let def_id = id.owner_id.to_def_id();1665                    // Type Consts don't have bodies to evaluate1666                    // nor do they make sense as a static.1667                    if self.tcx.const_of_item(def_id).is_some() {1668                        // FIXME(mgca): Is this actually what we want? We may want to1669                        // normalize to a ValTree then convert to a const allocation and1670                        // collect that?1671                        return;1672                    }1673                    if self.tcx.generics_of(id.owner_id).own_requires_monomorphization() {1674                        return;1675                    }1676                    let Ok(val) = self.tcx.const_eval_poly(def_id) else {1677                        return;1678                    };1679                    collect_const_value(self.tcx, val, self.output);1680                }1681            }1682            DefKind::Impl { of_trait: true } => {1683                if self.strategy == MonoItemCollectionStrategy::Eager {1684                    create_mono_items_for_default_impls(self.tcx, id, self.output);1685                }1686            }1687            DefKind::Fn => {1688                self.push_if_root(id.owner_id.def_id);1689            }1690            _ => {}1691        }1692    }16931694    fn process_impl_item(&mut self, id: hir::ImplItemId) {1695        if self.tcx.def_kind(id.owner_id) == DefKind::AssocFn {1696            self.push_if_root(id.owner_id.def_id);1697        }1698    }16991700    fn process_nested_body(&mut self, def_id: LocalDefId) {1701        match self.tcx.def_kind(def_id) {1702            DefKind::Closure => {1703                // for 'pub async fn foo(..)' also trying to monomorphize foo::{closure}1704                let is_pub_fn_coroutine =1705                    match *self.tcx.type_of(def_id).instantiate_identity().skip_norm_wip().kind() {1706                        ty::Coroutine(cor_id, _args) => {1707                            let tcx = self.tcx;1708                            let parent_id = tcx.parent(cor_id);1709                            tcx.def_kind(parent_id) == DefKind::Fn1710                                && tcx.asyncness(parent_id).is_async()1711                                && tcx.visibility(parent_id).is_public()1712                        }1713                        ty::Closure(..) | ty::CoroutineClosure(..) => false,1714                        _ => unreachable!(),1715                    };1716                if (self.strategy == MonoItemCollectionStrategy::Eager || is_pub_fn_coroutine)1717                    && !self1718                        .tcx1719                        .generics_of(self.tcx.typeck_root_def_id_local(def_id))1720                        .requires_monomorphization(self.tcx)1721                {1722                    let instance = match *self1723                        .tcx1724                        .type_of(def_id)1725                        .instantiate_identity()1726                        .skip_norm_wip()1727                        .kind()1728                    {1729                        ty::Closure(def_id, args)1730                        | ty::Coroutine(def_id, args)1731                        | ty::CoroutineClosure(def_id, args) => {1732                            Instance::new_raw(def_id, self.tcx.erase_and_anonymize_regions(args))1733                        }1734                        _ => unreachable!(),1735                    };1736                    let Ok(instance) = self.tcx.try_normalize_erasing_regions(1737                        ty::TypingEnv::fully_monomorphized(),1738                        Unnormalized::new_wip(instance),1739                    ) else {1740                        // Don't ICE on an impossible-to-normalize closure.1741                        return;1742                    };1743                    let mono_item = create_fn_mono_item(self.tcx, instance, DUMMY_SP);1744                    if mono_item.node.is_instantiable(self.tcx) {1745                        self.output.push(mono_item);1746                    }1747                }1748            }1749            _ => {}1750        }1751    }17521753    fn is_root(&self, def_id: LocalDefId) -> bool {1754        !self.tcx.generics_of(def_id).requires_monomorphization(self.tcx)1755            && match self.strategy {1756                MonoItemCollectionStrategy::Eager => {1757                    !matches!(self.tcx.codegen_fn_attrs(def_id).inline, InlineAttr::Force { .. })1758                    // comptime fns can't be codegenned, so we need to prevent collecting them even1759                    // with link-dead-code. Lazy mode prevents them by them not showing up in1760                    // `is_reachable_non_generic` (and `entry_fn` can't be comptime).1761                    && match self.tcx.def_kind(def_id) {1762                        DefKind::Fn | DefKind::AssocFn => {1763                            self.tcx.constness(def_id) != hir::Constness::Const { always: true }1764                        }1765                        _ => true,1766                    }1767                }1768                MonoItemCollectionStrategy::Lazy => {1769                    self.entry_fn.and_then(|(id, _)| id.as_local()) == Some(def_id)1770                        || self.tcx.is_reachable_non_generic(def_id)1771                        || {1772                            let flags = self.tcx.codegen_fn_attrs(def_id).flags;1773                            flags.intersects(1774                                CodegenFnAttrFlags::RUSTC_STD_INTERNAL_SYMBOL1775                                    | CodegenFnAttrFlags::EXTERNALLY_IMPLEMENTABLE_ITEM,1776                            )1777                        }1778                }1779            }1780    }17811782    /// If `def_id` represents a root, pushes it onto the list of1783    /// outputs. (Note that all roots must be monomorphic.)1784    #[instrument(skip(self), level = "debug")]1785    fn push_if_root(&mut self, def_id: LocalDefId) {1786        if self.is_root(def_id) {1787            debug!("found root");17881789            let instance = Instance::mono(self.tcx, def_id.to_def_id());1790            self.output.push(create_fn_mono_item(self.tcx, instance, DUMMY_SP));1791        }1792    }17931794    /// As a special case, when/if we encounter the1795    /// `main()` function, we also have to generate a1796    /// monomorphized copy of the start lang item based on1797    /// the return type of `main`. This is not needed when1798    /// the user writes their own `start` manually.1799    fn push_extra_entry_roots(&mut self) {1800        let Some((main_def_id, EntryFnType::Main { .. })) = self.entry_fn else {1801            return;1802        };18031804        let main_instance = Instance::mono(self.tcx, main_def_id);1805        if self.tcx.should_codegen_locally(main_instance) {1806            self.output.push(create_fn_mono_item(1807                self.tcx,1808                main_instance,1809                self.tcx.def_span(main_def_id),1810            ));1811        }18121813        let Some(start_def_id) = self.tcx.lang_items().start_fn() else {1814            self.tcx.dcx().emit_fatal(diagnostics::StartNotFound);1815        };1816        let main_ret_ty = self.tcx.fn_sig(main_def_id).no_bound_vars().unwrap().output();18171818        // Given that `main()` has no arguments,1819        // then its return type cannot have1820        // late-bound regions, since late-bound1821        // regions must appear in the argument1822        // listing.1823        let main_ret_ty = self.tcx.normalize_erasing_regions(1824            ty::TypingEnv::fully_monomorphized(),1825            Unnormalized::new_wip(main_ret_ty.no_bound_vars().unwrap()),1826        );18271828        let start_instance = Instance::expect_resolve(1829            self.tcx,1830            ty::TypingEnv::fully_monomorphized(),1831            start_def_id,1832            self.tcx.mk_args(&[main_ret_ty.into()]),1833            DUMMY_SP,1834        );18351836        self.output.push(create_fn_mono_item(self.tcx, start_instance, DUMMY_SP));1837    }1838}18391840#[instrument(level = "debug", skip(tcx, output))]1841fn create_mono_items_for_default_impls<'tcx>(1842    tcx: TyCtxt<'tcx>,1843    item: hir::ItemId,1844    output: &mut MonoItems<'tcx>,1845) {1846    let impl_ = tcx.impl_trait_header(item.owner_id);18471848    if impl_.polarity == ty::ImplPolarity::Negative {1849        return;1850    }18511852    if tcx.generics_of(item.owner_id).own_requires_monomorphization() {1853        return;1854    }18551856    // Lifetimes never affect trait selection, so we are allowed to eagerly1857    // instantiate an instance of an impl method if the impl (and method,1858    // which we check below) is only parameterized over lifetime. In that case,1859    // we use the ReErased, which has no lifetime information associated with1860    // it, to validate whether or not the impl is legal to instantiate at all.1861    let only_region_params = |param: &ty::GenericParamDef, _: &_| match param.kind {1862        GenericParamDefKind::Lifetime => tcx.lifetimes.re_erased.into(),1863        GenericParamDefKind::Type { .. } | GenericParamDefKind::Const { .. } => {1864            unreachable!(1865                "`own_requires_monomorphization` check means that \1866                we should have no type/const params"1867            )1868        }1869    };1870    let impl_args = GenericArgs::for_item(tcx, item.owner_id.to_def_id(), only_region_params);1871    let trait_ref = impl_.trait_ref.instantiate(tcx, impl_args).skip_norm_wip();18721873    // Unlike 'lazy' monomorphization that begins by collecting items transitively1874    // called by `main` or other global items, when eagerly monomorphizing impl1875    // items, we never actually check that the predicates of this impl are satisfied1876    // in a empty param env (i.e. with no assumptions).1877    //1878    // Even though this impl has no type or const generic parameters, because we don't1879    // consider higher-ranked predicates such as `for<'a> &'a mut [u8]: Copy` to1880    // be trivially false. We must now check that the impl has no impossible-to-satisfy1881    // clauses.1882    if tcx.instantiate_and_check_impossible_clauses((item.owner_id.to_def_id(), impl_args)) {1883        return;1884    }18851886    let typing_env = ty::TypingEnv::fully_monomorphized();1887    let trait_ref = tcx.normalize_erasing_regions(typing_env, Unnormalized::new_wip(trait_ref));1888    let overridden_methods = tcx.impl_item_implementor_ids(item.owner_id);1889    for method in tcx.provided_trait_methods(trait_ref.def_id) {1890        if overridden_methods.contains_key(&method.def_id) {1891            continue;1892        }18931894        if tcx.generics_of(method.def_id).own_requires_monomorphization() {1895            continue;1896        }18971898        // As mentioned above, the method is legal to eagerly instantiate if it1899        // only has lifetime generic parameters. This is validated by calling1900        // `own_requires_monomorphization` on both the impl and method.1901        let args = trait_ref.args.extend_to(tcx, method.def_id, only_region_params);1902        let instance = ty::Instance::expect_resolve(tcx, typing_env, method.def_id, args, DUMMY_SP);19031904        let mono_item = create_fn_mono_item(tcx, instance, DUMMY_SP);1905        if mono_item.node.is_instantiable(tcx) && tcx.should_codegen_locally(instance) {1906            output.push(mono_item);1907        }1908    }1909}19101911//=-----------------------------------------------------------------------------1912// Top-level entry point, tying it all together1913//=-----------------------------------------------------------------------------19141915#[instrument(skip(tcx, strategy), level = "debug")]1916pub(crate) fn collect_crate_mono_items<'tcx>(1917    tcx: TyCtxt<'tcx>,1918    strategy: MonoItemCollectionStrategy,1919) -> (Vec<MonoItem<'tcx>>, UsageMap<'tcx>) {1920    let _prof_timer = tcx.prof.generic_activity("monomorphization_collector");19211922    let roots = tcx1923        .sess1924        .time("monomorphization_collector_root_collections", || collect_roots(tcx, strategy));19251926    debug!("building mono item graph, beginning at roots");19271928    let state = SharedState {1929        visited: Lock::new(UnordSet::default()),1930        mentioned: Lock::new(UnordSet::default()),1931        usage_map: Lock::new(UsageMap::new()),1932    };1933    let recursion_limit = tcx.recursion_limit();19341935    tcx.sess.time("monomorphization_collector_graph_walk", || {1936        par_for_each_in(roots, |root| {1937            collect_items_root(tcx, dummy_spanned(*root), &state, recursion_limit);1938        });1939    });19401941    // The set of MonoItems was created in an inherently indeterministic order because1942    // of parallelism. We sort it here to ensure that the output is deterministic.1943    let mono_items = tcx.with_stable_hashing_context(move |mut hcx| {1944        state.visited.into_inner().into_sorted(&mut hcx, true)1945    });19461947    if tcx1948        .sess1949        .opts1950        .unstable_opts1951        .offload1952        .iter()1953        .any(|o| matches!(o, Offload::Device(p) if p.is_empty()))1954    {1955        crate::offload::check_offload_kernels_instantiated(tcx, &mono_items);1956    }19571958    (mono_items, state.usage_map.into_inner())1959}19601961pub(crate) fn provide(providers: &mut Providers) {1962    providers.hooks.should_codegen_locally = should_codegen_locally;1963    providers.queries.items_of_instance = items_of_instance;1964}