1# Rust Port Step 1: Babel AST Crate23## Goal45Create a Rust crate (`compiler/crates/react_compiler_ast`) that precisely models the Babel AST structure, enabling JSON round-tripping: parse JS with Babel in Node.js, serialize to JSON, deserialize into Rust, re-serialize back to JSON, and get an identical result.67This crate is the serialization boundary between the JS toolchain (Babel parser) and the Rust compiler. It must be a faithful 1:1 representation of Babel's AST output — not a simplified or custom IR.89**Current status**: Complete (human reviewed). All 1714 compiler test fixtures round-trip successfully (0 failures). No `Unknown` catch-all variants remain. Scope types are defined separately in [rust-port-0002-scope-types.md](rust-port-0002-scope-types.md).1011---1213## Crate Structure1415```16compiler/crates/17  react_compiler_ast/18    Cargo.toml19    src/20      lib.rs              # Re-exports, top-level File/Program types21      statements.rs       # Statement enum and statement node structs22      expressions.rs      # Expression enum and expression node structs23      literals.rs         # Literal node structs (StringLiteral, NumericLiteral, etc.)24      patterns.rs         # PatternLike enum and pattern node structs25      jsx.rs              # JSX node structs and enums26      declarations.rs     # Import/export, TS declaration, and Flow declaration structs27      common.rs           # SourceLocation, Position, Comment, BaseNode, helpers28      operators.rs        # Operator enums (BinaryOperator, UnaryOperator, etc.)29    tests/30      round_trip.rs       # Round-trip test harness31```3233TypeScript and Flow annotation types are co-located with the module that uses them — TS/Flow expressions live in `expressions.rs`, TS/Flow declarations live in `declarations.rs`. Class-related types are split between `expressions.rs` (ClassExpression, ClassBody) and `statements.rs` (ClassDeclaration). There is no single `Node` enum; the union types (`Statement`, `Expression`, `PatternLike`) serve as the dispatch enums directly.3435### Cargo.toml3637```toml38[package]39name = "react_compiler_ast"40version = "0.1.0"41edition = "2024"4243[dependencies]44serde = { version = "1", features = ["derive"] }45serde_json = "1"4647[dev-dependencies]48walkdir = "2"49similar = "2"           # for readable diffs in round-trip test50```5152No other dependencies. The crate is pure data types + serde.5354---5556## Core Design Decisions5758### 1. Internally tagged via `"type"` field5960Babel AST nodes use a `"type"` field as the discriminant (e.g., `"type": "FunctionDeclaration"`). Serde's default externally-tagged enum format doesn't match this. Use **internally tagged** enums with `#[serde(tag = "type")]`:6162```rust63#[derive(Debug, Clone, Serialize, Deserialize)]64#[serde(tag = "type")]65pub enum Statement {66    BlockStatement(BlockStatement),67    ReturnStatement(ReturnStatement),68    IfStatement(IfStatement),69    // ...70}71```7273Each variant's struct contains the node-specific fields. The `"type"` field is handled by serde's internal tagging.7475### 2. BaseNode fields via flattening7677Every Babel node shares common fields (`start`, `end`, `loc`, `leadingComments`, etc.). A `BaseNode` struct is flattened into each node struct:7879```rust80#[derive(Debug, Clone, Default, Serialize, Deserialize)]81pub struct BaseNode {82    #[serde(rename = "type", default, skip_serializing_if = "Option::is_none")]83    pub node_type: Option<String>,84    #[serde(default, skip_serializing_if = "Option::is_none")]85    pub start: Option<u32>,86    #[serde(default, skip_serializing_if = "Option::is_none")]87    pub end: Option<u32>,88    #[serde(default, skip_serializing_if = "Option::is_none")]89    pub loc: Option<SourceLocation>,90    #[serde(default, skip_serializing_if = "Option::is_none")]91    pub range: Option<(u32, u32)>,92    #[serde(default, skip_serializing_if = "Option::is_none")]93    pub extra: Option<serde_json::Value>,94    #[serde(default, skip_serializing_if = "Option::is_none", rename = "leadingComments")]95    pub leading_comments: Option<Vec<Comment>>,96    #[serde(default, skip_serializing_if = "Option::is_none", rename = "innerComments")]97    pub inner_comments: Option<Vec<Comment>>,98    #[serde(default, skip_serializing_if = "Option::is_none", rename = "trailingComments")]99    pub trailing_comments: Option<Vec<Comment>>,100}101```102103The `node_type` field captures the `"type"` string when `BaseNode` is deserialized directly (not through a `#[serde(tag = "type")]` enum, which consumes the field). It defaults to `None` and is skipped when absent, so it doesn't interfere with round-tripping in either context.104105Each node struct flattens this:106107```rust108#[derive(Debug, Clone, Serialize, Deserialize)]109pub struct FunctionDeclaration {110    #[serde(flatten)]111    pub base: BaseNode,112    pub id: Option<Identifier>,113    pub params: Vec<PatternLike>,114    pub body: BlockStatement,115    #[serde(default)]116    pub generator: bool,117    #[serde(default, rename = "async")]118    pub is_async: bool,119    // ...120}121```122123The `#[serde(flatten)]` + `#[serde(tag = "type")]` combination works correctly — the macro fallback described in the risk section was not needed.124125### 3. Naming conventions126127- Rust struct/enum names: PascalCase matching the Babel type name exactly (e.g., `FunctionDeclaration`, `JSXElement`)128- Rust field names: snake_case, with `#[serde(rename = "camelCase")]` for JSON mapping129- Reserved words: `#[serde(rename = "async")]` on field `is_async: bool`, `#[serde(rename = "type")]` handled by internal tagging130- Operator strings: mapped via `#[serde(rename = "+")]` etc. on enum variants131132### 4. Optional/nullable field patterns133134Babel's TypeScript definitions use several patterns. Map them consistently:135136| Babel TypeScript | JSON behavior | Rust type |137|---|---|---|138| `field: T` | Always present | `field: T` |139| `field?: T \| null` | Absent or `null` | `#[serde(default, skip_serializing_if = "Option::is_none")] field: Option<T>` |140| `field: Array<T \| null>` | Array with null holes | `field: Vec<Option<T>>` |141| `field: T \| null` (required but nullable) | Present, may be `null` | `field: Option<T>` (no `skip_serializing_if` — always serialize) |142143**Critical subtlety**: Some fields like `FunctionDeclaration.id` are typed `id?: Identifier | null` and appear as `"id": null` in JSON (present but null), not absent. The round-trip test catches any mismatches here. When Babel serializes `null` for a field, we must also serialize `null` — not omit it. The round-trip test is the source of truth for which fields use which pattern.144145A `nullable_value` custom deserializer in `common.rs` handles the case where a field needs to distinguish "absent" from "explicitly null" (deserializing the latter as `Some(Value::Null)`):146147```rust148pub fn nullable_value<'de, D>(149    deserializer: D,150) -> Result<Option<Box<serde_json::Value>>, D::Error>151```152153### 5. The `extra` field154155The `extra` field is an unstructured `Record<string, unknown>` in Babel. Use `serde_json::Value` to round-trip it exactly:156157```rust158#[serde(default, skip_serializing_if = "Option::is_none")]159pub extra: Option<serde_json::Value>,160```161162### 6. `#[serde(deny_unknown_fields)]` — do NOT use163164Babel's AST may include fields we don't model (e.g., from plugins, or parser-specific metadata). To ensure forward compatibility and avoid brittle failures, do **not** use `deny_unknown_fields`. Instead, unknown fields are silently dropped during deserialization. The round-trip test detects any fields we're missing, since they'll be absent in the re-serialized output.165166---167168## Node Type Coverage169170All node types that appear in the compiler's 1714 test fixtures are modeled and round-trip successfully. The types are organized as follows:171172### Statements (`statements.rs`, ~25 types)173174The `Statement` enum is the top-level dispatch for all statement and declaration nodes. It includes direct statement types and also pulls in declaration variants (import/export, TS, Flow) to avoid a separate `StatementOrDeclaration` wrapper.175176**Statement types**: `BlockStatement`, `ReturnStatement`, `IfStatement`, `ForStatement`, `WhileStatement`, `DoWhileStatement`, `ForInStatement`, `ForOfStatement`, `SwitchStatement` (+ `SwitchCase`), `ThrowStatement`, `TryStatement` (+ `CatchClause`), `BreakStatement`, `ContinueStatement`, `LabeledStatement`, `ExpressionStatement`, `EmptyStatement`, `DebuggerStatement`, `WithStatement`, `VariableDeclaration` (+ `VariableDeclarator`), `FunctionDeclaration`, `ClassDeclaration`177178**Helper enums**: `ForInit` (VariableDeclaration | Expression), `ForInOfLeft` (VariableDeclaration | PatternLike), `VariableDeclarationKind`179180### Declarations (`declarations.rs`, ~20 types)181182**Import/export**: `ImportDeclaration`, `ExportNamedDeclaration`, `ExportDefaultDeclaration`, `ExportAllDeclaration`, `ImportSpecifier` enum (ImportSpecifier | ImportDefaultSpecifier | ImportNamespaceSpecifier), `ExportSpecifier` enum (ExportSpecifier | ExportDefaultSpecifier | ExportNamespaceSpecifier), `ImportAttribute`, `ModuleExportName`, `Declaration` enum, `ExportDefaultDecl` enum183184**TypeScript declarations (pass-through)**: `TSTypeAliasDeclaration`, `TSInterfaceDeclaration`, `TSEnumDeclaration`, `TSModuleDeclaration`, `TSDeclareFunction`185186**Flow declarations (pass-through)**: `TypeAlias`, `OpaqueType`, `InterfaceDeclaration`, `DeclareVariable`, `DeclareFunction`, `DeclareClass`, `DeclareModule`, `DeclareModuleExports`, `DeclareExportDeclaration`, `DeclareExportAllDeclaration`, `DeclareInterface`, `DeclareTypeAlias`, `DeclareOpaqueType`, `EnumDeclaration`187188### Expressions (`expressions.rs`, ~35 types)189190**Core**: `Identifier`, `CallExpression`, `MemberExpression`, `OptionalCallExpression`, `OptionalMemberExpression`, `BinaryExpression`, `LogicalExpression`, `UnaryExpression`, `UpdateExpression`, `ConditionalExpression`, `AssignmentExpression`, `SequenceExpression`, `ArrowFunctionExpression` (+ `ArrowFunctionBody` enum), `FunctionExpression`, `ObjectExpression` (+ `ObjectExpressionProperty` enum, `ObjectProperty`, `ObjectMethod`), `ArrayExpression`, `NewExpression`, `TemplateLiteral`, `TaggedTemplateExpression`, `AwaitExpression`, `YieldExpression`, `SpreadElement`, `MetaProperty`, `ClassExpression` (+ `ClassBody`), `PrivateName`, `Super`, `Import`, `ThisExpression`, `ParenthesizedExpression`, `JSXElement`, `JSXFragment`, `AssignmentPattern`191192**TypeScript expressions**: `TSAsExpression`, `TSSatisfiesExpression`, `TSNonNullExpression`, `TSTypeAssertion`, `TSInstantiationExpression`193194**Flow expressions**: `TypeCastExpression`195196TypeScript and Flow type annotation bodies (e.g., `TSTypeAnnotation`, type parameters) use `serde_json::Value` for pass-through round-tripping rather than fully-typed structs. This is sufficient since the compiler doesn't inspect these deeply.197198### Literals (`literals.rs`, 7 types)199200`StringLiteral`, `NumericLiteral`, `BooleanLiteral`, `NullLiteral`, `BigIntLiteral`, `RegExpLiteral`, `TemplateElement` (+ `TemplateElementValue`)201202### Patterns (`patterns.rs`, ~5 types)203204`PatternLike` enum: `Identifier`, `ObjectPattern`, `ArrayPattern`, `AssignmentPattern`, `RestElement`, `MemberExpression`205206`ObjectPatternProperty` enum: `ObjectProperty` (as `ObjectPatternProp`), `RestElement`207208### JSX (`jsx.rs`, ~15 types)209210`JSXElement`, `JSXFragment`, `JSXOpeningElement`, `JSXClosingElement`, `JSXOpeningFragment`, `JSXClosingFragment`, `JSXAttribute`, `JSXSpreadAttribute`, `JSXExpressionContainer`, `JSXSpreadChild`, `JSXText`, `JSXEmptyExpression`, `JSXIdentifier`, `JSXMemberExpression`, `JSXNamespacedName`211212**Helper enums**: `JSXChild`, `JSXElementName`, `JSXAttributeItem`, `JSXAttributeName`, `JSXAttributeValue`, `JSXExpressionContainerExpr`, `JSXMemberExprObject`213214### Operators (`operators.rs`, 5 enums)215216`BinaryOperator`, `LogicalOperator`, `UnaryOperator`, `UpdateOperator`, `AssignmentOperator` — all variants mapped to their JS string representations via `#[serde(rename)]`.217218### Common types (`common.rs`)219220`Position` (line, column, optional index), `SourceLocation` (start, end, optional filename, optional identifierName), `Comment` enum (CommentBlock | CommentLine), `CommentData`, `BaseNode`221222### Top-level types (`lib.rs`)223224`File`, `Program`, `SourceType`, `InterpreterDirective`225226### Catch-all / Unknown variants: statements only227228Most enums do **not** have catch-all `Unknown(serde_json::Value)` variants: an unmodeled node type fails deserialization so the gap gets fixed rather than silently passing through an opaque blob.229230`Statement` is the one deliberate exception. Real TS module-interop syntax (`import x = require(...)`, `export = x`, `export as namespace X`) is legal Babel output that the model does not cover, and failing deserialization there failed entire files the TS reference compiles fine. `Statement::Unknown(UnknownStatement)` carries the complete raw node: top-level unknowns are preserved verbatim in output, function-body unknowns degrade to the standard `UnsupportedNode` bailout. Deserialization still dispatches modeled `type` tags through a typed helper, so a malformed modeled node errors with its precise message instead of degrading to `Unknown`; only genuinely unmodeled tags take the catch-all. The `known_statements!` macro in `statements.rs` is the single source for that dispatch.231232Expression/declaration/pattern enums keep the strict no-catch-all rule.233234This is distinct from unknown *fields*, which are silently dropped (see design decision #6 on `deny_unknown_fields`). An unknown field on a known node is harmless.235236### Union types as enums237238Fields typed as `Expression`, `Statement`, `LVal`, `Pattern`, etc. in Babel are Rust enums with `#[serde(tag = "type")]`. Where fields accept a union of specific types (e.g., `ObjectExpression.properties: Array<ObjectMethod | ObjectProperty | SpreadElement>`), purpose-specific enums are used.239240---241242## Common Types243244```rust245#[derive(Debug, Clone, Serialize, Deserialize)]246pub struct Position {247    pub line: u32,248    pub column: u32,249    #[serde(default, skip_serializing_if = "Option::is_none")]250    pub index: Option<u32>,251}252253#[derive(Debug, Clone, Serialize, Deserialize)]254pub struct SourceLocation {255    pub start: Position,256    pub end: Position,257    #[serde(default, skip_serializing_if = "Option::is_none")]258    pub filename: Option<String>,259    #[serde(default, skip_serializing_if = "Option::is_none", rename = "identifierName")]260    pub identifier_name: Option<String>,261}262263#[derive(Debug, Clone, Serialize, Deserialize)]264#[serde(tag = "type")]265pub enum Comment {266    CommentBlock(CommentData),267    CommentLine(CommentData),268}269270#[derive(Debug, Clone, Serialize, Deserialize)]271pub struct CommentData {272    pub value: String,273    #[serde(default, skip_serializing_if = "Option::is_none")]274    pub start: Option<u32>,275    #[serde(default, skip_serializing_if = "Option::is_none")]276    pub end: Option<u32>,277    #[serde(default, skip_serializing_if = "Option::is_none")]278    pub loc: Option<SourceLocation>,279}280```281282Note: `Position.index` and `SourceLocation.filename` are `Option` — Babel doesn't always emit these fields.283284---285286## Top-Level Types287288```rust289/// The root type returned by @babel/parser290#[derive(Debug, Clone, Serialize, Deserialize)]291pub struct File {292    #[serde(flatten)]293    pub base: BaseNode,294    pub program: Program,295    #[serde(default)]296    pub comments: Vec<Comment>,297    #[serde(default)]298    pub errors: Vec<serde_json::Value>,299}300301#[derive(Debug, Clone, Serialize, Deserialize)]302pub struct Program {303    #[serde(flatten)]304    pub base: BaseNode,305    pub body: Vec<Statement>,306    #[serde(default)]307    pub directives: Vec<Directive>,308    #[serde(rename = "sourceType")]309    pub source_type: SourceType,310    #[serde(default)]311    pub interpreter: Option<InterpreterDirective>,312    #[serde(rename = "sourceFile", default, skip_serializing_if = "Option::is_none")]313    pub source_file: Option<String>,314}315316#[derive(Debug, Clone, Serialize, Deserialize)]317#[serde(rename_all = "lowercase")]318pub enum SourceType {319    Module,320    Script,321}322```323324`Program.body` uses `Vec<Statement>` directly — declarations (import/export, TS, Flow) are variants of the `Statement` enum.325326---327328## Round-Trip Test Infrastructure329330### Overview331332```333                   Node.js                          Rust334                   ──────                          ────335fixture.js ──> @babel/parser ──> JSON ──> serde::from_str ──> serde::to_string ──> JSON336                                  │                                                  │337                                  └──────────────── diff ────────────────────────────┘338```339340### Node.js script: `compiler/scripts/babel-ast-to-json.mjs`341342Parses each fixture file with Babel and writes the AST JSON to a temp directory. Takes two arguments: source directory and output directory.343344```javascript345import { parse } from '@babel/parser';346// ...347const FIXTURE_DIR = process.argv[2]; // source dir with JS/TS files348const OUTPUT_DIR = process.argv[3];  // output dir for JSON files349```350351**Key details**:352- Uses `@babel/parser` directly (not Hermes) with `errorRecovery: true` and `allowReturnOutsideFunction: true`353- Selects plugins based on content: `['flow', 'jsx']` for files containing `@flow`, otherwise `['typescript', 'jsx']`354- Always uses `sourceType: 'module'`355- Matches `**/*.{js,ts,tsx,jsx}` files356- Writes each fixture's AST as a separate `.json` file357- Writes `.parse-error` marker files for fixtures that fail to parse (skipped by the Rust test)358359### JSON normalization360361Before diffing, both the original and round-tripped JSON are normalized on the Rust side:3623631. **Key ordering**: Both JSONs are parsed as `serde_json::Value`, keys are recursively sorted, then compared.3642. **`undefined` vs absent**: `JSON.stringify` omits `undefined` values; serde's `skip_serializing_if = "Option::is_none"` does the same.3653. **Number precision**: Whole-number floats (e.g., `1.0`) are normalized to integers (e.g., `1`) for comparison.366367### Rust test: `compiler/crates/react_compiler_ast/tests/round_trip.rs`368369The test walks all `.json` files in the fixture directory, deserializes each into `File`, re-serializes, normalizes both sides, and diffs. It reports the first 5 failures with unified diffs (capped at 50 lines per fixture) using the `similar` crate.370371The fixture JSON directory is specified via the `FIXTURE_JSON_DIR` environment variable, with a fallback to `tests/fixtures/` alongside the test file.372373### Test runner: `compiler/scripts/test-babel-ast.sh`374375```bash376#!/bin/bash377set -e378# Usage: bash compiler/scripts/test-babel-ast.sh [fixture-source-dir]379# Defaults to the compiler's own test fixtures.380```381382Generates fixture JSONs into a temp dir, runs the Rust round-trip test, and cleans up. Accepts an optional fixture source directory argument.383384**Running the test**:385386```bash387bash compiler/scripts/test-babel-ast.sh388```389390---391392## Remaining Work393394None — this plan is complete. All `Unknown` catch-all variants have been removed from every enum. During removal, three node types that were previously handled by the `Unknown` fallback were promoted to proper typed variants in the `Expression` enum: `JSXElement`, `JSXFragment`, and `AssignmentPattern`.395396Scope info types and scope resolution testing are tracked in [rust-port-0002-scope-types.md](rust-port-0002-scope-types.md).397398---399400## Resolved Risks401402### `#[serde(flatten)]` + `#[serde(tag = "type")]` interaction403404This combination works correctly. No macro fallback was needed. The `BaseNode` is flattened into each node struct, and enums use `#[serde(tag = "type")]` for dispatch. The `BaseNode.node_type` field (renamed from `"type"`) handles the case where `BaseNode` is deserialized outside of a tagged enum context.405406### Floating point precision407408Resolved via the `normalize_json` function in the round-trip test. Whole-number f64 values are normalized to i64 before comparison (e.g., `1.0` → `1`).409410### Fixture parse failures4114123 of 1717 fixtures fail to parse with `@babel/parser` and are skipped (marked with `.parse-error` files). This is expected — some fixtures use intentionally invalid syntax.413414### Performance415416All 1714 fixtures round-trip in ~12 seconds (debug build). Not a concern.417418### Field presence ambiguity419420Resolved empirically via the round-trip test. Fields that Babel always emits (even as `null`) use `Option<T>` without `skip_serializing_if`. Fields that may be absent use `#[serde(default, skip_serializing_if = "Option::is_none")]`. The test is the source of truth.