7 raco decompile: Decompiling Bytecode

The raco decompile command takes the path of a bytecode file (which usually has the file extension ".zo") or a source file with an associated bytecode file (usually created with raco make) and converts the bytecode file’s content back to an approximation of Racket code. Decompiled bytecode is mostly useful for checking the compiler’s transformation and optimization of the source program.

The raco decompile command accepts the following command-line flags:

• --force — skip modification-date comparison on the given file’s path and an associated ".zo" file (if any)

• -n ‹n› or --columns ‹n› — format output for a display with ‹n› columns

Many forms in the decompiled code, such as module, define, and lambda, have the same meanings as always. Other forms and transformations are specific to the rendering of bytecode, and they reflect a specific execution model:

• Top-level variables, variables defined within the module, and variables imported from other modules are prefixed with _, which helps expose the difference between uses of local variables versus other variables. Variables imported from other modules, moreover, have a suffix starting with @ that indicates the source module. Finally, imported variables with constantness have a midfix: :c to indicate constant shape across all instantiations, :f to indicate a fixed value after initialization, :p to indicate a procedure, :P to indicate a procedure that preserves continuation marks on return, :t to indicate a structure type, :mk to indicate a structure constructor, :? to indicate a structure predicate, :ref to indicate a structure accessor, or :set! to indicate a structure mutator.

  Non-local variables are always accessed indirectly though an implicit #%globals or #%modvars variable that resides on the value stack (which otherwise contains local variables). Variable accesses are further wrapped with #%checked when the compiler cannot prove that the variable will be defined before the access.

  Uses of core primitives are shown without a leading _, and they are never wrapped with #%checked.

• Local-variable access may be wrapped with #%sfs-clear, which indicates that the variable-stack location holding the variable will be cleared to prevent the variable’s value from being retained by the garbage collector. Variables whose name starts with unused are never actually stored on the stack, and so they never have #%sfs-clear annotations. (The bytecode compiler normally eliminates such bindings, but sometimes it cannot, either because it cannot prove that the right-hand side produces the right number of values, or the discovery that the variable is unused happens too late with the compiler.)

  Mutable variables are converted to explicitly boxed values using #%box, #%unbox, and #%set-boxes! (which works on multiple boxes at once). A set!-rec-values operation constructs mutually-recursive closures and simultaneously updates the corresponding variable-stack locations that bind the closures. A set!, set!-values, or set!-rec-values form is always used on a local variable before it is captured by a closure; that ordering reflects how closures capture values in variable-stack locations, as opposed to stack locations.

• In a lambda form, if the procedure produced by the lambda has a name (accessible via object-name) and/or source-location information, then it is shown as a quoted constant at the start of the procedure’s body. Afterward, if the lambda form captures any bindings from its context, those bindings are also shown in a quoted constant. Neither constant corresponds to a computation when the closure is called, though the list of captured bindings corresponds to a closure allocation when the lambda form itself is evaluated.

  A lambda form that closes over no bindings is wrapped with #%closed plus an identifier that is bound to the closure. The binding’s scope covers the entire decompiled output, and it may be referenced directly in other parts of the program; the binding corresponds to a constant closure value that is shared, and it may even contain cyclic references to itself or other constant closures.

• A form (#%apply-values proc expr) is equivalent to (call-with-values (lambda () expr) proc), but the run-time system avoids allocating a closure for expr. Similarly, a #%call-with-immediate-continuation-mark call is equivalent to a call-with-immediate-continuation-mark call, but avoiding a closure allocation.

• A define-values form may have (begin '%%inline-variant%% expr1 expr2) for its expression, in which case expr2 is the normal result, but expr1 may be inlined for calls to the definition from other modules. Definitions of functions without an '%%inline-variant%% are never inlined across modules.

• Function arguments and local bindings that are known to have a particular type have names that embed the known type. For example, an argument might have a name that starts argflonum or a local binding might have a name that starts flonum to indicate a flonum value.

• A #%decode-syntax form corresponds to a syntax object.

7.1 API for Decompiling

7.2 API for Parsing Bytecode

The compiler/zo-parse module re-exports compiler/zo-structs in addition to zo-parse.

The parsed bytecode is returned in a compilation-top structure. For a compiled module, the compilation-top structure will contain a mod structure. For a top-level sequence, it will normally contain a seq or splice structure with a list of top-level declarations and expressions.

The bytecode representation of an expression is closer to an S-expression than a traditional, flat control string. For example, an if form is represented by a branch structure that has three fields: a test expression, a “then” expression, and an “else” expression. Similarly, a function call is represented by an application structure that has a list of argument expressions.

Storage for local variables or intermediate values (such as the arguments for a function call) is explicitly specified in terms of a stack. For example, execution of an application structure reserves space on the stack for each argument result. Similarly, when a let-one structure (for a simple let) is executed, the value obtained by evaluating the right-hand side expression is pushed onto the stack, and then the body is evaluated. Local variables are always accessed as offsets from the current stack position. When a function is called, its arguments are passed on the stack. A closure is created by transferring values from the stack to a flat closure record, and when a closure is applied, the saved values are restored on the stack (though possibly in a different order and likely in a more compact layout than when they were captured).

When a sub-expression produces a value, then the stack pointer is restored to its location from before evaluating the sub-expression. For example, evaluating the right-hand size for a let-one structure may temporarily push values onto the stack, but the stack is restored to its pre-let-one position before pushing the resulting value and continuing with the body. In addition, a tail call resets the stack pointer to the position that follows the enclosing function’s arguments, and then the tail call continues by pushing onto the stack the arguments for the tail-called function.

Values for global and module-level variables are not put directly on the stack, but instead stored in “buckets,” and an array of accessible buckets is kept on the stack. When a closure body needs to access a global variable, the closure captures and later restores the bucket array in the same way that it captured and restores a local variable. Mutable local variables are boxed similarly to global variables, but individual boxes are referenced from the stack and closures.

Quoted syntax (in the sense of quote-syntax) is treated like a global variable, because it must be instantiated for an appropriate phase. A prefix structure within a compilation-top or mod structure indicates the list of global variables and quoted syntax that need to be instantiated (and put into an array on the stack) before evaluating expressions that might use them.

7.3 API for Marshaling Bytecode

7.4 Bytecode Representation

The compiler/zo-structs library defines the bytecode structures that are produced by zo-parse and consumed by decompile and zo-marshal.

Warning: The compiler/zo-structs library exposes internals of the Racket bytecode abstraction. Unlike other Racket libraries, compiler/zo-structs is subject to incompatible changes across Racket versions.

7.4.1 Prefix

The max-let-depth field indicates the maximum stack depth that code creates (not counting the prefix array).

The binding-namess field provides a per-phase mapping from symbols that appear in prefix for top-level def-values forms and in top-level def-syntaxes forms. Each symbol is mapped to an identifier that will be bound (after introduction into the namespace) by the definition.

The prefix field describes top-level variables, module-level variables, and quoted syntax-objects accessed by code.

The code field contains executable code; it is normally a form, but a literal value is represented as itself.

The variable buckets and syntax objects that are recorded in a prefix are accessed by toplevel and topsyntax expression forms.

When an element of stxs is #f, it coresponds to a syntax object that was optimized away at the last minute. The slot must not be referenced by a topsyntax form.

The src-inspector-desc field provides an inspector name that is used within syntax-object bindings. At run time, the prefix gets an inspector, and bindings that reference the same inspector name are granted access capabilities through that inspector.

7.4.2 Forms

After rhs is evaluated, the stack is restored to its depth from before evaluating rhs.

After each form in forms is evaluated, the stack is restored to its depth from before evaluating the form.

After each form in forms is evaluated, the stack is restored to its depth from before evaluating the form.

The provides and requires lists are each an association list from phases to exports or imports. In the case of provides, each phase maps to two lists: one for exported variables, and another for exported syntax. In the case of requires, each phase maps to a list of imported module paths.

The body field contains the module’s run-time (i.e., phase 0) code. The syntax-bodies list has a list of forms for each higher phase in the module body; the phases are in order starting with phase 1. The body forms use prefix, rather than any prefix in place for the module declaration itself, while members of lists in syntax-bodies have their own prefixes. After each form in body or syntax-bodies is evaluated, the stack is restored to its depth from before evaluating the form.

The unexported list contains lists of symbols for unexported definitions that can be accessed through macro expansion and that are implemented through the forms in body and syntax-bodies. Each list in unexported starts with a phase level.

The max-let-depth field indicates the maximum stack depth created by body forms (not counting the prefix array).

The dummy variable is used to access the top-level namespace.

The lang-info value specifies an optional module path that provides information about the module’s implementation language.

The internal-context value describes the lexical context of the body of the module. This value is used by module->namespace. A #f value means that the context is unavailable or empty. A #t value means that the context is computed by re-importing all required modules. A syntax-object value embeds lexical information; the syntax object should contain a vector of two elements, where the first element of the vector is a syntax object for the module’s body, which includes the outside-edge and inside-edge scopes, and the second element of the vector is a syntax object that has just the module’s inside-edge scope.

The binding-names value provides additional information to module->namespace to correlate symbol names for variables and syntax definitions to identifiers that map to those variables. A separate table of names exists for each phase, and a #t mapping for a name indicates that it is mapped but inaccessible (because the relevant scopes are inaccessible).

The flags field records certain properties of the module. The 'cross-phase flag indicates that the module body is evaluated once and the results shared across instances for all phases; such a module contains only definitions of functions, structure types, and structure type properties.

The pre-submodules field records module-declared submodules, while the post-submodules field records module*-declared submodules.

7.4.3 Expressions

The closure-map field is a vector of stack positions that are captured when evaluating the lambda form to create a closure. The closure-types field provides a corresponding list of types, but no distinction is made between normal values and boxed values; also, this information is redundant, since it can be inferred by the bindings referenced though closure-map.

When a closure captures top-level or module-level variables or refers to a syntax-object constant, the variables and constants are represented in the closure by capturing a prefix (in the sense of prefix). The toplevel-map field indicates which top-level and lifted variables are actually used by the closure (so that variables in a prefix can be pruned by the run-time system if they become unused) and whether any syntax objects are used (so that the syntax objects as a group can be similarly pruned). A #f value indicates either that no prefix is captured or all variables and syntax objects in the prefix should be considered used. Otherwise, numbers in the set indicate which variables and lifted variables are used. Variables are numbered consecutively by position in the prefix starting from 0, but the number equal to the number of non-lifted variables corresponds to syntax objects (i.e., the number is include if any syntax-object constant is used). Lifted variables are numbered immediately afterward—which means that, if the prefix contains any syntax objects, lifted-variable numbers are shifted down relative to a toplevel by the number of syntax object in the prefix (which makes the toplevel-map set more compact).

When the function is called, the rest-argument list (if any) is pushed onto the stack, then the normal arguments in reverse order, then the closure-captured values in reverse order. Thus, when body is run, the first value on the stack is the first value captured by the closure-map array, and so on.

The max-let-depth field indicates the maximum stack depth created by body plus the arguments and closure-captured values pushed onto the stack. The body field is the expression for the closure’s body.

Changed in version 6.1.1.8 of package zo-lib: Added a number to toplevel-map to indicate whether any syntax object is used, shifting numbers for lifted variables up by one if any syntax object is in the prefix.

After rhs is evaluated, the stack is restored to its depth from before evaluating rhs. Note that the new slot is created before evaluating rhs.

After rhs is evaluated, the stack is restored to its depth from before evaluating rhs.

When the toplevel is an expression, if both const? and ready? are #t, then the variable definitely will be defined, its value stays constant, and the constant is effectively the same for every module instantiation. If only const? is #t, then the value is constant, but it may vary across instantiations. If only ready? is #t, then the variable definitely will be defined, but its value may change. If const? and ready? are both #f, then a check is needed to determine whether the variable is defined.

When the toplevel is the right-hand side for def-values, then const? is #f. If ready? is #t, the variable is marked as immutable after it is defined.

After test is evaluated, the stack is restored to its depth from before evaluating test.

After each of key and val is evaluated, the stack is restored to its depth from before evaluating key or val.

After each expression in seq is evaluated, the stack is restored to its depth from before evaluating the expression.

Unlike the begin0 source form, the first expression in seq is never in tail position, even if it is the only expression in the list.

After rhs is evaluated, the stack is restored to its depth from before evaluating rhs.

After each of key and val is evaluated, the stack is restored to its depth from before evaluating key or val.

7.4.4 Syntax Objects

The content of wrap is typically cyclic, since it includes scopes that contain bindings that refer to scopes.

The from-inspector-desc and to-inspector-desc fields similarly should be both #f or both non-#f. They record a history of code-inspector replacements.

The bindings list indicates some bindings that are associated with the scope. Each element of the list includes a symbolic name, a list of scopes (including the enclosing one), and the binding for the combination of name and scope set. A given symbol can appear in multiple elements of bindings, but the combination of the symbol and scope set are unique within bindings and across all scopes. The mapping of a symbol and scope set to a binding is recorded with an arbitrary member of the scope set.

The bulk-bindings field lists bindings of all exports from a given module, which is an optimization over including each export in bindings. Elements of bindings take precedence over elements of bulk-bindings, and earlier elements of bulk-bindings take precedence over later elements.

If the scope represents a scope at a particular phase for a group of phase-specific scopes, mark-owner refers to the group.

Scopes within the group are instantiated at different phases on demand. The scopes field lists all of the scopes instantiated for the group, and the phase at which it is instantiated. Each element of scopes must have a multi-owner field value that refers back to the multi-scope.

• path: the referenced module.

• name: the referenced definition within its module.

• phase: the phase of the referenced definition within its module.

• nominal-path: the module that was explicitly imported into the binding context; this path can be different from path when a definition is re-exported.

• nominal-export-name: the name of the binding as exported from nominal-path, which can be different from name due to renaming on export.

• nominal-phase: the phase of the export from nominal-path, which can be different from phase due to re-export from a module that imports at a phase level other than 0.

• import-phase: the phase of the import of nominal-path, which shifted (if non-0) the binding phase relative to the export phase from nominal-path.

• inspector-desc: a name for an inspector (mapped to a specific inspector at run time) that determines access to the definition.

• path: the imported module.

• phase: the phase of the import module’s exports.

• src-phase: the phase at which path was imported; src-phase combined with phase determines the phase of the bindings.

• inspector-desc: a name for an inspector (mapped to a specific inspector at run time) that determines access to the definition.

• exceptions: exports of path that are omitted from the bulk import.

• prefix: a prefix, if any, applied (after exceptions) to each of the imported names.