7 raco decompile: Decompiling Bytecode

The raco decompile command takes the path of a bytecode file (which usually has the file extension ".zo") or a source file with an associated bytecode file (usually created with raco make) and converts the bytecode file’s content back to an approximation of Racket code. Decompiled bytecode is mostly useful for checking the compiler’s transformation and optimization of the source program.

The raco decompile command accepts the following command-line flags:

• --force — skip modification-date comparison on the given file’s path and an associated ".zo" file (if any)

• -n ‹n› or --columns ‹n› — format output for a display with ‹n› columns

Many forms in the decompiled code, such as module, define, and lambda, have the same meanings as always. Other forms and transformations are specific to the rendering of bytecode, and they reflect a specific execution model:

• Top-level variables, variables defined within the module, and variables imported from other modules are prefixed with _, which helps expose the difference between uses of local variables versus other variables. Variables imported from other modules, moreover, have a suffix starting with @ that indicates the source module. Finally, imported variables with constantness have a midfix: :c to indicate constant shape across all instantiations, :f to indicate a fixed value after initialization, :p to indicate a procedure, :P to indicate a procedure that preserves continuation marks on return, :t to indicate a structure type, :mk to indicate a structure constructor, :? to indicate a structure predicate, :ref to indicate a structure accessor, or :set! to indicate a structure mutator.

  Non-local variables are always accessed indirectly though an implicit #%globals or #%modvars variable that resides on the value stack (which otherwise contains local variables). Variable accesses are further wrapped with #%checked when the compiler cannot prove that the variable will be defined before the access.

  Uses of core primitives are shown without a leading _, and they are never wrapped with #%checked.

• Local-variable access may be wrapped with #%sfs-clear, which indicates that the variable-stack location holding the variable will be cleared to prevent the variable’s value from being retained by the garbage collector. Variables whose name starts with unused are never actually stored on the stack, and so they never have #%sfs-clear annotations. (The bytecode compiler normally eliminates such bindings, but sometimes it cannot, either because it cannot prove that the right-hand side produces the right number of values, or the discovery that the variable is unused happens too late with the compiler.)

  Mutable variables are converted to explicitly boxed values using #%box, #%unbox, and #%set-boxes! (which works on multiple boxes at once). A set!-rec-values operation constructs mutually-recursive closures and simultaneously updates the corresponding variable-stack locations that bind the closures. A set!, set!-values, or set!-rec-values form is always used on a local variable before it is captured by a closure; that ordering reflects how closures capture values in variable-stack locations, as opposed to stack locations.

• In a lambda form, if the procedure produced by the lambda has a name (accessible via object-name) and/or source-location information, then it is shown as a quoted constant at the start of the procedure’s body. Afterward, if the lambda form captures any bindings from its context, those bindings are also shown in a quoted constant. Neither constant corresponds to a computation when the closure is called, though the list of captured bindings corresponds to a closure allocation when the lambda form itself is evaluated.

  A lambda form that closes over no bindings is wrapped with #%closed plus an identifier that is bound to the closure. The binding’s scope covers the entire decompiled output, and it may be referenced directly in other parts of the program; the binding corresponds to a constant closure value that is shared, and it may even contain cyclic references to itself or other constant closures.

• A form (#%apply-values proc expr) is equivalent to (call-with-values (lambda () expr) proc), but the run-time system avoids allocating a closure for expr. Similarly, a #%call-with-immediate-continuation-mark call is equivalent to a call-with-immediate-continuation-mark call, but avoiding a closure allocation.

• A define-values form may have (begin '%%inline-variant%% expr1 expr2) for its expression, in which case expr2 is the normal result, but expr1 may be inlined for calls to the definition from other modules. Definitions of functions without an '%%inline-variant%% are never inlined across modules.

• Function arguments and local bindings that are known to have a particular type have names that embed the known type. For example, an argument might have a name that starts argflonum or a local binding might have a name that starts flonum to indicate a flonum value.

• A #%decode-syntax form corresponds to a syntax object.

7.1 API for Decompiling

7.2 API for Parsing Bytecode

The compiler/zo-parse module re-exports compiler/zo-structs in addition to zo-parse.

The parsed bytecode is returned in a linkl-directory or linkl-bundle structure—the latter only for the compilation of a module that contains no submodules.

Within a linklet, the bytecode representation of an expression is closer to an S-expression than a traditional, flat control string. For example, an if form is represented by a branch structure that has three fields: a test expression, a “then” expression, and an “else” expression. Similarly, a function call is represented by an application structure that has a list of argument expressions.

Storage for local variables or intermediate values (such as the arguments for a function call) is explicitly specified in terms of a stack. For example, execution of an application structure reserves space on the stack for each argument result. Similarly, when a let-one structure (for a simple let) is executed, the value obtained by evaluating the right-hand side expression is pushed onto the stack, and then the body is evaluated. Local variables are always accessed as offsets from the current stack position. When a function is called, its arguments are passed on the stack. A closure is created by transferring values from the stack to a flat closure record, and when a closure is applied, the saved values are restored on the stack (though possibly in a different order and likely in a more compact layout than when they were captured).

When a sub-expression produces a value, then the stack pointer is restored to its location from before evaluating the sub-expression. For example, evaluating the right-hand size for a let-one structure may temporarily push values onto the stack, but the stack is restored to its pre-let-one position before pushing the resulting value and continuing with the body. In addition, a tail call resets the stack pointer to the position that follows the enclosing function’s arguments, and then the tail call continues by pushing onto the stack the arguments for the tail-called function.

Values for global and module-level variables are not put directly on the stack, but instead stored in “buckets,” and an array of accessible buckets is kept on the stack. When a closure body needs to access a global variable, the closure captures and later restores the bucket array in the same way that it captured and restores a local variable. Mutable local variables are boxed similarly to global variables, but individual boxes are referenced from the stack and closures.

7.3 API for Marshaling Bytecode

7.4 Bytecode Representation

The compiler/zo-structs library defines the bytecode structures that are produced by zo-parse and consumed by decompile and zo-marshal.

Warning: The compiler/zo-structs library exposes internals of the Racket bytecode abstraction. Unlike other Racket libraries, compiler/zo-structs is subject to incompatible changes across Racket versions.

7.4.1 Prefix

Module and top-level compilation produce one or more linklets that represent independent evaluation in a specific phase. Even a single top-level expression or a module with only run-time code will generate multiple linklets to implement metadata and syntax data. A module with no submodules is represented directly by a linkl-bundle, while any other compiled form is represented by a linkl-directory.

A linklet bundle maps an integer to a linklet representing forms to evaluate at the integer-indicated phase. Symbols are mapped to metadata, such as a module’s name as compiled or a linklet implementing literal syntax objects. A linklet directory normally maps '() to the main linklet bundle for a module or a single top-level form; for a linklet directory that corresponds to a sequence of top-level forms, however, there is no “main” linklet bundle, and symbol forms of integers are used to order the linkets.

For a module with submodules, the linklet directory maps submodule paths (as lists of symbols) to linklet bundles for the corresponding submodules.

The name of a linklet is for debugging purposes, similar to the inferred name of a lambda form.

The importss list of lists describes the linklet’s imports. Each of the elements of the out list corresponds to an import source, and each element of an inner list is the symbolic name of an export from that source. The import-shapess list is in parallel to imports; it reflects optimization assumptions by the compiler that are used by the bytecode validator and checked when the linklet is instantiated.

The exports list describes the linklet’s defined names that are exported. The internals list describes additional definitions within the linket, but they are not accessible from the outside of a linklet or one of its instances; a #f can appear in place of an unreferenced internal definition that has been removed. The lifts list is an extension of internals for procedures that are lifted by the compiler; these procedures have certain properties that can be checked by the bytecode validator.

Each symbol in exports, internals, and lifts must be distinct from any other symbol in those lists. The source-names table maps symbols in exports, internals, and lifts to other symbols, potentially not distinct, that correspond to original source names for the definition. The source-names table is used only for debugging.

When a linklet is instantiated, variables corresponding to the flattening of the lists importss, exports, internals, and lifts are placed in an array (in that order) for access via toplevel references. The initial slot is reserved for a variable-like reference that strongly retains a connection to an instance of its enclosing linklet.

The bodys list is the executable content of the linklet. The value of the last element in bodys may be returned when the linklet is instantiated, depending on the way that it’s instantiated.

The max-let-depth field indicates the maximum size of the stack that will be created by any body.

The need-instance-access? boolean indicates whether the linklet contains a toplevel for position 0. A #t is allowed (but suboptimal) if not such reference is present in the linklet body.

7.4.2 Forms and Inline Variants

After rhs is evaluated, the stack is restored to its depth from before evaluating rhs.

7.4.3 Expressions

The closure-map field is a vector of stack positions that are captured when evaluating the lambda form to create a closure. The closure-types field provides a corresponding list of types, but no distinction is made between normal values and boxed values; also, this information is redundant, since it can be inferred by the bindings referenced though closure-map.

When a closure captures top-level or module-level variables or refers to a syntax-object constant, the variables and constants are represented in the closure by capturing a prefix (in the sense of prefix). The toplevel-map field indicates which top-level variables (i.e., linklet imports and definitions) are actually used by the closure (so that variables in a prefix can be pruned by the run-time system if they become unused) and whether any syntax objects are used (so that the syntax objects as a group can be similarly pruned). A #f value indicates either that no prefix is captured or all variables and syntax objects in the prefix should be considered used. Otherwise, numbers in the set indicate which variables and lifted variables are used. Variables are numbered consecutively by position in the prefix starting from 0, but the number equal to the number of non-lifted variables corresponds to syntax objects (i.e., the number is include if any syntax-object constant is used). Lifted variables are numbered immediately afterward—which means that, if the prefix contains any syntax objects, lifted-variable numbers are shifted down relative to a toplevel by the number of syntax object in the prefix (which makes the toplevel-map set more compact).

When the function is called, the rest-argument list (if any) is pushed onto the stack, then the normal arguments in reverse order, then the closure-captured values in reverse order. Thus, when body is run, the first value on the stack is the first value captured by the closure-map array, and so on.

The max-let-depth field indicates the maximum stack depth created by body plus the arguments and closure-captured values pushed onto the stack. The body field is the expression for the closure’s body.

Changed in version 6.1.1.8 of package zo-lib: Added a number to toplevel-map to indicate whether any syntax object is used, shifting numbers for lifted variables up by one if any syntax object is in the prefix.

After rhs is evaluated, the stack is restored to its depth from before evaluating rhs. Note that the new slot is created before evaluating rhs.

After rhs is evaluated, the stack is restored to its depth from before evaluating rhs.

When the toplevel is an expression, if both const? and ready? are #t, then the variable definitely will be defined, its value stays constant, and the constant is effectively the same for every module instantiation. If only const? is #t, then the value is constant, but it may vary across instantiations. If only ready? is #t, then the variable definitely will be defined, but its value may change. If const? and ready? are both #f, then a check is needed to determine whether the variable is defined.

When the toplevel is the left-hand side for def-values, then const? is #f. If ready? is #t, the variable is marked as immutable after it is defined.

After test is evaluated, the stack is restored to its depth from before evaluating test.

After each of key and val is evaluated, the stack is restored to its depth from before evaluating key or val.

After each form in forms is evaluated, the stack is restored to its depth from before evaluating the form.

After each expression in seq is evaluated, the stack is restored to its depth from before evaluating the expression.

Unlike the begin0 source form, the first expression in seq is never in tail position, even if it is the only expression in the list.

The value of constant? is true when the toplevel field is not #t but the referenced variable is known to be constant. The value of from-unsafe? is true when the module that created the reference was compiled in unsafe mode.

After rhs is evaluated, the stack is restored to its depth from before evaluating rhs.

After each of key and val is evaluated, the stack is restored to its depth from before evaluating key or val.