8 raco demod: Demodularizing Programs
The raco demodularize command (usually used with the shorthand raco demod) takes a Racket module and flattens its dependencies into a single compiled module, potentially with submodules. A file "‹name›.rkt" is demodularized into "‹name›_rkt_merged.zo".
See compiler/demod for an alternative way to use the demodularizer. Using #lang compiler/demod can cooperate with tools like raco make and raco setup, which is especially important for library modules (as opposed to end-user programs).
In its default configuration, raco demod supports flattening a module that represents an end-user program, so it discards all syntax and compile-time support in the module and its dependencies. Submodules are preserved, but their syntax and compile-time support are similarly discarded. The demodularized ".zo" file can be run by passing it as an argument to the racket command-line program, or it can be turned into an executable with raco exe.
Supply the -s or --syntax flag to preserve syntax and compile-time components of the module, so that it can be required the same as the original module. In that case, modules whose instances need to be shared with other libraries should be omitted from the demodularization using -x or --exclude-library. For example, -x racket/base is normally needed.
A large single module generated by the demodularizer is compiled as if (#%declare #:unlimited-compile) is specified, so the value of the PLT_CS_COMPILE_LIMIT environment variable does not limit compilation of the module.
The raco demod command accepts these flags:
-o ‹file› —
writes the flattened module to ‹file› instead of "‹name›_‹ext›_merged.zo" for an input file "‹name›.‹ext›". -x ‹module-path› or --exclude-library ‹module-path› —
excludes the module in ‹module-path› from flattening, as well as all of its dependencies. -e ‹path› or --exclude-module ‹path› —
excludes the module in relative-file ‹path› from flattening, as well as all of its dependencies. For backward compatibility, --exclude-modules is an alias for --exclude-module. -s or --syntax —
preserve syntax objects and phase levels greater than the run-time phase in the flattened result. Otherwise, only the run-time phase is preserved, and unused (or merely exported) definitions are pruned, since they cannot be referenced through syntax. -M or --compile-any —
flattens the module to machine-independent form, instead of recompiling the flattened module to the current platform and Racket virtual machine; the output generated with -M loads more slowly than a machine-specific form, but raco decompile can show the flattened module in a format that is closer to source. See also --dump-mi. -r or --recompile —
(re)compiles the module to machine-dependent form after flattening; this mode is the default. --work ‹dir› —
uses ‹dir› to cache compiled modules in an intermediate form for flattening; using --work with the same ‹dir› for multiple uses of raco demod can greatly speed up demodularization, and since the cache is based on raco make, it works even with different input files or when modules to be flattened have changed since the last use of the cache. -g or --prune-definitions —
increases pruning of definitions that are unreferenced on the unsound assumption that the right-hand side of a definition has no side effect. When syntax is preserved, a definition can be pruned as long as no syntax literal includes an identifier that is bound to the definition. Since these assumptions are unchecked, conversion may not preserve the behavior of the input module. For backward compatibility, --garbage-collect is an alias for --prune-definitions. --dump ‹file› —
writes an S-expression representation of the module’s content to ‹file›, which can be helpful for understanding the content that is in the compiled flatten module. --dump-mi ‹file› —
writes a machine-independent form of the flattened module to ‹file›, the same as -M would write, but useful when -M is not used.
In addition to preserving submodules or of the source module, demodularization may introduce new submodules to hold portions of the flattening. The introduced submodules have names demod-pane- followed by an integer.
Changed in version 1.10 of package compiler-lib: Added -M/--compile-any,
--work, and support for Racket CS.
Changed in version 1.15: Added -x/--exclude-library,
-s/--syntax, --dump,
--dump-mi, --prune-definitions
(as a new name for --garbage-collect),
and preservation of submodules.
8.1 Demodularizing Libraries
Demodularization of a library module with compiler/demod can create a module whose meaning is different than the original, since transitive dependencies (that are not specified as excluded) are copied into the flattened module. That copying can break sharing as needed for generated structure types or bindings. As a specific example, separate copies of racket/base will have distinct and incompatible implementations of keyword arguments for procedures.
To avoid problems, a good general strategy for flattening is
put all modules to be flattened into an "private/amalgam" subcollection of, where modules within "private/amalgam" can freely refer to each other;
create a module "private/amalgam-src.rkt" that requires modules from "private/amalgam" that need to be accessible from outside, where submodules in "private/amalgam-src.rkt" can provided different subsets of bindings from "private/amalgam";
create a module "mine/private/amalgam.rkt" as
#lang compiler/demod "amalgam-src.rkt" #:include (#:dir "amalgam") and
from outside "private/amalgam", use only "private/amalgam.rkt", perhaps via public modules that reprovide from "private/amalgam.rkt".
8.2 Language for Demodularizing
| #lang compiler/demod | package: compiler-lib |
A module using compiler/demod language compiles to a form that is the flattened (in the same sense as raco demod) version of a source module. See also Demodularizing Libraries.
A #lang compiler/demod module body starts with a module-path to flatten, it may be followed by options:
syntax
module-path option ...
option = mode | #:include (mod-spec ...) | #:exclude (mod-spec ...) | #:submodule-include (submod-spec ...) | #:submodule-exclude (submod-spec ...) | #:prune-definitions | #:dump file | #:dump-mi file | #:no-demod mode = #:exe | #:dynamic | #:static mod-spec = #:module module-path | #:dir dir-path | #:collect collect-name submod-spec = identifier | (identifier ...)
The default mode is #:dynamic, which preserves syntax objects and compile-time support (like macros), but does not insist that all modules are copied into the flattened module. For example, if a module is referenced by a combination of submodules within module-path and no other module is reached by the same combination, then the benefit of copying the module into a submodule is limited. The #:static mode is like #:dynamic, but it ensures that all modules are included unless they are specified as excluded. The #:exe mode discards syntax and compile-time support, so it may be suitable for flattening a module that implements an end-user program.
When the #:include option is specified, then only modules covered by a mod-spec will be included in the flattened form; otherwise, all modules are candidates for inclusion. When the #:exclude option is specified, the modules covered by the mod-specs are excluded, even if they would otherwise be included according to a #:include specification. In other words, #:exclude is applied after #:include.
The #:submodule-include and #:submodule-exclude specifications are analogous to #:include and #:exclude, but for submodules immediately with module-path. If mode is #:exe, then the list of inclusions defaults to main and configure-runtime, otherwise the default is to have no specific inclusions.
A mod-spec either indicates a specific module with #:module or it indicates all modules in a given collection (and its subcollections) with #:collect. A collect-name is always a string with /-separated components.
If the #:prune-definitions option is specified, then unused definitions from the original module and its dependencies are more aggressively pruned, but unsoundly. When syntax is preserved for #:dynamic or #:static mode, then all definitions are normally preserved from the original module, because they might be reachable via datum->syntax; when #:prune-definitions is specified, a definition can be pruned if no syntax object literal includes an identifier bound to the definition. Meanwhile, in all modes including #:exe, a definition is normally preserved if its right-hand side might have a side effect, but #:prune-definitions allows pruning on the unchecked assumption that a definition has no side effect. Due to its unchecked assumptions, #:prune-definitions may not preserve the behavior of the input module. As an example of where #:prune-definitions can go wrong, a module could export a macro that expands to a use of syntax-parse, and that use could include a : shorthand to combine a pattern variable and a syntax class (also defined in the module) as one identifier. The identifier would be split into variable and syntax-class components only when the macro is used, so the shorthand does not count as a literal that is bound to the syntax class. In that particular situation, use ~var instead of the shorthand, and then the syntax class is referenced by its own identifier. Meanwhile, a macro that is not exported (directly or indirectly through another macro) can safely use the : shorthand, since its expansions are part of the module’s implementation.
If the #:no-demod option is specified, then mod-spec is not flattened, after all. Instead, the new module requires and reprovides mod-spec and each of its submodules. This mode is always used when a compiler/demod module is expanded, since expansion must produce syntax instead of a compiled module. This mode also may be useful during for development to avoid longer compile times from flattening or to check whether copying of modules for flattening creates any trouble.
A flattened module using compiler/demod has a build dependency on the original module, so a tool like raco make or raco setup will trigger reflattening if the source module changes, but the flattened module does not have a run-time or expand-time dependency on the original module. Modules excluded from the flattening via #:include and #:exclude remain as run-time and expand-time dependencies of the flattened module. In the default #:dynamic mode, additional dependencies may be preserved for modules that cannot be usefully merged, but #:static or #:exe mode copies even those modules into new submodules.
Compilation and expansion of a compiler/demod module creates a "compiled/demod" subdirectory in the same directory as the module. That subdirectory that holds freshly compiled versions of all dependencies of the flattened module in a form that is suitable for demodularization. This extra compilation is managed using compiler/cm, so changes to dependencies can be handled incrementally, but still separate from normal compilation of the dependencies.
Added in version 1.15 of package compiler-lib.