On this page:
1.1 Bytecode Files
1.2 Dependency Files
1.3 API for Making Bytecode
make-compilation-manager-load/  use-compiled-handler
managed-compile-zo
managed-compiled-context-key
make-compilation-context-error-display-handler
trust-existing-zos
make-caching-managed-compile-zo
manager-compile-notify-handler
manager-trace-handler
manager-skip-file-handler
current-path->mode
file-stamp-in-collection
file-stamp-in-paths
get-file-sha1
get-compiled-file-sha1
with-compile-output
parallel-lock-client
compile-lock->parallel-lock-client
make-compile-lock
install-module-hashes!
1.4 API for Parallel Builds
parallel-compile-files
parallel-compile
1.5 Compilation Manager Hook for Syntax Transformers
register-external-file
register-external-module
1.6 API for Simple Bytecode Creation
compile-file
1.7 API for Bytecode Paths
get-compilation-dir+  name
get-compilation-dir
get-compilation-bytecode-file
1.8 Compiling to Raw Bytecode

1 raco make: Compiling Source to Bytecode

The raco make command accept filenames for Racket modules to be compiled to bytecode format. Modules are re-compiled only if the source Racket file is newer than the bytecode file and has a different SHA-1 hash, or if any imported module is recompiled or has a different SHA-1 hash for its compiled form plus dependencies.

The raco make command accepts a few flags:

1.1 Bytecode Files

A file "name.ext" is compiled to bytecode that is saved as "compiled/name_ext.zo" relative to the file. As a result, the bytecode file is normally used automatically when "name.ext" is required as a module, since the underlying load/use-compiled operation detects such a bytecode file.

For example, in a directory that contains the following files:

then

  raco make a.rkt

triggers the creation of "compiled/a_rkt.zo", "compiled/b_rkt.zo", and "compiled/c_rkt.zo". A subsequent

  racket a.rkt

loads bytecode from the generated ".zo" files, paying attention to the ".rkt" sources only to confirm that each ".zo" file has a later timestamp.

In contrast,

  raco make b.rkt c.rkt

would create only "compiled/b_rkt.zo" and "compiled/c_rkt.zo", since neither "b.rkt" nor "c.rkt" imports "a.rkt".

1.2 Dependency Files

In addition to a bytecode file, raco make creates a file "compiled/name_ext.dep" that records dependencies of the compiled module on other module files and the source file’s SHA-1 hash. Using this dependency information, a re-compilation request via raco make can consult both the source file’s timestamp/hash and the timestamps/hashes for the bytecode of imported modules. Furthermore, imported modules are themselves compiled as necessary, including updating the bytecode and dependency files for the imported modules, transitively.

Continuing the raco make a.rkt example from the previous section, the raco make command creates "compiled/a_rkt.dep", "compiled/b_rkt.dep", and "compiled/c_rkt.dep" at the same time as the ".zo" files. The "compiled/a_rkt.dep" file records the dependency of "a.rkt" on "b.rkt", "c.rkt" and the racket library. If the "b.rkt" file is modified (so that its timestamp and SHA-1 hash changes), then running

  raco make a.rkt

again rebuilds "compiled/a_rkt.zo" and "compiled/b_rkt.zo".

For module files that are within library collections, raco setup uses the same ".zo" and ".dep" conventions and files as raco make, so the two tools can be used together.

1.3 API for Making Bytecode

 (require compiler/cm) package: base
The compiler/cm module implements the compilation and dependency management used by raco make and raco setup.

procedure

(make-compilation-manager-load/use-compiled-handler 
  [delete-zos-when-rkt-file-does-not-exist? 
  #:security-guard security-guard]) 
  (path? (or/c symbol? false/c) . -> . any)
  delete-zos-when-rkt-file-does-not-exist? : any/c = #f
  security-guard : (or/c security-guard? #f) = #f
Returns a procedure suitable as a value for the current-load/use-compiled parameter. The returned procedure passes its arguments on to the current-load/use-compiled procedure that is installed when make-compilation-manager-load/use-compiled-handler is called, but first it automatically compiles a source file to a ".zo" file if

If SHA-1 hashes override a timestamp-based decision to recompile the file, then the target ".zo" file’s timestamp is updated to the current time.

After the handler procedure compiles a ".zo" file, it creates a corresponding ".dep" file that lists the current version and the identification of every file that is directly required by the module in the compiled file. Additional dependencies can be installed during compilation via compiler/cm-accomplice. The ".dep" file also records the SHA-1 hash of the module’s source, and it records a combined SHA-1 hash of all of the dependencies that includes their recursive dependencies.

The handler caches timestamps when it checks ".dep" files, and the cache is maintained across calls to the same handler. The cache is not consulted to compare the immediate source file to its ".zo" file, which means that the caching behavior is consistent with the caching of the default module name resolver (see current-module-name-resolver).

If use-compiled-file-paths contains an empty list when make-compilation-manager-load/use-compiled-handler is called, then exn:fail:contract exception is raised.

If the delete-zos-when-rkt-file-does-not-exist? argument is a true value, then the returned handler will delete ".zo" files when there is no corresponding original source file.

If the security-guard argument is supplied, it is used when creating ".zo" files, ".dep" files, and "compiled/" directories, and when it adjusts the timestamps for existing files. If it is #f, then the security guard in the current-security-guard when the files are created is used (not the security guard at the point make-compilation-manager-load/use-compiled-handler is called).

The continuation the compilation of a module is marked with a managed-compiled-context-key and the module’s source path.

Do not install the result of make-compilation-manager-load/use-compiled-handler when the current namespace contains already-loaded versions of modules that may need to be recompiled—unless the already-loaded modules are never referenced by not-yet-loaded modules. References to already-loaded modules may produce compiled files with inconsistent timestamps and/or ".dep" files with incorrect information.

The handler logs messages to the topic 'compiler/cm at the level 'info. These messages are instances of a compile-event prefab structure:

(struct compile-event (timestamp path type) #:prefab)

The timestamp field is the time at which the event occured in milliseconds since the epoch. The path field is the path of a file being compiled for which the event is about. The type field is a symbol which describes the action the event corresponds to. The currently logged values are 'locking, 'start-compile, 'finish-compile, and 'already-done.

Changed in version 6.1.1.8 of package base: Added identification of the compilation context via managed-compiled-context-key.

procedure

(managed-compile-zo file    
  [read-src-syntax    
  #:security-guard security-guard])  void?
  file : path-string?
  read-src-syntax : (any/c input-port? . -> . syntax?)
   = read-syntax
  security-guard : (or/c security-guard? #f) = #f
Compiles the given module source file to a ".zo", installing a compilation-manager handler while the file is compiled (so that required modules are also compiled), and creating a ".dep" file to record the timestamps of immediate files used to compile the source (i.e., files required in the source).

If file is compiled from source, then read-src-syntax is used in the same way as read-syntax to read the source module. The normal read-syntax is used for any required files, however.

If security-guard is not #f, then the provided security guard is used when creating the "compiled/" directories, ".dep" and ".zo" files, and when it adjusts the timestamps of existing files. If it is #f, then the security guard in the current-security-guard when the files are created is used (not the security guard at the point managed-compile-zo is called).

While compiling file, the error-display-handler parameter is set to (make-compilation-context-error-display-handler (error-display-handler)), so that errors from uncaught exceptions will report the compilation context.

Changed in version 6.1.1.8 of package base: Added error-display-handler configuration.

A key used as a continuation mark key by make-compilation-manager-load/use-compiled-handler for the continuation of a module compilation. The associated value is a path to the module’s source.

Added in version 6.1.1.8 of package base.

procedure

(make-compilation-context-error-display-handler orig-handlers)

  (string? any/c . -> . void?)
  orig-handlers : (string? any/c . -> . void?)
Produces a handler suitable for use as a error-display-handler value, given an existing such value. The generated handler shows information about the compilation context when the handler’s second argument is an exception whose continuation marks include managed-compiled-context-key keys.

Added in version 6.1.1.8 of package base.

parameter

(trust-existing-zos)  boolean?

(trust-existing-zos trust?)  void?
  trust? : any/c
A parameter that is intended for use by setup-plt when installing with pre-built ".zo" files. It causes a compilation-manager load/use-compiled handler to “touch” out-of-date ".zo" files instead of re-compiling from source.

procedure

(make-caching-managed-compile-zo 
  [read-src-syntax 
  #:security-guard security-guard]) 
  (path-string? . -> . void?)
  read-src-syntax : (any/c input-port? . -> . syntax?)
   = read-syntax
  security-guard : (or/c security-guard? #f) = #f
Returns a procedure that behaves like managed-compile-zo (providing the same read-src-syntax each time), but a cache of timestamp information is preserved across calls to the procedure.

parameter

(manager-compile-notify-handler)  (path? . -> . any)

(manager-compile-notify-handler notify)  void?
  notify : (path? . -> . any)
A parameter for a procedure of one argument that is called whenever a compilation starts. The argument to the procedure is the file’s path.

parameter

(manager-trace-handler)  (string? . -> . any)

(manager-trace-handler notify)  void?
  notify : (string? . -> . any)
A parameter for a procedure of one argument that is called to report compilation-manager actions, such as checking a file. The argument to the procedure is a string.

The default value of the parameter logs the argument, along with current-inexact-milliseconds to, a logger named 'compiler/cm at the 'debug level.

parameter

(manager-skip-file-handler)

  (-> path? (or/c (cons/c number? promise?) #f))
(manager-skip-file-handler proc)  void?
  proc : (-> path? (or/c (cons/c number? promise?) #f))
A parameter whose value is called for each file that is loaded and needs recompilation. If the procedure returns a pair, then the file is skipped (i.e., not compiled); the number in the pair is used as the timestamp for the file’s bytecode, and the promise may be forced to obtain a string that is used as hash of the compiled file plus its dependencies. If the procedure returns #f, then the file is compiled as usual. The default is (lambda (x) #f).

parameter

(current-path->mode)

  (or/c #f (-> path? (and/c path? relative-path?)))
(current-path->mode path->mode)  void?
  path->mode : (or/c #f (-> path? (and/c path? relative-path?)))
 = #f
Used by make-compilation-manager-load/use-compiled-handler and make-caching-managed-compile-zo to override use-compiled-file-paths for deciding where to write compiled ".zo" files. If it is #f, then the first element of use-compiled-file-paths is used. If it isn’t #f, then it is called with the original source file’s location and its result is treated the same as if it had been the first element of use-compiled-file-paths.

Note that this parameter is not used by current-load/use-compiled. So if the parameter causes ".zo" files to be placed in different directories, then the correct ".zo" file must still be communicated via use-compiled-file-paths, and one way to do that is to override current-load/use-compiled to delete ".zo" files that would cause the wrong one to be chosen right before they are loaded.

Added in version 6.4.0.14 of package base.

procedure

(file-stamp-in-collection p)

  (or/c (cons/c number? promise?) #f)
  p : path?

procedure

(file-stamp-in-paths p paths)

  (or/c (cons/c number? promise?) #f)
  p : path?
  paths : (listof path?)
Returns the file-modification date and delayed hash of p or its bytecode form (i.e., ".zo" file), whichever exists and is newer, if p is an extension of any path in paths (i.e., exists in the directory, a subdirectory, etc.). Otherwise, the result is #f.

This function is intended for use with manager-skip-file-handler.

procedure

(get-file-sha1 p)  (or/c string? #f)

  p : path?
Computes a SHA-1 hash for the file p; the result is #f if p cannot be opened.

procedure

(get-compiled-file-sha1 p)  (or/c string? #f)

  p : path?
Computes a SHA-1 hash for the bytecode file p, appending any dependency-describing hash available from a ".dep" file when available (i.e., the suffix on p is replaced by ".dep" to locate dependency information). The result is #f if p cannot be opened.

procedure

(with-compile-output p proc)  any

  p : path-string?
  proc : ([port input-port?] [tmp-path path?]  . -> . any)
A wrapper on call-with-atomic-output-file that passes along any security guard put in place by make-compilation-manager-load/use-compiled-handler, etc.

parameter

(parallel-lock-client)

  
(or/c #f
      (->i ([command (or/c 'lock 'unlock)]
            [file bytes?])
           [res (command) (if (eq? command 'lock)
                              boolean?
                              void?)]))
(parallel-lock-client proc)  void?
  proc : 
(or/c #f
      (->i ([command (or/c 'lock 'unlock)]
            [file bytes?])
           [res (command) (if (eq? command 'lock)
                              boolean?
                              void?)]))
Holds the parallel compilation lock client, which is used by the result of make-compilation-manager-load/use-compiled-handler to prevent compilation races between parallel builders.

When proc is #f (the default), no checking for parallel compilation is done (and thus multiple threads or places running compilations via make-compilation-manager-load/use-compiled-handler will potentially corrupt each other’s ".zo" files).

When proc is a function, its first argument is a command, indicating if it wants to lock or unlock the path specified in the second argument.

When the proc 'lock command returns #t, the current builder has obtained the lock for zo-path. Once compilation of zo-path is complete, the builder process must release the lock by calling proc 'unlock with the exact same zo-path.

When the proc 'lock command returns #f, another parallel builder obtained the lock first and has already compiled the zo. The parallel builder should continue without compiling zo-path. (In this case, make-compilation-manager-load/use-compiled-handler’s result will not call proc with 'unlock.)

Example:
> (let* ([lc (parallel-lock-client)]
         [zo-name  #"collects/racket/compiled/draw_rkt.zo"]
         [locked? (and lc (lc 'lock zo-name))]
         [ok-to-compile? (or (not lc) locked?)])
    (dynamic-wind
      (lambda () (void))
      (lambda ()
        (when ok-to-compile?
          (printf "Do compile here ...\n")))
      (lambda ()
        (when locked?
          (lc 'unlock zo-name)))))

Do compile here ...

procedure

(compile-lock->parallel-lock-client pc 
  [cust]) 
  (-> (or/c 'lock 'unlock) bytes? boolean?)
  pc : place-channel?
  cust : (or/c #f custodian?) = #f
Returns a function that follows the parallel-lock-client by communicating over pc. The argument must have be the result of make-compile-lock.

This communication protocol implementation is not kill safe. To make it kill safe, it needs a sufficiently powerful custodian, i.e., one that is not subject to termination (unless all of the participants in the compilation are also terminated). It uses this custodian to create a thread that monitors the threads that are doing the compilation. If one of them is terminated, the presence of the custodian lets another one continue. (The custodian is also used to create a thread that manages a thread safe table.)

procedure

(make-compile-lock)  place-channel?

Creates a place-channel? that can be used with compile-lock->parallel-lock-client to avoid concurrent compilations of the same racket source files in multiple places.

procedure

(install-module-hashes! bstr [start end])  void?

  bstr : btyes?
  start : exact-nonnegatve-integer? = 0
  end : exact-nonnegatve-integer? = (bytes-length bstr)
Adjusts the bytecode representation in bstr (from bytes start to end) to install a hash code, including any submodules within the region. The existing representation should have zero bytes in place of each hash string, which is what write produces for a compiled form.

Added in version 6.3 of package base.

1.4 API for Parallel Builds

 (require setup/parallel-build) package: base
The setup/parallel-build library provides the parallel-compilation functionality of raco setup and raco make.

Both parallel-compile-files and parallel-compile log messages to the topic 'setup/parallel-build at the level 'info. These messages are instances of a parallel-compile-event prefab structure:

(struct parallel-compile-event (worker event) #:prefab)

The worker field is the index of the worker that the created the event. The event field is a compile-event as document in make-compilation-manager-load/use-compiled-handler.

procedure

(parallel-compile-files list-of-files 
  [#:worker-count worker-count 
  #:handler handler]) 
  (or/c void? #f)
  list-of-files : (listof path-string?)
  worker-count : exact-positive-integer? = (processor-count)
  handler : 
(->i ([worker-id exact-integer?]
      [handler-type symbol?]
      [path path-string?]
      [msg string?]
      [out string?]
      [err string?])
     void?)
 = void
The parallel-compile-files utility function is used by raco make to compile a list of paths in parallel. The optional #:worker-count argument specifies the number of compile workers to spawn during parallel compilation. The callback, handler, is called with the symbol 'done as the handler-type argument for each successfully compiled file, 'output when a successful compilation produces stdout/stderr output, 'error when a compilation error has occured, or 'fatal-error when a unrecoverable error occurs. The other arguments give more information for each status update. The return value is (void) if it was successful, or #f if there was an error.

(parallel-compile-files
  source-files
  #:worker-count 4
  #:handler (lambda (type work msg out err)
    (match type
      ['done (when (verbose) (printf " Made ~a\n" work))]
      ['output (printf " Output from: ~a\n~a~a" work out err)]
      [else (printf " Error compiling ~a\n~a\n~a~a"
                    work
                    msg
                    out
                    err)])))

procedure

(parallel-compile worker-count    
  setup-fprintf    
  append-error    
  collects-tree)  (void)
  worker-count : non-negative-integer?
  setup-fprintf : 
(->i ([stage string?] [format string?])
     ()
     #:rest (listof any/c) void)
  append-error : 
(->i ([cc cc?]
      [prefix string?]
      [exn (or/c exn? (cons/c string? string?) #f)]
      [out string?]
      [err srtring?]
      [message string?])
     void?)
  collects-tree : (listof any/c)
The parallel-compile function is used by raco setup to compile collections in parallel. The worker-count argument specifies the number of compilation workers to spawn during parallel compilation. The setup-fprintf and append-error functions communicate intermediate compilation results and errors. The collects-tree argument is a compound datastructure containing an in-memory tree representation of the collects directory.

When the exn argument to append-error is a part of strings, the first string is a long form of the error message, and the second string is a short form (omitting evaluation context information, for example).

Changed in version 6.1.1.8 of package base: Changed append-error to allow a pair of error strings.

1.5 Compilation Manager Hook for Syntax Transformers

 (require compiler/cm-accomplice) package: base

procedure

(register-external-file file    
  [#:indirect? indirect?])  void?
  file : (and path? complete-path?)
  indirect? : any/c = #f
Logs a message (see log-message) at level 'info to a logger named 'cm-accomplice. The message data is a file-dependency prefab structure type with two fields; the first field’s value is file and the second field’s value is #f (to indicate a non-module dependency). If the indirect? argument is true, the data is more specifically an instance of a file-dependency/options prefab structure type that is a subtype of file-dependency with one extra field: a hash table mapping 'indirect to #t.

A compilation manager implemented by compiler/cm looks for such messages to register an external dependency. In response, the compilation manager records (in a ".dep" file) the path as contributing to the implementation of the module currently being compiled. Afterward, if the registered file is modified, the compilation manager will know to recompile the module. An indirect dependency has no effect on recompilation, but it can signal to other tools, such as a package-dependency checker, that the dependency is indirect (and should not imply a direct package dependency).

The include macro, for example, calls this procedure with the path of an included file as it expands an include form.

procedure

(register-external-module file    
  [#:indirect? indirect?])  void?
  file : (and path? complete-path?)
  indirect? : any/c = #f
Like register-external-file, but logs a message with a file-dependency prefab structure type whose second field is #t.

A compilation manager implemented by compiler/cm recognizes the message to register a dependency on a module (which implies a dependency on all of that module’s dependencies, etc.).

1.6 API for Simple Bytecode Creation

 (require compiler/compile-file) package: base

procedure

(compile-file src [dest filter])  path?

  src : path-string?
  dest : path-string?
   = 
(let-values ([(base name dir?) (split-path src)])
  (build-path base "compiled"
              (path-add-suffix name #".zo")))
  filter : (any/c . -> . any/c) = values
Compiles the Racket file src and saves the compiled code to dest. If dest is not provided and the "compiled" subdirectory does not already exist, the subdirectory is created. The result of compile-file is the destination file’s path.

If the filter procedure is provided, it is applied to each source expression, and the result is compiled.

The compile-file procedure is designed for compiling modules files, in that each expression in src is compiled independently. If src does not contain a single module expression, then earlier expressions can affect the compilation of later expressions when src is loaded directly. An appropriate filter can make compilation behave like evaluation, but the problem is also solved (as much as possible) by the compile-zos procedure.

See also managed-compile-zo.

1.7 API for Bytecode Paths

 (require compiler/compilation-path) package: base

Added in version 6.0.1.10 of package base.

procedure

(get-compilation-dir+name path    
  [#:modes modes    
  #:roots roots])  
path? path?
  path : path-string?
  modes : (non-empty-listof (and/c path-string? relative-path?))
   = (use-compiled-file-paths)
  roots : (non-empty-listof (or/c path-string? 'same))
   = (current-compiled-file-roots)
Determines the directory that holds the bytecode form of path plus base name of path.

The directory is determined by checking roots in order, and for each element of roots checking modes in order. The first such directory that contains a file whose name matches path with ".zo" added (in the sense of path-add-suffix) is reported as the return directory path. If no such file is found, the result corresponds to the first elements of modes and roots.

procedure

(get-compilation-dir path    
  [#:modes modes    
  #:roots roots])  path?
  path : path-string?
  modes : (non-empty-listof (and/c path-string? relative-path?))
   = (use-compiled-file-paths)
  roots : (non-empty-listof (or/c path-string? 'same))
   = (current-compiled-file-roots)
The same as get-compilation-dir+name, but returning only the first result.

procedure

(get-compilation-bytecode-file path    
  [#:modes modes    
  #:roots roots])  path?
  path : path-string?
  modes : (non-empty-listof (and/c path-string? relative-path?))
   = (use-compiled-file-paths)
  roots : (non-empty-listof (or/c path-string? 'same))
   = (current-compiled-file-roots)
The same as get-compilation-dir+name, but combines the results and adds a ".zo" suffix to arrive at a bytecode file path.

1.8 Compiling to Raw Bytecode

The --no-deps mode for raco make is an improverished form of the compilation, because it does not track import dependencies. It does, however, support compilation of non-module source in an namespace that initially imports scheme.

Outside of a module, top-level define-syntaxes, module, #%require, define-values-for-syntax, and begin expressions are handled specially by raco make --no-deps: the compile-time portion of the expression is evaluated, because it might affect later expressions.

For example, when compiling the file containing

(require racket/class)
(define f (class object% (super-new)))

the class form from the racket/class library must be bound in the compilation namespace at compile time. Thus, the require expression is both compiled (to appear in the output code) and evaluated (for further computation).

Many definition forms expand to define-syntaxes. For example, define-signature expands to define-syntaxes. In --no-deps mode, raco make --no-deps detects define-syntaxes and other expressions after expansion, so top-level define-signature expressions affect the compilation of later expressions, as a programmer would expect.

In contrast, a load or eval expression in a source file is compiled—but not evaluated!as the source file is compiled. Even if the load expression loads syntax or signature definitions, these will not be loaded as the file is compiled. The same is true of application expressions that affect the reader, such as (read-case-sensitive #t). The -p or --prefix flag for raco make takes a file and loads it before compiling the source files specified on the command line.

By default, the namespace for compilation is initialized by a require of scheme. If the --no-prim flag is specified, the namespace is instead initialized with namespace-require/copy, which allows mutation and redefinition of all initial bindings (other than syntactic forms, in the case of mutation).

In general, a better solution is to put all code to compile into a module and use raco make in its default mode.