16.2.6 General Phase Levels

A phase can be thought of as a way to separate computations in a pipeline of processes where one produces code that is used by the next. (E.g., a pipeline that consists of a preprocessor process, a compiler, and an assembler.)

Imagine starting two Racket processes for this purpose. If you ignore inter-process communication channels like sockets and files, the processes will have no way to share anything other than the text that is piped from the standard output of one process into the standard input of the other. Similarly, Racket effectively allows multiple invocations of a module to exist in the same process but separated by phase. Racket enforces separation of such phases, where different phases cannot communicate in any way other than via the protocol of macro expansion, where the output of one phases is the code used in the next.

16.2.6.1 Phases and Bindings

Every binding of an identifier exists in a particular phase. The link between a binding and its phase is represented by an integer phase level. Phase level 0 is the phase used for “plain” (or “runtime”) definitions, so

(define age 5)

adds a binding for age into phase level 0. The identifier age can be defined at a higher phase level using begin-for-syntax:

(begin-for-syntax

(define age 5))

With a single begin-for-syntax wrapper, age is defined at phase level 1. We can easily mix these two definitions in the same module or in a top-level namespace, and there is no clash between the two ages that are defined at different phase levels:

> (define age 3)

> (begin-for-syntax     (define age 9))

The age binding at phase level 0 has a value of 3, and the age binding at phase level 1 has a value of 9.

Syntax objects capture binding information as a first-class value. Thus,

#'age

is a syntax object that represents the age binding—but since there are two ages (one at phase level 0 and one at phase level 1), which one does it capture? In fact, Racket imbues #'age with lexical information for all phase levels, so the answer is that #'age captures both.

The relevant binding of age captured by #'age is determined when #'age is eventually used. As an example, we bind #'age to a pattern variable so we can use it in a template, and then we evaluate the template: We use eval here to demonstrate phases, but see Reflection and Dynamic Evaluation for caveats about eval.

> (eval (with-syntax ([age #'age])           #'(displayln age)))

3

The result is 3 because age is used at phase 0 level. We can try again with the use of age inside begin-for-syntax:

> (eval (with-syntax ([age #'age])           #'(begin-for-syntax               (displayln age))))

9

In this case, the answer is 9, because we are using age at phase level 1 instead of 0 (i.e., begin-for-syntax evaluates its expressions at phase level 1). So, you can see that we started with the same syntax object, #'age, and we were able to use it in two different ways: at phase level 0 and at phase level 1.

A syntax object has a lexical context from the moment it first exists. A syntax object that is provided from a module retains its lexical context, and so it references bindings in the context of its source module, not the context of its use. The following example defines button at phase level 0 and binds it to 0, while see-button binds the syntax object for button in module a:

> (module a racket     (define button 0)     (provide (for-syntax see-button))     ; Why not (define see-button #'button)? We explain later.     (define-for-syntax see-button #'button))

> (module b racket     (require 'a)     (define button 8)     (define-syntax (m stx)       see-button)     (m))

> (require 'b)

0

The result of the m macro is the value of see-button, which is #'button with the lexical context of the a module. Even though there is another button in b, the second button will not confuse Racket, because the lexical context of #'button (the value bound to see-button) is a.

Note that see-button is bound at phase level 1 by virtue of defining it with define-for-syntax. Phase level 1 is needed because m is a macro, so its body executes at one phase higher than the context of its definition. Since m is defined at phase level 0, its body is at phase level 1, so any bindings referenced by the body must be at phase level 1.

16.2.6.2 Phases and Modules

A phase level is a module-relative concept. When importing from another module via require, Racket lets us shift imported bindings to a phase level that is different from the original one:

(require "a.rkt")                ; import with no phase shift

(require (for-syntax "a.rkt"))   ; shift phase by +1

(require (for-template "a.rkt")) ; shift phase by -1

(require (for-meta 5 "a.rkt"))   ; shift phase by +5

That is, using for-syntax in require means that all of the bindings from that module will have their phase levels increased by one. A binding that is defined at phase level 0 and imported with for-syntax becomes a phase-level 1 binding:

> (module c racket     (define x 0) ; defined at phase level 0     (provide x))

> (module d racket     (require (for-syntax 'c))     ; has a binding at phase level 1, not 0:     #'x)

Let’s see what happens if we try to create a binding for the #'button syntax object at phase level 0:

> (define button 0)

> (define see-button #'button)

Now both button and see-button are defined at phase 0. The lexical context of #'button will know that there is a binding for button at phase 0. In fact, it seems like things are working just fine if we try to eval see-button:

> (eval see-button)

0

Now, let’s use see-button in a macro:

> (define-syntax (m stx)     see-button)

> (m)

see-button: undefined;

cannot reference undefined identifier

Clearly, see-button is not defined at phase level 1, so we cannot refer to it inside the macro body. Let’s try to use see-button in another module by putting the button definitions in a module and importing it at phase level 1. Then, we will get see-button at phase level 1:

> (module a racket     (define button 0)     (define see-button #'button)     (provide see-button))

> (module b racket     (require (for-syntax 'a)) ; gets see-button at phase level 1     (define-syntax (m stx)       see-button)     (m))

eval:1:0: button: unbound identifier;

also, no #%top syntax transformer is bound

in: button

Racket says that button is unbound now! When a is imported at phase level 1, we have the following bindings:

button     at phase level 1

see-button at phase level 1

So the macro m can see a binding for see-button at phase level 1 and will return the #'button syntax object, which refers to button binding at phase level 1. But the use of m is at phase level 0, and there is no button at phase level 0 in b. That is why see-button needs to be bound at phase level 1, as in the original a. In the original b, then, we have the following bindings:

button     at phase level 0

see-button at phase level 1

In this scenario, we can use see-button in the macro, since see-button is bound at phase level 1. When the macro expands, it will refer to a button binding at phase level 0.

Defining see-button with (define see-button #'button) isn’t inherently wrong; it depends on how we intend to use see-button. For example, we can arrange for m to sensibly use see-button because it puts it in a phase level 1 context using begin-for-syntax:

> (module a racket     (define button 0)     (define see-button #'button)     (provide see-button))

> (module b racket     (require (for-syntax 'a))     (define-syntax (m stx)       (with-syntax ([x see-button])         #'(begin-for-syntax             (displayln x))))     (m))

0

In this case, module b has both button and see-button bound at phase level 1. The expansion of the macro is

(begin-for-syntax

(displayln button))

which works, because button is bound at phase level 1.

Now, you might try to cheat the phase system by importing a at both phase level 0 and phase level 1. Then you would have the following bindings

button     at phase level 0

see-button at phase level 0

button     at phase level 1

see-button at phase level 1

You might expect now that see-button in a macro would work, but it doesn’t:

> (module a racket     (define button 0)     (define see-button #'button)     (provide see-button))

> (module b racket     (require 'a              (for-syntax 'a))     (define-syntax (m stx)       see-button)     (m))

eval:1:0: button: unbound identifier;

also, no #%top syntax transformer is bound

in: button

The see-button inside macro m comes from the (for-syntax 'a) import. For macro m to work, it needs to have button bound at phase 0. That binding exists—it’s implied by (require 'a). However, (require 'a) and (require (for-syntax 'a)) are different instantiations of the same module. The see-button at phase 1 only refers to the button at phase 1, not the button bound at phase 0 from a different instantiation—even from the same source module.

This kind of phase-level mismatch between instantiations can be repaired with syntax-shift-phase-level. Recall that a syntax object like #'button captures lexical information at all phase levels. The problem here is that see-button is invoked at phase 1, but needs to return a syntax object that can be evaluated at phase 0. By default, see-button is bound to #'button at the same phase level. But with syntax-shift-phase-level, we can make see-button refer to #'button at a different relative phase level. In this case, we use a phase shift of -1 to make see-button at phase 1 refer to #'button at phase 0. (Because the phase shift happens at every level, it will also make see-button at phase 0 refer to #'button at phase -1.)

Note that syntax-shift-phase-level merely creates a reference across phases. To make that reference work, we still need to instantiate our module at both phases so the reference and its target have their bindings available. Thus, in module 'b, we still import module 'a at both phase 0 and phase 1—using (require 'a (for-syntax 'a))—so we have a phase-1 binding for see-button and a phase-0 binding for button. Now macro m will work.

> (module a racket     (define button 0)     (define see-button (syntax-shift-phase-level #'button -1))     (provide see-button))

> (module b racket     (require 'a (for-syntax 'a))     (define-syntax (m stx)       see-button)     (m))

> (require 'b)

0

By the way, what happens to the see-button that’s bound at phase 0? Its #'button binding has likewise been shifted, but to phase -1. Since button itself isn’t bound at phase -1, if we try to evaluate see-button at phase 0, we get an error. In other words, we haven’t permanently cured our mismatch problem—we’ve just shifted it to a less bothersome location.

> (module a racket     (define button 0)     (define see-button (syntax-shift-phase-level #'button -1))     (provide see-button))

> (module b racket     (require 'a (for-syntax 'a))     (define-syntax (m stx)       see-button)     (m))

> (module b2 racket     (require 'a)     (eval see-button))

> (require 'b2)

button: undefined;

cannot reference undefined identifier

Mismatches like the one above can also arise when a macro tries to match literal bindings—using syntax-case or syntax-parse.

> (module x racket     (require (for-syntax syntax/parse)              (for-template racket/base))        (provide (all-defined-out))        (define button 0)     (define (make) #'button)     (define-syntax (process stx)       (define-literal-set locals (button))       (syntax-parse stx         [(_ (n (~literal button))) #'#''ok])))

> (module y racket     (require (for-meta 1 'x)              (for-meta 2 'x racket/base))        (begin-for-syntax       (define-syntax (m stx)         (with-syntax ([out (make)])           #'(process (0 out)))))        (define-syntax (p stx)       (m))        (p))

eval:2.0: process: expected the identifier `button'

at: button

in: (process (0 button))

In this example, make is being used in y at phase level 2, and it returns the #'button syntax object—which refers to button bound at phase level 0 inside x and at phase level 2 in y from (for-meta 2 'x). The process macro is imported at phase level 1 from (for-meta 1 'x), and it knows that button should be bound at phase level 1. When the syntax-parse is executed inside process, it is looking for button bound at phase level 1 but it sees only a phase level 2 binding and doesn’t match.

To fix the example, we can provide make at phase level 1 relative to x, and then we import it at phase level 1 in y:

> (module x racket     (require (for-syntax syntax/parse)              (for-template racket/base))        (provide (all-defined-out))        (define button 0)        (provide (for-syntax make))     (define-for-syntax (make) #'button)     (define-syntax (process stx)       (define-literal-set locals (button))       (syntax-parse stx         [(_ (n (~literal button))) #'#''ok])))

> (module y racket     (require (for-meta 1 'x)              (for-meta 2 racket/base))        (begin-for-syntax       (define-syntax (m stx)         (with-syntax ([out (make)])           #'(process (0 out)))))        (define-syntax (p stx)       (m))        (p))

> (require 'y)

'ok