13.11 Sandboxed Evaluation
The racket/sandbox module provides utilities for
creating “sandboxed” evaluators, which are configured in a
particular way and can have restricted resources (memory and time),
filesystem and network access, and much more. Sandboxed evaluators can be
configured through numerous parameters — and the defaults are set
for the common use case where sandboxes are very limited.
The
make-evaluator function creates an evaluator with a
language and
requires specification, and starts
evaluating the given
input-programs. The
make-module-evaluator function creates an evaluator that
works in the context of a given module. The result in either case is a
function for further evaluation.
The returned evaluator operates in an isolated and limited
environment. In particular, filesystem access is restricted. The
allow argument extends the set of files that are readable by
the evaluator to include the specified modules and their imports
(transitively). When language is a module path and when
requires is provided, the indicated modules are implicitly
included in the allow list.
Each input-program or module-decl argument provides
a program in one of the following forms:
an input port used to read the program;
a string or a byte string holding the complete input;
a path that names a file holding the input; or
an S-expression or a syntax object, which is evaluated
as with eval (see also
get-uncovered-expressions).
In the first three cases above, the program is read using
sandbox-reader, with line-counting enabled for sensible error
messages, and with 'program as the source (used for testing
coverage). In the last case, the input is expected to be the complete
program, and is converted to a syntax object (using
'program as the source), unless it already is a syntax
object.
The returned evaluator function accepts additional expressions
(each time it is called) in essentially the same form: a string or
byte string holding a sequence of expressions, a path for a file
holding expressions, an S-expression, or a syntax object. If
the evaluator receives an eof value, it is terminated and
raises errors thereafter. See also kill-evaluator, which
terminates the evaluator without raising an exception.
For make-evaluator, multiple input-programs are
effectively concatenated to form a single program. The way that the
input-programs are evaluated depends on the language
argument:
The language argument can be a module path (i.e., a
datum that matches the grammar for module-path of
require).
In this case, the input-programs are automatically
wrapped in a module, and the resulting evaluator works
within the resulting module’s namespace.
The language argument can be a list starting with
'special, which indicates a built-in language with
special input configuration. The possible values are
'(special r5rs) or a value indicating a teaching
language: '(special beginner), '(special beginner-abbr), '(special intermediate),
'(special intermediate-lambda), or '(special advanced).
In this case, the input-programs are automatically
wrapped in a module, and the resulting evaluator works
within the resulting module’s namespace. In addition, certain
parameters (such as such as read-accept-infix-dot) are
set to customize reading programs from strings and ports.
This option is provided mainly for older test systems. Using
make-module-evaluator with input starting with
#lang is generally better.
Finally, language can be a list whose first element is
'begin.
In this case, a new namespace is created using
sandbox-namespace-specs, which by default creates a
new namespace using make-base-namespace or
make-gui-namespace (depending on gui?).
In the new namespace, language is evaluated as an
expression to further initialize the namespace.
The requires list adds additional imports to the module or
namespace for the input-programs, even in the case that
require is not made available through the language.
The following examples illustrate the difference between an evaluator
that puts the program in a module and one that merely initializes a
top-level namespace:
| > (define base-module-eval | | ; a module cannot have free variables... | | (make-evaluator 'racket/base '(define (f) later))) |
|
program:1:0: expand: unbound identifier in module in: later |
|
|
| > (base-module-eval '(f)) |
5 |
| > (define base-top-eval | | ; non-module code can have free variables: | | (make-evaluator '(begin) '(define (f) later))) |
|
|
| > (base-top-eval '(+ 1 2)) |
3 |
| > (base-top-eval '(define later 5)) |
|
| > (base-top-eval '(f)) |
5 |
The make-module-evaluator function is essentially a
restriction of make-evaluator, where the program must be a
module, and all imports are part of the program. In some cases it is
useful to restrict the program to be a module using a spcific module
in its language position — use the optional lang argument
to specify such a restriction (the default, #f, means no
restriction is enforced).
| (define base-module-eval2 |
| ; equivalent to base-module-eval: |
| (make-module-evaluator '(module m racket/base |
| (define (f) later) |
| (define later 5)))) |
make-module-evaluator can be very convenient for testing
module files: all you need to do is pass in a path value for the file
name, and you get back an evaluator in the module’s context which you
can use with your favorite test facility.
In all cases, the evaluator operates in an isolated and limited
environment:
Note that these limits apply to the creation of the sandbox
environment too —
so, for example, if the memory that is required to
create the sandbox is higher than the limit, then
make-evaluator will fail with a memory limit exception.
The sandboxed evironment is well isolated, and the evaluator function
essentially sends it an expression and waits for a result. This form
of communication makes it impossible to have nested (or concurrent)
calls to a single evaluator. Usually this is not a problem, but in
some cases you can get the evaluator function available inside the
sandboxed code, for example:
An error will be signalled in such cases.
Evaluation can also be instrumented to track coverage information when
sandbox-coverage-enabled is set. Exceptions (both syntax and
run-time) are propagated as usual to the caller of the evaluation
function (i.e., catch it with with-handlers). However, note
that a sandboxed evaluator is convenient for testing, since all
exceptions happen in the same way, so you don’t need special code to
catch syntax errors.
Finally, the fact that a sandboxed evaluator accept syntax objects
makes it usable as the value for current-eval, which means
that you can easily start a sandboxed read-eval-print-loop. For
example, here is a quick implementation of a networked REPL:
Note that in this code it is only the REPL interactions that are going
over the network connection; using I/O operations inside the REPL will
still use the usual sandbox parameters (defaulting to no I/O). In
addition, the code works only from an existing toplevel REPL —
specifically, read-eval-print-loop reads a syntax value and
gives it the lexical context of the current namespace. Here is a
variation that uses the networked ports for user I/O, and works when
used from a module (by using a new namespace):
A predicate and accessor for exceptions that are raised when a sandbox
is terminated. Once a sandbox raises such an exception, it will
continue to raise it on further evaluation attempts.
call-with-limits. The resource field holds a symbol,
either 'time or 'memory.
13.11.1 Customizing Evaluators
The sandboxed evaluators that make-evaluator creates can be
customized via many parameters. Most of the configuration parameters
affect newly created evaluators; changing them has no effect on
already-running evaluators.
The default configuration options are set for a very restricted
sandboxed environment — one that is safe to make publicly available.
Further customizations might be needed in case more privileges are
needed, or if you want tighter restrictions. Another useful approach
for customizing an evaluator is to begin with a relatively
unrestricted configuration and add the desired restrictions. This approach is made
possible by the call-with-trusted-sandbox-configuration
function.
Invokes the thunk in a context where sandbox configuration
parameters are set for minimal restrictions. More specifically, there
are no memory or time limits, and the existing existing inspectors,
security guard, exit handler, and logger are used. (Note that the I/O
ports settings are not included.)
A parameter that determines a thunk to be called for initializing a
new evaluator. The hook is called just before the program is
evaluated in a newly-created evaluator context. It can be used to
setup environment parameters related to reading, writing, evaluation,
and so on. Certain languages ('(special r5rs) and the
teaching languages) have initializations specific to the language; the
hook is used after that initialization, so it can override settings.
A parameter that specifies a function that reads all expressions from
(current-input-port). The function is used to read program
source for an evaluator when a string, byte string, or port is
supplied. The reader function receives a value to be used as input
source (i.e., the first argument to
read-syntax), and it
should return a list of
syntax objects. The default reader
calls
read-syntax, accumulating results in a list until it
receives
eof.
Note that the reader function is usually called as is, but when it is
used to read the program input for make-module-evaluator,
read-accept-lang and read-accept-reader are set to
#t.
A parameter that determines the initial
current-input-port
setting for a newly created evaluator. It defaults to
#f,
which creates an empty port. The following other values are allowed:
a string or byte string, which is converted to a port using
open-input-string or open-input-bytes;
an input port;
the symbol 'pipe, which triggers the creation of a
pipe, where put-input can return the output end of the
pipe or write directly to it;
a thunk, which is called to obtain a port (e.g., using
current-input-port means that the evaluator input is
the same as the calling context’s input).
A parameter that determines the initial
current-output-port
setting for a newly created evaluator. It defaults to
#f,
which creates a port that discrds all data. The following other
values are allowed:
an output port, which is used as-is;
the symbol 'bytes, which causes get-output
to return the complete output as a byte string;
the symbol 'string, which is similar to
'bytes, but makes get-output produce a
string;
the symbol 'pipe, which triggers the creation of a
pipe, where get-output returns the input end of the
pipe;
a thunk, which is called to obtain a port (e.g., using
current-output-port means that the evaluator output is
not diverted).
Like
sandbox-output, but for the initial
current-error-port value. An evaluator’s error output is set
after its output, so using
current-output-port (the parameter
itself, not its value) for this parameter value means that the error
port is the same as the evaluator’s initial output port.
The default is (lambda () (dup-output-port (current-error-port))), which means that the error output of the
generated evaluator goes to the calling context’s error port.
A parameter that controls whether syntactic coverage information is
collected by sandbox evaluators. Use
get-uncovered-expressions to retrieve coverage information.
When both this boolean parameter and
(break-enabled) are true,
breaking while an evaluator is
running propagates the break signal to the sandboxed
context. This makes the sandboxed evaluator break, typically, but
beware that sandboxed evaluation can capture and avoid the breaks (so
if safe execution of code is your goal, make sure you use it with a
time limit). Also, beware that a break may be received after the
evaluator’s result, in which case the evaluation result is lost. Finally,
beware that a break may be propagated after an evaluator has produced
a result, so that the break is visible on the next interaction with
the evaluator (or the break is lost if the evaluator is not used
further). The default is
#t.
A parameter that holds a list of values that specify how to create a
namespace for evaluation in
make-evaluator or
make-module-evaluator. The first item in the list is a thunk
that creates the namespace, and the rest are module paths for modules
to be attached to the created namespace using
namespace-attach-module.
The default is (list make-base-namespace) if gui? is
#f, (list make-gui-namespace) if gui? is
#t.
The module paths are needed for sharing module instantiations between
the sandbox and the caller. For example, sandbox code that returns
posn values (from the lang/posn module) will
not be recognized as such by your own code by default, since the
sandbox will have its own instance of lang/posn and
thus its own struct type for posns. To be able to use such
values, include 'lang/posn in the list of module paths.
When testing code that uses a teaching language, the following piece
of code can be helpful:
A parameter that determines a list of collection directories to prefix
current-library-collection-paths in an evaluator. This
parameter is useful for cases when you want to test code using an
alternate, test-friendly version of a collection, for example, testing
code that uses a GUI (like the
htdp/world teachpack) can be
done using a fake library that provides the same interface but no
actual interaction. The default is
null.
A parameter that determines the initial
(current-security-guard) for sandboxed evaluations. It can
be either a security guard, or a function to construct one. The
default is a function that restricts the access of the current
security guard by forbidding all filesystem I/O except for
specifications in
sandbox-path-permissions, and it uses
sandbox-network-guard for network connections.
A parameter that configures the behavior of the default sandbox
security guard by listing paths and access modes that are allowed for
them. The contents of this parameter is a list of specifications,
each is an access mode and a byte-regexp for paths that are granted this
access.
The access mode symbol is one of: 'execute, 'write,
'delete, 'read, or 'exists. These symbols
are in decreasing order: each implies access for the following modes
too (e.g., 'read allows reading or checking for existence).
The path regexp is used to identify paths that are granted access. It
can also be given as a path (or a string or a byte string), which is
(made into a complete path, cleansed, simplified, and then) converted
to a regexp that allows the path and sub-directories; e.g.,
"/foo/bar" applies to "/foo/bar/baz".
An additional mode symbol, 'read-bytecode, is not part of the
linear order of these modes. Specifying this mode is similar to
specifying 'read, but it is not implied by any other mode.
(For example, even if you specify 'write for a certain path,
you need to also specify 'read-bytecode to grant this
permission.) The sandbox usually works in the context of a lower code
inspector (see sandbox-make-code-inspector) which prevents
loading of untrusted bytecode files — the sandbox is set-up to allow
loading bytecode from files that are specified with
'read-bytecode. This specification is given by default to
the Racket collection hierarchy (including user-specific libraries) and
to libraries that are explicitly specified in an #:allow-read
argument. (Note that this applies for loading bytecode files only,
under a lower code inspector it is still impossible to use protected
module bindings (see Code Inspectors).)
The default value is null, but when an evaluator is created, it is
augmented by 'read-bytecode permissions that make it possible
to use collection libraries (including
sandbox-override-collection-paths). See
make-evalautor for more information.
A parameter that specifieds a procedure to be used (as is) by the
default
sandbox-security-guard. The default forbids all
network connection.
A parameter that determines the total memory limit on the sandbox in
megabytes (it can hold a rational or a floating point number). When
this limit is exceeded, the sandbox is terminated. This value is used
when the sandbox is created and the limit cannot be changed
afterwards. It defaults to 30mb. See
sandbox-eval-limits
for per-evaluation limits and a description of how the two limits work
together.
Note that (when memory accounting is enabled) memory is attributed to
the highest custodian that refers to it. This means that if you
inspect a value that sandboxed evaluation returns outside of the
sandbox, your own custodian will be charged for it. To ensure that it
is charged back to the sandbox, you should remove references to such
values when the code is done inspecting it.
This policy has an impact on how the sandbox memory limit interacts
with the per-expression limit specified by
sandbox-eval-limits: values that are reachable from the
sandbox, as well as from the interaction will count against the
sandbox limit. For example, in the last interaction of this code,
| (define e (make-evaluator 'racket/base)) |
| (e '(define a 1)) |
| (e '(for ([i (in-range 20)]) (set! a (cons (make-bytes 500000) a)))) |
the memory blocks are allocated within the interaction limit, but
since they’re chained to the defined variable, they’re also reachable
from the sandbox — so they will count against the sandbox memory
limit but not against the interaction limit (more precisely, no more
than one block counts against the interaction limit).
A parameter that determines the default limits on
each use of
a
make-evaluator function, including the initial evaluation
of the input program. Its value should be a list of two numbers;
where the first is a timeout value in seconds, and the second is a
memory limit in megabytes (note that they don’t have to be integers).
Either one can be
#f for disabling the corresponding limit;
alternately, the parameter can be set to
#f to disable all
per-evaluation limits (useful in case more limit kinds are available
in future versions). The default is
(list 30 20).
Note that these limits apply to the creation of the sandbox
environment too —
even
(make-evaluator 'racket/base) can
fail if the limits are strict enough. For example,
will throw an error instead of creating an evaluator. Therefore, to
avoid surprises you need to catch errors that happen when the sandbox
is created.
When limits are set, call-with-limits (see below) is wrapped
around each use of the evaluator, so consuming too much time or memory
results in an exception. Change the limits of a running evaluator
using set-eval-limits.
A custodian’s limit is checked only after a garbage
collection, except that it may also be checked during
certain large allocations that are individually larger
than the custodian’s limit.
The memory limit that is specified by this parameter applies to each
individual evaluation, but not to the whole sandbox —
that limit is
specified via
sandbox-memory-limit. When the global limit is
exceeded, the sandbox is terminated, but when the per-evaluation limit
is exceeded the
exn:fail:resource exception is raised. For example, say that
you evaluate an expression like
then, assuming sufficiently small limits,
if a global limit is set but no per-evaluation limit, the
sandbox will eventually be terminated and no further
evaluations possible;
if there is a per-evaluation limit, but no global limit, the
evaluation will abort with an error and it can be used again
— specifically, a will still hold a number of
blocks, and you can evaluate the same expression again which
will add more blocks to it;
if both limits are set, with the global one larger than the
per-evaluation limit, then the evaluation will abort and you
will be able to repeat it, but doing so several times will
eventually terminate the sandbox (this will be indicated by
the error message, and by the evaluator-alive?
predicate).
A parameter that determines two (optional) handlers that wrap
sandboxed evaluations. The first one is used when evaluating the
initial program when the sandbox is being set-up, and the second is
used for each interaction. Each of these handlers should expect a
thunk as an argument, and they should execute these thunks —
possibly imposing further restrictions. The default values are
#f and
call-with-custodian-shutdown, meaning no
additional restrictions on initial sandbox code (e.g., it can start
background threads), and a custodian-shutdown around each interaction
that follows. Another useful function for this is
call-with-killing-threads which kills all threads, but leaves
other resources intact.
A parameter that determines the (nullary) procedure that is used to
create the inspector for sandboxed evaluation. The procedure is called
when initializing an evaluator. The default parameter value is
(lambda () (make-inspector (current-inspector))).
A parameter that determines the (nullary) procedure that is used to
create the code inspector for sandboxed evaluation. The procedure is
called when initializing an evaluator. The default parameter value is
(lambda () (make-inspector (current-code-inspector))).
The current-load/use-compiled handler is setup to allow loading
of bytecode files under the original code inspector when
sandbox-path-permissions allows it through a
'read-bytecode mode symbol, which makes loading libraries
possible.
A parameter that determines the procedure used to create the logger
for sandboxed evaluation. The procedure is called when initializing
an evaluator, and the default parameter value is
current-logger. This means that it is not creating a new
logger (this might change in the future).
13.11.2 Interacting with Evaluators
The following functions are used to interact with a sandboxed
evaluator in addition to using it to evaluate code.
Determines whether the evaluator is still alive.
Releases the resources that are held by evaluator by shutting
down the evaluator’s custodian. Attempting to use an evaluator after
killing raises an exception, and attempts to kill a dead evaluator are
ignored.
Killing an evaluator is similar to sending an eof value to
the evaluator, except that an eof value will raise an error
immediately.
Sends a break to the running evaluator. The effect of this is as if
Ctrl-C was typed when the evaluator is currently executing, which
propagates the break to the evaluator’s context.
Retrieves the
evaluator’s toplevel custodian. This returns a
value that is different from
(evaluator '(current-custodian))
or
call-in-sandbox-context evaluator current-custodian —
each
sandbox interaction is wrapped in its own custodian, which is what these
would return.
(One use for this custodian is with current-memory-use, where
the per-interaction sub-custodians will not be charged with the memory
for the whole sandbox.)
Changes the per-expression limits that evaluator uses to
sec seconds and mb megabytes (either one can be
#f, indicating no limit).
This procedure should be used to modify an existing evaluator limits,
because changing the sandbox-eval-limits parameter does not
affect existing evaluators. See also call-with-limits.
Changes the per-expression handler that the evaluator uses
around each interaction. A #f value means no handler is
used.
This procedure should be used to modify an existing evaluator handler,
because changing the sandbox-eval-handlers parameter does not
affect existing evaluators. See also
call-with-custodian-shutdown and
call-with-killing-threads for two useful handlers that are
provided.
These functions are useful for use as an evaluation handler.
call-with-custodian-shutdown will execute the
thunk
in a fresh custodian, then shutdown that custodian, making sure that
thunk could not have left behind any resources.
call-with-killing-threads is similar, except that it kills
threads that were left, but leaves other resources as is.
If
(sandbox-input) is
'pipe when an evaluator is
created, then this procedure can be used to retrieve the output port
end of the pipe (when used with no arguments), or to add a string or a
byte string into the pipe. It can also be used with
eof,
which closes the pipe.
if it was 'pipe, then get-output returns the
input port end of the created pipe;
if it was 'bytes or 'string, then the result
is the accumulated output, and the output port is reset so each
call returns a different piece of the evaluator’s output (note
that any allocations of such output are still subject to the
sandbox memory limit);
otherwise, it returns #f.
Retrieves uncovered expression from an evaluator, as longs as the
sandbox-coverage-enabled parameter had a true value when the
evaluator was created. Otherwise, an exception is raised to indicate
that no coverage information is available.
The prog? argument specifies whether to obtain expressions that
were uncovered after only the original input program was evaluated
(#t) or after all later uses of the evaluator (#f).
Using #t retrieves a list that is saved after the input
program is evaluated, and before the evaluator is used, so the result is
always the same.
A #t value of prog? is useful for testing student
programs to find out whether a submission has sufficient test coverage
built in. A #f value is useful for writing test suites for a
program to ensure that your tests cover the whole code.
The second optional argument, src, specifies that the result
should be filtered to hold only syntax objects whose source
matches src. The default is the source that was used in the
program code, if there was one. Note that 'program is used as
the source value if the input program was given as S-expressions or as a
string (and in these cases it will be the default for filtering). If given
#f, the result is the unfiltered list of expressions.
The resulting list of syntax objects has at most one expression
for each position and span. Thus, the contents may be unreliable, but
the position information is reliable (i.e., it always indicates source
code that would be painted red in DrRacket when coverage information
is used).
Note that if the input program is a sequence of syntax values, either
make sure that they have 'program as the source field, or use
the src argument. Using a sequence of S-expressions (not
syntax objects) for an input program leads to unreliable
coverage results, since each expression may be assigned a single
source location.
Calls the given thunk in the context of a sandboxed
evaluator. The call is performed under the resource limits and
evaluation handler that are used for evaluating expressions, unless
unrestricted? is specified as true.
This process is usually similar to
(evaluator (list thunk)), except
that it relies on the common meaning of list expressions as function
application (which is not true in all languages), and it relies on
eval allowing non-S-expression input. In
addition, you can avoid some of the sandboxed restrictions by using
your own permissions, for example,
13.11.3 Miscellaneous
Various aspects of the racket/sandbox library change
when the GUI library is available, such as using a new eventspace for
each evaluator.
Executes the given
thunk with memory and time restrictions:
if execution consumes more than
mb megabytes or more than
sec seconds, then the computation is aborted and the
exn:fail:resource exception is raised. Otherwise the result of the thunk is
returned as usual (a value, multiple values, or an exception). Each
of the two limits can be
#f to indicate the absence of a
limit. See also
custodian-limit-memory for information on
memory limits.
Sandboxed evaluators use call-with-limits, according to the
sandbox-eval-limits setting and uses of
set-eval-limits: each expression evaluation is protected from
timeouts and memory problems. Use call-with-limits directly
only to limit a whole testing session, instead of each expression.
A predicate and accessor for exceptions that are raised by
call-with-limits. The
resource field holds a symbol,
either
'time or
'memory.