3.18 Iterations and Comprehensions: for, for/list, ...

Iterations and Comprehensions in The Racket Guide introduces iterations and comprehensions.

The for iteration forms are based on SRFI-42 [SRFI-42].

3.18.1 Iteration and Comprehension Forms

In the simple case, each for-clause has one of its first two forms, where [id seq-expr] is a shorthand for [(id) seq-expr]. In this simple case, the seq-exprs are evaluated left-to-right, and each must produce a sequence value (see Sequences).

The for form iterates by drawing an element from each sequence; if any sequence is empty, then the iteration stops, and #<void> is the result of the for expression. Otherwise a location is created for each id to hold the values of each element; the sequence produced by a seq-expr must return as many values for each iteration as corresponding ids.

The ids are then bound in the body, which is evaluated, and whose results are ignored. Iteration continues with the next element in each sequence and with fresh locations for each id.

A for form with zero for-clauses is equivalent to a single for-clause that binds an unreferenced id to a sequence containing a single element. All of the ids must be distinct according to bound-identifier=?.

If any for-clause has the form #:when guard-expr, then only the preceding clauses (containing no #:when or #:unless) determine iteration as above, and the body is effectively wrapped as

(when guard-expr

(for (for-clause ...) body ...+))

using the remaining for-clauses. A for-clause of the form #:unless guard-expr corresponds to the same transformation with unless in place of when.

A #:break guard-expr clause is similar to a #:unless guard-expr clause, but when #:break avoids evaluation of the bodys, it also effectively ends all sequences within the for form. A #:final guard-expr clause is similar to #:break guard-expr, but instead of immediately ending sequences and skipping the bodys, it allows at most one more element from each later sequence and at most one more evaluation of the following bodys. Among the bodys, besides stopping the iteration and preventing later body evaluations, a #:break guard-expr or #:final guard-expr clause starts a new internal-definition context.

In the case of list and stream sequences, the for form itself does not keep each element reachable. If a list or stream produced by a seq-expr is otherwise unreachable, and if the for body can no longer reference an id for a list element, then the element is subject to garbage collection. The make-do-sequence sequence constructor supports additional sequences that behave like lists and streams in this way.

If a seq-expr is a quoted literal list, vector, exact integer, string, byte string, immutable hash, or expands to such a literal, then it may be treated as if a sequence transformer such as in-list was used, unless the seq-expr has a true value for the 'for:no-implicit-optimization syntax property; in most cases this improves performance.

Changed in version 6.7.0.4 of package base: Added support for the optional second result.
Changed in version 7.8.0.11: Added support for implicit optimization.

If the optional #:length clause is specified, the result of length-expr determines the length of the result vector. In that case, the iteration can be performed more efficiently, and it terminates when the vector is full or the requested number of iterations have been performed, whichever comes first. If length-expr specifies a length longer than the number of iterations, then the remaining slots of the vector are initialized to the value of fill-expr, which defaults to 0 (i.e., the default argument of make-vector).

The for/vector form may allocate a vector and mutate it after each iteration of body, which means that capturing a continuation during body and applying it multiple times may mutate a shared vector.

If a result-expr is provided, it is used as with for/fold when iteration terminates; otherwise, the result is as many lists as supplied ids.

The scope of id bindings is the same as for accumulator identifiers in for/fold. Mutating a id affects the accumulated lists, and mutating it in a way that produces a non-list can cause a final reverse for each id to fail.

Changed in version 7.1.0.2 of package base: Added the #:result form.

The binding and evaluation order of accum-ids and init-exprs do not completely follow the textual, left-to-right order relative to the for-clauses. Instead, the sequence expressions in for-clauses that determine the outermost iteration are evaluated first, then the init-exprs are evaluated and the accum-ids are bound, and finally the outermost iteration’s identifiers are bound. One consequence is that the accum-ids are not bound in for-clauses for the outermost initialization. At the same time, when a accum-id is used as a for-clause binding for the outermost iteration, the for-clause binding shadows the accum-id binding in the loop body (which is what you would expect syntactically). A fresh variable for each accum-id (at a fresh location) is bound in each nested iteration that is created by a later group for for-clauses (after a #:when or #:unless, for example).

Changed in version 6.11.0.1 of package base: Added the #:result form.

Furthermore, unlike for/fold, the accum-ids are not bound within guard-exprs or body-or-break forms that appear before a break-clause.

While the aforementioned limitations make for/foldr less generally useful than for/fold, for/foldr provides the additional capability to iterate lazily via the #:delay, #:delay-as, and #:delay-with options, which can mitigate many of for/foldr’s disadvantages. If at least one such option is specified, the loop body is given explicit control over when iteration continues: by default, each accum-id is bound to a promise that, when forced, produces the accum-id’s current value.

In this mode, iteration does not continue until one such promise is forced, which triggers any additional iteration necessary to produce a value. If the loop body is lazy in its accum-ids—that is, it returns a value without forcing any of them—then the loop (or any of its iterations) will produce a value before iteration has completely finished. If a reference to at least one such promise is retained, then forcing it will resume iteration from the point at which it was suspended, even if control has left the dynamic extent of the loop body.

This extra control over iteration order allows for/foldr to both consume and construct infinite sequences, so long as it is at least sometimes lazy in its accumulators.

See also for/stream for a more convenient (albeit less flexible) way to lazily transform infinite sequences. (Internally, for/stream is defined in terms of for/foldr.)

The suspension introduced by the #:delay option does not ordinarily affect the loop’s eventual return value, but if #:delay and #:result are combined, the accum-ids will be delayed in the scope of the result-expr in the same way they are delayed within the loop body. This can be used to introduce an additional layer of suspension around the evaluation of the entire loop, if desired.

If the #:delay-as option is provided, then delayed-id is bound to an additional promise that returns the values of all accum-ids at once. When multiple accum-ids are provided, forcing this promise can be slightly more efficient than forcing the promises bound to the accum-ids individually.

If the #:delay-with option is provided, the given delayer-id is used to suspend nested iterations (instead of the default, delay). A form of the shape (delayer-id recur-expr) is constructed and placed in expression position, where recur-expr is an expression that, when evaluated, will perform the next iteration and return its result (or results). Sensible choices for delayer-id include lazy, delay/sync, delay/thread, or any of the other promise constructors from racket/promise, as well as thunk from racket/function. However, beware that choices such as thunk or delay/name may evaluate their subexpression multiple times, which can lead to nonsensical results for sequences that have state, as the state will be shared between all evaluations of the recur-expr.

If multiple accum-ids are given, the #:delay-with option is provided, and delayer-id is not bound to one of delay, lazy, delay/strict, delay/sync, delay/thread, or delay/idle, the accum-ids will not be bound at all, even within the loop body. Instead, the #:delay-as option must be specified to access the accumulator values via delayed-id.

Added in version 7.3.0.3 of package base.

Changed in version 7.3.0.3 of package base: Added the for*/foldr form.

3.18.2 Deriving New Iteration Forms

A macro that expands to for/fold/derived should typically use split-for-body to handle the possibility of macros and other definitions mixed with keywords like #:break.

Changed in version 6.11.0.1 of package base: Added the #:result form.

Changed in version 6.11.0.1 of package base: Added the #:result form.

Added in version 7.3.0.3 of package base.

When id is used in any other expression position, the result of expr-transform-expr is used. If it is a procedure of zero arguments, then the result must be an identifier other-id, and any use of id is converted to a use of other-id. Otherwise, expr-transform-expr must produce a procedure (of one argument) that is used as a macro transformer.

When the clause-transform-expr transformer is used, it is given a for-clause as an argument, where the clause’s form is normalized so that the left-hand side is a parenthesized sequence of identifiers. The right-hand side is of the form (id . rest). The result can be either #f, to indicate that the forms should not be treated specially (perhaps because the number of bound identifiers is inconsistent with the (id . rest) form), or a new for-clause to replace the given one. The new clause might use :do-in. To protect identifiers in the result of clause-transform-expr, use for-clause-syntax-protect instead of syntax-protect.

Within a for, the pieces of the :do-in form are spliced into the iteration essentially as follows:

(let-values ([(outer-id ...) outer-expr] ...)

outer-check

(let loop ([loop-id loop-expr] ...)

(if pos-guard

(let-values ([(inner-id ...) inner-expr] ...)

(if pre-guard

(let body-bindings

(if post-guard

(loop loop-arg ...)

done-expr))

done-expr))

done-expr)))

where body-bindings and done-expr are from the context of the :do-in use. The identifiers bound by the for clause are typically part of the ([(inner-id ...) inner-expr] ...) section.

Beware that body-bindings and done-expr can contain arbitrary expressions, potentially including set! on outer-id or inner-id identifiers if they are visible in the original for form, so beware of depending on such identifiers in post-guard and loop-arg.

The actual loop binding and call has additional loop arguments to support iterations in parallel with the :do-in form, and the other pieces are similarly accompanied by pieces from parallel iterations.

For an example of :do-in, see define-sequence-syntax.

Changed in version 8.2.0.4 of package base: Changed to just return stx instead of returning “armed” syntax.

3.18.3 Do Loops

To initialize the loop, the init-exprs are evaluated in order and bound to the corresponding ids. The ids are bound in all expressions within the form other than the init-exprs.

After the ids have been bound, the stop?-expr is evaluated. If it produces #f, each expr is evaluated for its side-effect. The ids are then effectively updated with the values of the step-exprs, where the default step-expr for id is just id; more precisely, iteration continues with fresh locations for the ids that are initialized with the values of the corresponding step-exprs.

When stop?-expr produces a true value, then the finish-exprs are evaluated in order, and the last one is evaluated in tail position to produce the overall value for the do form. If no finish-expr is provided, the value of the do form is #<void>.