Syntax Color:   Utilities
1 Parenthesis Matching
paren-tree%
2 Lexer Contract & the Don’t Stop struct
lexer/  c
dont-stop
3 Racket Lexer
racket-lexer
racket-lexer/  status
racket-nobar-lexer/  status
4 Default lexer
default-lexer
5 Module Lexer
module-lexer
6 Scribble Lexer
scribble-lexer
scribble-inside-lexer
7 Splay Tree for Tokenization
token-tree%
new
get-root
search!
node?
node-token-length
node-token-data
node-left-subtree-length
node-left
node-right
insert-first!
insert-last!
insert-last-spec!
6.1

Syntax Color: Utilities

Scott Owens

The "syntax-color" collection provides the underlying data structures and some helpful utilities for the color:text<%> class of framework.

1 Parenthesis Matching

 (require syntax-color/paren-tree)
  package: syntax-color-lib

class

paren-tree% : class?

  superclass: object%

Parenthesis matching code built on top of token-tree%.

2 Lexer Contract & the Don’t Stop struct

 (require syntax-color/lexer-contract)
  package: syntax-color-lib

value

lexer/c : contract?

Checks to be sure a lexing function is well-behaved. For more details, see start-colorer in color:text<%>.

struct

(struct dont-stop (val)
    #:extra-constructor-name make-dont-stop)
  val : any/c
A struct used to indicate to the lexer that it should not allow itself to be interrupted. For more details, see start-colorer in color:text<%>.

3 Racket Lexer

 (require syntax-color/racket-lexer)
  package: syntax-color-lib

procedure

(racket-lexer in)  
(or/c string? eof-object?)
symbol?
(or/c symbol? #f)
(or/c number? #f)
(or/c number? #f)
  in : input-port?
A lexer for Racket, including reader extensions (Reader Extension), built specifically for color:text<%>.

The racket-lexer function returns 5 values:

procedure

(racket-lexer/status in)  
(or/c string? eof-object?)
symbol?
(or/c symbol? #f)
(or/c number? #f)
(or/c number? #f)
(or/c 'datum 'open 'close 'continue)
  in : input-port?
Like racket-lexer, but returns an extra value. The last return value indicates whether the consumed token should count as a datum, an opening parenthesis (or similar starting token to group other tokens), a closing parenthesis (or similar), or a prefix (such as whitespace) on a datum.

procedure

(racket-nobar-lexer/status in)

  
(or/c string? eof-object?)
symbol?
(or/c symbol? #f)
(or/c number? #f)
(or/c number? #f)
(or/c 'datum 'open 'close 'continue)
  in : input-port?
Like racket-lexer/status, except it treats | as a delimiter instead of quoting syntax for a symbol. This function is used by scribble-lexer.

4 Default lexer

 (require syntax-color/default-lexer)
  package: syntax-color-lib

procedure

(default-lexer in)  
(or/c string? eof-object?)
symbol?
(or/c symbol? #f)
(or/c number? #f)
(or/c number? #f)
  in : input-port?

A lexer that only identifies (, ), [, ], {, and } built specifically for color:text<%>.

default-lexer returns 5 values:

5 Module Lexer

 (require syntax-color/module-lexer)
  package: syntax-color-lib

procedure

(module-lexer in offset mode)

  
(or/c string? eof-object?)
symbol?
(or/c symbol? #f)
(or/c number? #f)
(or/c number? #f)
exact-nonnegative-integer?
(or/c #f
      (-> input-port? any)
      (cons/c (-> input-port? any/c any) any/c))
  in : input-port?
  offset : exact-nonnegative-integer?
  mode : 
(or/c #f
      (-> input-port? any)
      (cons/c (-> input-port? any/c any) any/c))
Like racket-lexer, but with several differences:

6 Scribble Lexer

 (require syntax-color/scribble-lexer)
  package: syntax-color-lib

procedure

(scribble-lexer in offset mode)  
(or/c string? eof-object?)
symbol?
(or/c symbol? #f)
(or/c number? #f)
(or/c number? #f)
exact-nonnegative-integer?
any/c
  in : input-port?
  offset : exact-nonnegative-integer?
  mode : any/c
Like racket-lexer, but for Racket extended with Scribble’s @ notation (see @ Syntax).

procedure

(scribble-inside-lexer in offset mode)

  
(or/c string? eof-object?)
symbol?
(or/c symbol? #f)
(or/c number? #f)
(or/c number? #f)
exact-nonnegative-integer?
any/c
  in : input-port?
  offset : exact-nonnegative-integer?
  mode : any/c
Like scribble-lexer, but starting in “text” mode instead of Racket mode.

7 Splay Tree for Tokenization

 (require syntax-color/token-tree)
  package: syntax-color-lib

class

token-tree% : class?

  superclass: object%

A splay-tree class specifically geared for the task of on-the-fly tokenization. Instead of keying nodes on values, each node has a length, and they are found by finding a node that follows a certain total length of preceding nodes.
FIXME: many methods are not yet documented.

constructor

(new token-tree% [len len] [data data])

  (is-a?/c token-tree%)
  len : (or/c exact-nonnegative-integer? fasle/c)
  data : any/c
Creates a token tree with a single element.

method

(send a-token-tree get-root)  (or/c node? #f)

Returns the root node in the tree.

method

(send a-token-tree search! key-position)  void?

  key-position : natural-number/c
Splays, setting the root node to be the closest node to offset key-position (i.e., making the total length of the left tree at least key-position, if possible).

procedure

(node? v)  boolean?

  v : any/c

procedure

(node-token-length n)  natural-number/c

  n : node?

procedure

(node-token-data n)  any/c

  n : node?

procedure

(node-left-subtree-length n)  natural-number/c

  n : node?

procedure

(node-left n)  (or/c node? #f)

  n : node?

procedure

(node-right n)  (or/c node? #f)

  n : node?
Functions for working with nodes in a token-tree%.

procedure

(insert-first! tree1 tree2)  void?

  tree1 : (is-a?/c token-tree%)
  tree2 : (is-a?/c token-tree%)
Inserts tree1 into tree2 as the first thing, setting tree2’s root to #f.

procedure

(insert-last! tree1 tree2)  void?

  tree1 : (is-a?/c token-tree%)
  tree2 : (is-a?/c token-tree%)
Inserts tree1 into tree2 as the last thing, setting tree2’s root to #f.

procedure

(insert-last-spec! tree n v)  void?

  tree : (is-a?/c token-tree%)
  n : natural-number/c
  v : any/c
Same as
(insert-last! tree
              (new token-tree%
                   [length n]
                   [data v]))

This optimization is important for the colorer.