method (send a-color:text start-colorer   token-sym->style             get-token             pairs)   →   void?

token-sym->style : (-> symbol? string?)

get-token   :   (or/c (-> input-port?           (values any/c                   (or/c symbol?                         (hash/c symbol? any/c #:immutable #t))                   (or/c symbol? #f)                   (or/c exact-positive-integer? #f)                   (or/c exact-positive-integer? #f)))       (-> input-port?           exact-nonnegative-integer?           (not/c dont-stop?)           (values any/c                   (or/c symbol?                         (hash/c symbol? any/c #:immutable #t))                   (or/c symbol? #f)                   (or/c exact-positive-integer? #f)                   (or/c exact-positive-integer? #f)                   exact-nonnegative-integer?                   any/c)))

pairs : (listof (list/c symbol? symbol?))

Starts tokenizing the buffer for coloring and parenthesis matching.

The main argument is get-token. It accepts either three arguments or only the first of these three:

• input-port — An input port to parse from. The port is not necessarily the same for every call to get-token.

• offset — An integer that can be added to the position of input-port to obtain an absolute coordinate within a text stream.

• mode — An arbitrary value that is #f when input-port represents the start of the input stream, and otherwise is the last result of get-token as returned for the just-preceding token.

  The mode value is intended to record the state of parsing in a way that allows it to be restarted mid-stream. The mode value should not be a mutable value; if part of the input stream is re-tokenized, the mode saved from the immediately preceding token is given again to the get-token function.

The get-token function produces either 7 results or the first 5 of these results, depending on how many arguments get-token accepts:

• token — A value intended to represent the textual component of the token. This value is ignored by start-colorer.

• attribs — Either a symbol or a hash table with symbol keys. Except for 'eof, a symbol by itself is treated the same as a hash table that maps 'type to the symbol. A get-token that accepts only a single argument must always produce just a symbol for attribs.

  The symbol 'eof (not a hash table) must be returned as attribs to indicate when all the tokens have been consumed.

  The value of 'color in attribs is passed to token-sym->style, which returns a style name that that is used to “color” the token. If 'color is not mapped by attribs, then the value of 'type is used, instead. In addition, if 'comment? is mapped to a true value, then the token’s color is adjusted to de-emphasize it relative to surrounding text.

  Certain values for 'type in attribs are treated specially. The symbols 'white-space and 'comment should always be used for whitespace and comment tokens, respectively. The symbol 'no-color can be used to indicate that although the token is not whitespace, it should not be colored.

  These and other keys in attribs can be used by tools that call classify-position*.

• paren — A symbol indicating how the token should be treated by the parenthesis matcher, or #f if the token does not correspond to an open or close parentheses. A parens symbol should be one of the symbols in the pairs argument.

  Parenthesis matching uses this symbol in combination with parens to determine matching pairs and to enable navigation options that take matches into account.

  For example, suppose pairs is '((|(| |)|) (|[| |]|) (begin end)). This means that there are three kinds of parentheses. Any token that has 'begin as its paren value will act as an open for matching tokens that have 'end as paren. Similarly, any token with '|]| will act as a closing match for tokens with '|[|. When trying to correct a mismatched closing parenthesis, each closing symbol in pairs will be converted to a string and tried as a closing parenthesis.

• start — The starting position of the token (or #f for an end-of-file). This number is relative to the third result of (port-next-location input-port).

• _end — The ending position of the token (or #f for an end-of-file). This is number is also relative to the port’s location, like start.

• backup — A backup distance, which indicates the maximum number of characters to back up (counting from the start of the token) and for re-parsing after a change to the editor within the token’s region. A backup is typically 0.

• mode (the new one) — A value that is passed to a later call to get-token to continue parsing the input program.

  If mode is a dont-stop structure, then the value inside the structure is considered the new mode, and the colorer is guaranteed not to be interrupted until at least the next call to get-token that does not return a dont-stop structure (unless, of course, it returns an 'eof value for attribs, in which case the new mode result is ignored). A dont-stop result is useful, for example, when a lexer has to read ahead in input-port to decide on the tokens at this point; that read-ahead will be inconsistent if an edit happens, so a dont-stop structure ensures that no changes to the buffer happen between calls.

  As mentioned above, the mode result should not be a mutable value. Also, it should be comparable with equal? to short-circuit reparsing when get-token returns the same results for an input position.

The token-sym->style and parens arguments are used as described above with the attribs and paren results, respectively.

The get-token argument’s contract above reflects just the basic constraints it should satisfy. It is also expected to satisfy the lexer*/c contract, which attempts to check the following additional invariants:

• Every position in the buffer must be accounted for in exactly one token, and every token must have a non-zero width. Accordingly, get-token must never refuse to accept certain token streams (e.g., by raising an exception). The idea is that, while a normal parser for the language could signal errors by helpfully raising exceptions, a colorer should instead return a token with the type 'error and possibly continue to color the remainder of the buffer. For example, the racket-lexer identifiers strings that have malformed escape characters inside strings by returning 'error, but then continuing to color the rest of text as normal.

• The token returned by get-token must rely only on the contents of the input port argument plus the mode argument. This constraint means that the tokenization of some part of the input cannot depend on earlier parts of the input except through the mode (and implicitly through the starting positions for tokens).

• A change to the stream must not change the tokenization of the stream prior to the token immediately preceding the change plus the backup distance. In the following example, this invariant does not hold for a zero backup distance: If the buffer contains

  " 1 2 3

  and the tokenizer treats the unmatched " as its own token (a string error token), and separately tokenizes the 1 2 and 3, an edit to make the buffer look like

  " 1 2 3"

  would result in a single string token modifying previous tokens. To handle these situations, get-token can treat the first line as a single token, or it can precisely track backup distances.

The get-token function is usually be implemented with a lexer using the parser-tools/lex library, but can be implemented directly. For example, here is a lexer that colors alternating characters as if they were symbols and strings:

(λ (port offset mode)

(define-values (line col pos) (port-next-location port))

(define c (read-char port))

(cond

[(eof-object? c)

(values c 'eof #f #f #f 0 mode)]

[else

(values (string c)

(if mode 'symbol 'string)

#f

(+ pos)

(+ pos 1)

0

(not mode))]))

Changed in version 1.63 of package gui-lib: Added support for hash-table attribs results.

method (send a-color:text stop-colorer [clear-colors?]) → void?

clear-colors? : boolean? = #t

Stops coloring and paren matching the buffer.

If clear-colors? is true all the text in the buffer will have its style set to Standard.

method (send a-color:text force-stop-colorer stop?) → void?

stop? : boolean?

Causes the entire tokenizing/coloring system to become inactive. Intended for debugging purposes only.

stop? determines whether the system is being forced to stop or allowed to wake back up.

method (send a-color:text is-stopped?) → boolean?

Indicates if the colorer for this editor has been stopped, or not.

method (send a-color:text is-frozen?) → boolean?

Indicates if this editor’s colorer is frozen. See also freeze-colorer and thaw-colorer.

method (send a-color:text freeze-colorer) → void?

Keep the text tokenized and paren matched, but stop altering the colors.

freeze-colorer will not return until the coloring/tokenization of the entire text is brought up-to-date. It must not be called on a locked text.

method (send a-color:text thaw-colorer  [ recolor?             retokenize?])   →   void?

recolor? : boolean? = #t

retokenize? : boolean? = #f

Start coloring a frozen buffer again.

If recolor? is #t, the text is re-colored. If it is #f the text is not recolored. When recolor? is #t, retokenize? controls how the text is recolored. #f causes the text to be entirely re-colored before thaw-colorer returns using the existing tokenization. #t causes the entire text to be retokenized and recolored from scratch. This will happen in the background after the call to thaw-colorer returns.

method (send a-color:text reset-region start end) → void?

start : exact-nonnegative-integer?

end : (or/c exact-nonnegative-integer? 'end)

Set the region of the text that is tokenized.

method (send a-color:text reset-regions regions) → void?

regions   :   (listof (list/c exact-nonnegative-integer?                 (or/c exact-nonnegative-integer? 'end)))

Sets the currently active regions to be regions. The numbers in the regions argument must be increasing and only the last number can be replaced with 'end.

Note that editing outside of the active regions violates (unchecked) invariants of this class and edits that move text across region boundaries may also violate (unchecked) invariants. DrRacket uses this method in the interactions window in a way that disallows edits anywhere except the last region and the last region has 'end as its second argument.

method (send a-color:text get-spell-check-strings) → boolean?

Returns #t if the colorer will attempt to spell-check string constants.

method (send a-color:text set-spell-check-strings b?) → void?

b? : boolean?

If called with #t, tell the colorer to spell-check string constants. Otherwise, disable spell-checking of string constants.

method (send a-color:text get-spell-check-text) → boolean?

Returns #t if the colorer will attempt to spell-check text (e.g., the words inside { and } in Scribble documents).

method (send a-color:text set-spell-check-text b?) → void?

b? : boolean?

If called with #t, tell the colorer to spell-check text constants. Otherwise, disable spell-checking of text.

method (send a-color:text set-spell-current-dict dict) → void?

dict : (or/c string? #f)

Sets the current dictionary used with aspell to dict. If dict is #f, then the default dictionary is used.

method (send a-color:text get-spell-current-dict) → (or/c string? #f)

Get the current dictionary used with aspell. If the result is #f, then the default dictionary is used.

method (send a-color:text get-spell-suggestions pos)

→   (or/c #f (list/c exact-nonnegative-integer?                  exact-nonnegative-integer?                  (listof string?)))

pos : exact-nonnegative-integer?

Returns suggested spelling corrections (and the span of the entire word) to replace the word at pos. If the word is spelled correctly or spell checking is disabled, returns #f.

method (send a-color:text get-regions)

→ (listof (list/c exact-nonnegative-integer? (or/c exact-nonnegative-integer? 'end)))

This returns the list of regions that are currently being colored in the editor.

method (send a-color:text get-matching-paren-string   paren-str      [ get-side])

→ (or/c string? #f)

paren-str : string?

get-side : (or/c 'open 'close 'either) = 'either

Returns a string of a paren matching the other side of paren-str as specified by the pairs argument of start-colorer, if one exists on the side indicated by get-side. If there is no match on the corresponding side, including if paren-str contains any characters other than a single paren token (even whitespace), returns #f instead.

method (send a-color:text skip-whitespace   position       direction       comments?)

→ exact-nonnegative-integer?

position : exact-nonnegative-integer?

direction : (or/c 'forward 'backward)

comments? : boolean?

Returns the next non-whitespace character.

Starts from position and skips whitespace in the direction indicated by direction. If comments? is true, comments are skipped as well as whitespace. skip-whitespace determines whitespaces and comments by comparing the token type to 'white-space and 'comment.

Must only be called while the tokenizer is started.

method (send a-color:text backward-match   position       cutoff)

→ (or/c exact-nonnegative-integer? #f)

position : exact-nonnegative-integer?

cutoff : exact-nonnegative-integer?

Skip all consecutive whitespaces and comments (using skip-whitespace) immediately preceding the position. If the token at this position is a close, return the position of the matching open, or #f if there is none. If the token was an open, return #f. For any other token, return the start of that token.

Must only be called while the tokenizer is started.

method (send a-color:text backward-containing-sexp   position       cutoff)

→ (or/c exact-nonnegative-integer? #f)

position : exact-nonnegative-integer?

cutoff : exact-nonnegative-integer?

Return the starting position of the interior of the (non-atomic) s-expression containing position, or #f is there is none.

Must only be called while the tokenizer is started.

method (send a-color:text forward-match   position       cutoff)

→ (or/c exact-nonnegative-integer? #f)

position : exact-nonnegative-integer?

cutoff : exact-nonnegative-integer?

Skip all consecutive whitespaces and comments (using skip-whitespace) immediately following position. If the token at this position is an open, return the position of the matching close, or #f if there is none. For any other token, return the end of that token.

Must only be called while the tokenizer is started.

method (send a-color:text insert-close-paren   position             char             flash?             fixup?            [ smart-skip?])   →   void?

position : exact-nonnegative-integer?

char : char?

flash? : boolean?

fixup? : boolean?

smart-skip? : (or/c #f 'adjacent 'forward) = #f

Inserts a close parentheses, or, under scenarios described further below, skips past a subsequent one. The position is the place to put the parenthesis, or from which to start searching for a subsequent one, and char is the parenthesis to be added (e.g., that the user typed). If fixup? is true, the right kind of closing parenthesis will be chosen from the set previously passed to start-colorer—but only if an inserted char would be colored as a parenthesis (i.e., with the 'parenthesis classification). Otherwise, char will be inserted (or skipped past), even if it is not the right kind. If flash? is true, the matching open parenthesis will be flashed when the insertion or skip is done.

The "smart skipping" behavior of this function is determined by smart-skip?. If smart-skip? is false, no skip will take place. A parenthesis will simply be inserted as described in the paragraph above. When smart-skip? is 'adjacent, if the next token after position, ignoring whitespace and comments (see skip-whitespace), is a properly matched closing parenthesis (which may not necessarily match char if fixup? is true) then simply move the cursor to the position immediately after that already present closing parenthesis. When smart-skip? is 'forward, this function attempts to determine the closest pair of properly balanced parentheses around position. If that exists, then the cursor position skips to the position immediately after the closing parenthesis of that outer pair. If a properly balanced outer pair is not present, then the cursor attempts to skip immediately after the next closing parenthesis that occurs after position, ignoring whitespace, comments, and all other tokens. In both non-false cases of smart-skip?, if there is no subsequent parenthesis, then a parenthesis is simply inserted, as previously described.

method (send a-color:text classify-position position)

→ (or/c symbol? #f)

position : exact-nonnegative-integer?

Return a symbol for the lexer-determined token type for the token that contains the item after position. Using classify-position is the same as using classify-position and checking for a 'type value.

Must only be called while the tokenizer is started.

method (send a-color:text classify-position* position)

→ (or/c (and/c (hash/c symbol? any/c) immutable?) #f)

position : exact-nonnegative-integer?

Return a hash table for the lexer-determined token attributes for the token that contains the item after position. The result is #f if no attributes are available for the position.

Must only be called while the tokenizer is started.

Added in version 1.63 of package gui-lib.

method (send a-color:text get-token-range position)

→   (or/c #f exact-nonnegative-integer?) (or/c #f exact-nonnegative-integer?)

position : exact-nonnegative-integer?

Returns the range of the token surrounding position, if there is a token there.

This method must be called only when the tokenizer is started.

method (send a-color:text get-backward-navigation-limit start)

→ exact-integer?

start : exact-integer?

Returns a limit for backward-matching parenthesis starting at position start.

Added in version 1.65 of package gui-lib.

method (send a-color:text on-lexer-valid valid?) → any

valid? : boolean?

Refine this method with augment.

This method is an observer for when the lexer is working. It is called when the lexer’s state changes from valid to invalid (and back). The valid? argument indicates if the lexer has finished running over the editor (or not).

The default method just returns (void?).

method (send a-color:text is-lexer-valid?) → boolean?

This method is final, so it cannot be overridden.

Indicates if the lexer is currently valid for this editor.

method (send a-color:text lock) → void?

Overrides lock in editor<%>.

method (send a-color:text on-focus) → void?

Overrides on-focus in editor<%>.

method (send a-color:text after-edit-sequence) → void?

Augments after-edit-sequence in editor<%>.

method (send a-color:text after-set-position) → void?

Augments after-set-position in text%.

method (send a-color:text after-change-style) → void?

Augments after-change-style in text%.

method (send a-color:text on-set-size-constraint) → void?

Augments on-set-size-constraint in text%.

method (send a-color:text after-insert) → void?

Augments after-insert in text%.

method (send a-color:text after-delete) → void?

Augments after-delete in text%.

method (send a-color:text-mode set-get-token get-token) → void?

get-token : procedure?

Sets the get-token function used to color the contents of the editor.

See start-colorer’s get-token argument for the contract on this method’s get-token argument.

method (send a-color:text-mode set-matches matches) → void?

matches : (listof (list/c symbol? symbol?))

Sets the matching parentheses pairs for this editor.

See start-colorer’s pairs argument for more information about this argument.

Added in version 1.60 of package gui-lib.

constructor (new color:text-mode-mixin        [ [get-token get-token]         [token-sym->style token-sym->style]         [matches matches]])

→ (is-a?/c color:text-mode-mixin)

get-token : lexer = default-lexer

token-sym->style   :   (symbol? . -> . string?)       =   (λ (x) "Standard")

matches : (listof (list/c symbol? symbol?)) = null

The arguments are passed to start-colorer.

method (send a-color:text-mode on-disable-surrogate) → void?

Overrides on-disable-surrogate in mode:surrogate-text<%>.

method (send a-color:text-mode on-enable-surrogate) → void?

Overrides on-enable-surrogate in mode:surrogate-text<%>.

procedure (color:get-parenthesis-colors-table)

→ (listof (list/c symbol? string? (vectorof (is-a?/c color%)) (or/c 'low 'high)))

value color:misspelled-text-color-style-name : string?