2 Reader
2.1 Tokens
The Honu reader, honu-read, will tokenize the input stream according to the following regular expressions.
Identifiers are [a-zA-Z_?][a-zA-Z_?0-9]*
Strings are "[^"]*"
Numbers are \d+(\.\d+)?
And the following tokens + = * / - ^ || | && <= >= <- < > ! :: := : ; ‘ ’ . , ( ) { } [ ]
Comments can be written for a single line or in block form. Use # or // for a line comment and /* */ for block comments. Block comments can be nested.
# i am a comment |
// i am also a comment |
/* start of a comment /* with an inner comment */ end of first comment */ |
2.2 Structure
| (require honu/core/read) | package: honu |
After tokenization a Honu program will be converted into a tree with minimal structure. Enclosing tokens will be grouped into a single object represented as an s-expression. Enclosing tokens are pairs of (), {}, and [].
Consider the following stream of tokens
x ( 5 + 2 )
(x (#%parens 5 + 2))
{} will be converted to (#%braces ...) and [] will be conveted to (#%brackets ...)
procedure
(honu-read port) → any
port : port?
procedure
(honu-read-syntax name port) → any
name : any port : port?
procedure
(honu-lexer port) → (list position-token?)
port : port?