diff options
| author | Marc Vertes <mvertes@free.fr> | 2023-08-24 10:59:39 +0200 |
|---|---|---|
| committer | Marc Vertes <mvertes@free.fr> | 2023-08-24 10:59:39 +0200 |
| commit | 8080afb5a9e38116646d69155365fb6bd09ea40c (patch) | |
| tree | ff8e7c62694a722106417eee8f8e0c6bde12667a /scanner/readme.md | |
| parent | 9146b90fe74f9615b1f4ab9a340ec038e0bd2264 (diff) | |
fix: parser must be initialized before use
Diffstat (limited to 'scanner/readme.md')
| -rw-r--r-- | scanner/readme.md | 43 |
1 files changed, 0 insertions, 43 deletions
diff --git a/scanner/readme.md b/scanner/readme.md deleted file mode 100644 index c131a9f..0000000 --- a/scanner/readme.md +++ /dev/null @@ -1,43 +0,0 @@ -# Scanner - -A scanner takes a string in input and returns an array of tokens. - -Tokens can be of the following kinds: -- identifier -- number -- operator -- separator -- string -- block - -Resolving nested blocks in the scanner is making the parser simple -and generic, without having to resort to parse tables. - -The lexical rules are provided by a language specification at language -level which includes the following: - -- a set of composable properties (1 per bit, on an integer) for each - character in the ASCII range (where all separator, operators and - reserved keywords must be defined). -- for each block or string, the specification of starting and ending - delimiter. - -## Development status - -A successful test must be provided to check the status. - -- [x] numbers starting with a digit -- [ ] numbers starting otherwise -- [x] unescaped strings (including multiline) -- [x] escaped string (including multiline) -- [x] separators (in UTF-8 range) -- [x] single line string (\n not allowed) -- [x] identifiers (in UTF-8 range) -- [x] operators, concatenated or not -- [x] single character block/string delimiters -- [x] arbitrarly nested blocks and strings -- [x] multiple characters block/string delimiters -- [x] blocks delimited by operator characters -- [ ] blocks delimited by identifiers -- [x] blocks with delimiter inclusion/exclusion rules -- [ ] blocks delimited by indentation level (python, yaml, ...) |
