Questions tagged [lexical-analysis]
Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.
73
questions
10
votes
6
answers
2k
views
Efficient C Tokenizer/Lexer in C++
I wanted to create my own compiler for C and it's the first prototype of the Lexer. Any thoughts on improvements?
Note: I forgot to add comment lines in header files while I was coding, but I assure ...
- 103
8
votes
1
answer
299
views
C++ - Lexer for the Monkey Programming Language from the book "Writing An Interpreter In Go"
Below is a C++ version of a lexer for the Monkey programming language, originally written in Go, as presented in the book Writing An Interpreter In Go.
LOC: 262. (Excluding the tests.)
Language:
Copy ...
- 9,462
4
votes
1
answer
111
views
C99 - Lexer for the Monkey Programming Language from the book "Writing An Interpreter In Go"
Below is a C version of a lexer for the Monkey programming language, originally written in Go, as presented in the book Writing An Interpreter In Go.
LOC: 371. (Excluding the tests.)
Language:
Copy ...
- 9,462
-4
votes
2
answers
61
views
How to refactor "long" if statment in lexer? [closed]
I have a Tokenizer doing lexer thing.
It is used like
String line = "class Main {"
...
- 1
3
votes
1
answer
252
views
ANTLR4 grammar for Conventional Commits spec
I would like to create a grammar for the Conventional Commits spec and I would love to hear any feedback for what I wrote.
The spec has some ambiguities, I think, hence my usage of "island ...
- 133
6
votes
2
answers
192
views
Programming Language Source Lexer
I have a lexer (program that turns source code into tokens) written in Go that I am re-writing in Rust.
The lexer should take in a sequence of tokens, generally different special characters, integers, ...
- 163
1
vote
1
answer
157
views
Rust state-machine lexer
I tried implementing a lexer in rust that peeks ahead at the next character and makes a decision based on that.
However, i am told that this is bad practice, and instead i should be using finite-state-...
- 57
0
votes
1
answer
171
views
supporting unicode in rust lexer
I am working on a lexer in rust.
It has been through many different iterations, going from iterating over a Vec<char> to ...
- 57
4
votes
1
answer
424
views
lexer for programming languages in rust
I'm relatively new to rust, so you don't need to go easy on the criticism.
right now, this lexes 1MB of text (well, trims a 1MB-long word) in 30ms.
is there any way I can make this faster?
cursor.rs:
<...
- 57
0
votes
1
answer
127
views
top-down lexer in rust
the other day, i decided to build a top-down lexer in rust, just for fun.
this is what i have so far:
...
- 37
3
votes
1
answer
279
views
Generating indent and dedent tokens for python-style indenting code
I found lexy, a C++ template-based parser combinator, that helps a lot to build a language parser.
The problem is that lexy can only deal with context free grammar, and python indentation is not ...
- 293
2
votes
1
answer
149
views
Simple Manual Lexer in Haskell
As a beginner exercise, I made small manual lexer that recognizes three types of inputs:
integers: /[-]?[0-9]+/
strings, inside double quotes, with backslash ...
- 121
1
vote
1
answer
99
views
Lexer for shell-like language [rust]
I am working on writing a rust implementation of a sh-like language.
Rather than posting 600 lines of code here, you can just go to the GitHub repository.
Is this a ...
- 192
1
vote
1
answer
206
views
C# Language Lexer
Here is a Lexer for a programming language I am working on. Any feedback would be appreciated. I only started learning C# a couple of days ago, so please excuse my newbie code :)
...
- 125
3
votes
1
answer
291
views
Simple tokenizer in C
I implemented a simple tokenizer. Would love to hear your feedback on code style, best practices:
...
- 269