Highlights
Memory Safety: 1 prior fix. Scrutinize any change in this area.
tokenizer.rs: most-fixed (2 issues). Treat as high-risk during review.
2 high-severity fixes in this history; regressions here are high-impact.
Recurring patterns
The bug types that recur here, drawn from past fixes, not open vulnerabilities.
Memory Safety: The tokenizer previously performed direct, unsafe casting of unvalidated u32 values to char types, which can result in undefined behavior when encountering invalid Unicode escape sequences. Future parsing routines must always use safe validation APIs like char::from_u32.
Out-of-bounds Read: Relying on implicit delimiters (such as NULL terminators) instead of explicit length tracking led to out-of-bounds memory access during tokenization lookaheads. Explicit bounds checks against buffer lengths must be strictly maintained.