Security context

Highlights

Out-of-Bounds Read: 5 prior fixes. Scrutinize any change in this area.

src/normalizer.cc: most-fixed (4 issues). Treat as high-risk during review.

7 high-severity fixes in this history; regressions here are high-impact.

Recurring patterns

Out-of-Bounds Read: Malicious model files with corrupted or invalid trie structures bypass structural validation, leading to out-of-bounds reads during prefix searches and normalization operations. Ensure all trie-based offset scans are fully validated prior to traversal.

Memory Safety: Stack overflows and uninitialized memory reads arise during the processing of highly nested or recursive token configurations in deep models, requiring strict recursion depth checks during resegmentation and parsing.

Out-of-Bounds Read: Zero-copy token decoding exposed to raw numpy arrays can trigger memory access violations if strides are non-contiguous or dimensions are misrepresented. This requires proactive checking of the numpy buffer structure.