ArkhAngelLifeJiggy e92e5274b2 feat: add model-specific tokenizers and compression ratio detection (#799) ...

- ModelTokenizerConfig for different model families
- getTokenizerConfig() / getBytesPerTokenForModel()
- Content type detection (json, code, prose, list, technical)
- COMPRESSION_RATIOS - empirical ratios per content type
- estimateWithBounds() - confidence intervals

Features: 1.1, 1.14, 1.15
Tests: 13 passing

2026-04-22 13:24:12 +08:00

21 KiB

Raw Blame History

View Raw