ArkhAngelLifeJiggy
|
e92e5274b2
|
feat: add model-specific tokenizers and compression ratio detection (#799)
- ModelTokenizerConfig for different model families
- getTokenizerConfig() / getBytesPerTokenForModel()
- Content type detection (json, code, prose, list, technical)
- COMPRESSION_RATIOS - empirical ratios per content type
- estimateWithBounds() - confidence intervals
Features: 1.1, 1.14, 1.15
Tests: 13 passing
|
2026-04-22 13:24:12 +08:00 |
|