ArkhAngelLifeJiggy
e92e5274b2
feat: add model-specific tokenizers and compression ratio detection (#799)
- ModelTokenizerConfig for different model families
- getTokenizerConfig() / getBytesPerTokenForModel()
- Content type detection (json, code, prose, list, technical)
- COMPRESSION_RATIOS - empirical ratios per content type
- estimateWithBounds() - confidence intervals
Features: 1.1, 1.14, 1.15
Tests: 13 passing
2026-04-22 13:24:12 +08:00
..
2026-03-31 03:34:03 -07:00
2026-04-20 17:13:09 +08:00
2026-04-21 23:17:12 +08:00
2026-04-04 23:26:14 +05:30
2026-04-20 16:24:02 +08:00
2026-04-22 09:37:57 +08:00
2026-04-01 02:36:07 +08:00
2026-03-31 03:34:03 -07:00
2026-04-15 19:38:46 +08:00
2026-03-31 03:34:03 -07:00
2026-03-31 03:34:03 -07:00
2026-04-21 18:28:03 +08:00
2026-04-13 22:34:16 +08:00
2026-03-31 03:34:03 -07:00
2026-03-31 03:34:03 -07:00
2026-04-04 23:58:34 +05:30
2026-04-04 21:19:27 +08:00
2026-04-04 23:26:14 +05:30
2026-03-31 03:34:03 -07:00
2026-03-31 03:34:03 -07:00
2026-04-14 19:08:54 +08:00
2026-04-09 21:18:57 +08:00
2026-03-31 03:34:03 -07:00
2026-04-20 16:24:02 +08:00
2026-03-31 03:34:03 -07:00
2026-04-01 13:31:18 +08:00
2026-03-31 03:34:03 -07:00
2026-04-19 09:02:52 +08:00
2026-04-19 09:02:52 +08:00
2026-04-04 23:04:34 +05:30
2026-04-04 21:19:27 +08:00
2026-04-08 16:03:31 +08:00
2026-04-02 11:04:35 +05:30
2026-03-31 03:34:03 -07:00
2026-03-31 03:34:03 -07:00
2026-03-31 03:34:03 -07:00
2026-04-22 13:24:12 +08:00
2026-04-22 13:24:12 +08:00
2026-03-31 03:34:03 -07:00
2026-03-31 03:34:03 -07:00
2026-03-31 03:34:03 -07:00
2026-03-31 03:34:03 -07:00