* feat: add streaming token counter
- Add StreamingTokenCounter for real-time token counting during generation
- Tracks output tokens as they arrive from stream
- Calculates tokens per second rate
- Add tests (4 passing)
PR 4A: Streaming Token Counter (Features 1.2, 1.7)
* refactor: move StreamingTokenCounter to separate file
- Extract StreamingTokenCounter from tokens.ts to streamingTokenCounter.ts
- Add getEstimatedRemainingTokens() method
- Update test import
* fix: word-boundary token counting for stable stream totals
- Accumulate raw content, count only at word boundaries
- Eliminates instability from arbitrary chunk boundaries
- Add finalize() to flush remaining content on stream end
- Add characterCount getter for raw content tracking
- Rename getEstimatedRemainingTokens -> getEstimatedGenerationTimeMs
- Add comprehensive tests
* fix: update streamingTokens test for word-boundary API
- Add finalize() call before checking output tokens
- Use characterCount for interim checks
- Add spaces to trigger word boundary counting
* fix: add estimateRemainingTokens/Time methods
- Add estimateRemainingTokens(target) method
- Add estimateRemainingTimeMs(target) method
- Covers non-blocking: now properly estimates remaining tokens
* fix: PR 797 - fix word boundary counting, consolidate tests
Blockers (Vasanthdev2004):
- recountAtWordBoundary now searches forward from lastCountedIndex+1
- Finds NEXT space after already-counted region, not before it
- Provides accurate live token counts during streaming, not just finalize()
Non-blocking (gnanam1990):
- Delete streamingTokens.test.ts, merge tests into streamingTokenCounter.test.ts
- Added interim-counting test to verify counting updates during streaming
* fix: PR 797 - fix word boundary advancement after space
Blocking:
- Fix recountAtWordBoundary to skip past space when searching for next boundary
- After counting at a space, indexOf(' ') returns 0 (the space itself)
- Now starts search from index 1 to find the NEXT word boundary
- Short chunks now properly trigger count advancement
Non-blocking:
- Add test verifying count increases after each word boundary
- Add test for space-skipping behavior
