| feat: improve attachment and tool result handling
Co-authored-by: Cursor <cursoragent@cursor.com>
| 15 天前 |
| feat: improve attachment and tool result handling
Co-authored-by: Cursor <cursoragent@cursor.com>
| 15 天前 |
| feat(tokenizer): replace char/4 estimator with js-tiktoken o200k_base
Switch token counting from the legacy char/N heuristic to the o200k_base
BPE tokenizer via js-tiktoken for substantially better accuracy, especially
with non-ASCII content (CJK characters, code, JSON). The old
bytesPerTokenForExt and per-extension bytesPerToken logic is removed.
Tests are updated to compare against countTokens() dynamically rather than
hardcoded char/4 values, and legacy parity scenarios are marked as
intentional_difference.
Co-authored-by: Cursor <cursoragent@cursor.com>
| 20 天前 |