DeepTalk - Deep Learning Community
Next Token tokenizer idx
boazlavon
May 5, 2025, 12:34pm
1
Hi
Can I access the idx (from the tokenizer) of the next token some how?
Thanks!
Related topics
Topic
Replies
Views
Activity
Deepseek v3 Billing
2
72
April 23, 2025
Hermes 405 Issues
3
35
February 13, 2025
I am Struggling with Transformer inference latency
Technical Help
0
31
April 21, 2025
Inferrence Streaming Changed
Technical Help
0
19
May 13, 2025
Tensorflow graphics
0
1452
November 8, 2019