Next Token tokenizer idx

boazlavon May 5, 2025, 12:34pm 1

Hi
Can I access the idx (from the tokenizer) of the next token some how?
Thanks!

Topic		Replies	Views
Tensor Book factory reset Technical Help	0	1879	April 10, 2019
Generate utterances Machine Learning Research	0	1524	March 16, 2020
Ignoring visible gpu device Technical Help	0	1728	December 19, 2018
TPU TensorBooks	1	1743	October 4, 2018
Sim Card Slot on 2-19 Tensorbook?	0	1364	August 1, 2019