dataflare
/

df-arc

Model card Files Files and versions

df-arc / special_tokens_map.json

fr3on's picture

Upload custom Unigram tokenizer (v1)

ea23987 verified 4 months ago

history blame contribute delete

169 Bytes

	{
	"bos_token": "<s>",
	"eos_token": "</s>",
	"unk_token": "[UNK]",
	"pad_token": "[PAD]",
	"cls_token": "[CLS]",
	"sep_token": "[SEP]",
	"mask_token": "[MASK]"
	}