# Load model directly
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("Wikidepia/IndoConvBERT-base")
model = AutoModel.from_pretrained("Wikidepia/IndoConvBERT-base")Quick Links
IndoConvBERT Base Model
IndoConvBERT is a ConvBERT model pretrained on Indo4B.
Pretraining details
We follow a different training procedure: instead of using a two-phase approach, that pre-trains the model for 90% with 128 sequence length and 10% with 512 sequence length, we pre-train the model with 512 sequence length for 1M steps on a v3-8 TPU.
The current version of the model is trained on Indo4B and small Twitter dump.
Acknowledgement
Big thanks to TFRC (TensorFlow Research Cloud) for providing free TPU.
- Downloads last month
- 5
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="Wikidepia/IndoConvBERT-base")