VDrontV3-Mini - custom MoE model. (gpt 2 tokenizer)

You can find model.py in the repository; it describes the model architecture.

The repository also contains use.py and GUIvdront.py—example scripts for running the model. To use them, download the model files and place them in the VDrontV3-Mini/ folder, then place model.py, use.py, and GUIvdront.py alongside that folder and run the script.

The model has two modes: "Base" and "Quality" In "Base" mode, responses are shorter and more varied, whereas in "Quality" mode, they are more similar in structure but longer than those in "Base".

GUIvdront.py

1024 ctx

Format user, assistant, endoftext

cc0-1.0

model architecture

Base model VDrontV2-0.1b

VDrontV3 is a transformer-based architecture that implements a hybrid Mixture of Experts (MoE) design with two specialized expert blocks.

Embedding & Routing: The input passes through standard token and position embeddings. A "Router top-1" (a linear layer) processes the embedding to select a single expert (top-1 routing).

Expert Block: A sequence of transformer layers is replicated into multiple parallel "Expert Blocks" (purple boxes). The router dynamically activates exactly one of these expert versions during each forward pass.

Base Block: After the expert block, the hidden states pass through a shared, fixed "Base Block" (green box) consisting of standard transformer layers. This provides a common foundation for all routing paths.

Output Block & Mode Block: The final layers form an "Output Block" (also multi-version) that feeds into the LM head. Crucially, the architecture includes a "ModeBlock" (yellow box), which suggests the model can switch between different inference modes by selecting different versions of the output block.

Chat Quality example

You: Hello, what is a Python?
AI: Python is an interpreter that provides the code and its high-level programming language. It was widely used to perform several popular
programming languages like Ruby on your programming language known for its high-level programming language's high-level, easy-to-read syntax.
Its powerful, easy-to-use, easy-to-read syntax and easy-to-read syntax. It has an excellent choice for beginners. Python is one of the most
popular programming languages in Python, and has been popular among developers. It is one of the standard programming languages in the world today.
It is one of the most popular programming languages in the world. It has a high-level, easy-to-read syntax and easy-to-read syntax but can be as easy and accessible.

You: Who are you?
AI: I am an artificial intelligence assistant, a language model designed to help people with tasks. I don't have a
physical body and generate human consciousness. I'm trained on a large dataset of data, which allows me to understand patterns in
language and information. I don't produce human-like responses. My primary purpose is to make your life easier and more enjoyable. I'm
trained on a vast range of tasks without having a physical body or emotions like humans do my job is to assist with writing, and more interesting
conversation. I don't have a physical body or emotions like humans do. I'm here to help with everything from writing, and more interesting. So,
what can I do today? I'm here for you? I'm here today? I'm here to help you with today?

You: What is a PC?
AI: A computer, or computer's computer system that runs on the internet and manages the Internet. It acts as an
operating system to access and control over the computer and the computer and all computers in the computer and software
that uses a network of computers, including both on your computer and on your computer and the computer and all over the internet.
 The OSI, which handles tasks without having a physical computer, and users to communicate with each other. A computer and the internet.
It handles servers, and manages and controls the internet through a single computer and the internet. There are three main types: the internet,
and the internet has several key components: the internet and the internet. It operates on the internet through a network of computers and networks
throughout various computers and devices. It provides a vast amount of data; and the most important aspect of modern computing, enabling everything from
email communication, and the internet. 

Chat Base example

You: Hello!
AI: Hello, I'm here to help you with your questions.

You: What is a Python?
AI: A.py is a program that uses Python to create and execute code. It can be used for programming, and data analysis. Key aspects:
1. Python-based
2. Python-based
3. Data analysis
4. Data analysis
5. Data analysis

You: Who are you?
AI: I am a professional golfer who is passionate about golf. I believe in the importance of golfing and the benefits of playing with golfers.        
* The first time you played this game  
2. Golfers enjoy golfing and playing with golf 
3. You can play golf with golfers

VDrontV3-Mini is a test version of the VDrontV3 architecture; the "Base" and "Quality" modes merely make the response format resemble those versions but do not significantly improve quality, as Mini is based on the 0.1b base model(VDrontV2-0.1b).