| # Running Validator |
|
|
| --- |
|
|
| ## Machine Requirements |
|
|
| To achieve optimal results, we recommend the following setup: |
|
|
| - **Operating System**: Ubuntu 24.04 LTS or higher |
| [Learn more about Ubuntu 24.04 LTS](https://ubuntu.com/blog/tag/ubuntu-24-04-lts) |
| - **GPU**: NVIDIA RTX A6000 with 48GB VRAM and 30 CPU Cores |
|
|
| --- |
|
|
| ## Install PM2 (Process Manager) |
|
|
| **PM2** is used to manage and monitor the validator process. If you haven’t installed PM2 yet, follow these steps: |
|
|
| 1. Install `npm` and PM2: |
| ```bash |
| sudo apt update |
| sudo apt install npm -y |
| sudo npm install pm2 -g |
| pm2 update |
| ``` |
|
|
| 2. For more details, refer to the [PM2 Documentation](https://pm2.io/docs/runtime/guide/installation/). |
|
|
| --- |
|
|
| ## Install Redis |
|
|
| 1. Install 'redis' |
| ```bash |
| sudo apt update |
| sudo apt install redis-server |
| sudo systemctl start redis |
| sudo systemctl enable redis-server |
| sudo systemctl status redis |
| ``` |
|
|
|
|
| ## Install Project Dependencies |
|
|
| ### Prerequisites |
|
|
| - **Python**: Version 3.10 or higher |
| - **pip**: Python package manager |
| - **virtualenv** (optional): For dependency isolation |
|
|
| --- |
|
|
| ### 1. Clone the Repository |
|
|
| Clone the project repository to your local machine: |
| ```bash |
| git clone https://github.com/vidaio-subnet/vidaio-subnet.git |
| cd vidaio-subnet |
| ``` |
|
|
| --- |
|
|
| ### 2. Set Up a Virtual Environment (Recommended) |
|
|
| Create and activate a virtual environment to isolate project dependencies: |
| ```bash |
| python3 -m venv venv |
| source venv/bin/activate |
| ``` |
|
|
| --- |
|
|
| ### 3. Install the Package and Dependencies |
|
|
| Install the project and its dependencies using `pip`: |
| ```bash |
| pip install -e . |
| ``` |
|
|
| --- |
|
|
| ### 4. Configure Environment Variables |
|
|
| To configure environment variables, follow these steps: |
|
|
| 1. Create a `.env` file in the project root directory by referencing the provided `.env.template` file: |
| ```bash |
| cp .env.template .env |
| ``` |
| 2. Set up a bucket in cloud storage. The base miner code utilizes MinIO to connect with cloud storage services, so you'll need to prepare your bucket using a platform that supports MinIO integration, such as Backblaze. Alternatively, you can modify the code to suit your specific requirements. *IMPORTANT*: Note that currently the `region` of the storage is hardcoded, and must be adjusted in `vidaio_subnet_core/utilities/storage_client.py` for corresponding storage, such as AWS. |
| 3. Add the required variables to the `.env` file. For example: |
| ```env |
| BUCKET_NAME="S3 buckent name" |
| BUCKET_COMPATIBLE_ENDPOINT="S3 bucket endpoint" |
| BUTKET_COMPATIBLE_ACCESS_KEY="S3 bucket personal access key" |
| BUCKET_COMPATIBLE_SECRET_KEY="S3 bucket personal secret key" |
| PEXELS_API_KEY="Your Pexels account api key" |
| WANDB_API_KEY="Your WANDB account api key" |
| ``` |
|
|
| 4. Ensure that the bucket is configured with the appropriate permissions to allow file uploads and enable public access for downloads via presigned URLs. |
|
|
| 5. Create your Pexels API key and replace it. (https://www.pexels.com/) |
| 6. Once the `.env` file is properly configured, the application will use the specified credentials for S3 bucket, Pexels and Wandb. |
|
|
|
|
| --- |
|
|
| ## Install FFMPEG |
|
|
| FFMPEG is required for processing video files. Install it using the following commands: |
| ```bash |
| sudo apt update |
| sudo apt install ffmpeg -y |
| ``` |
|
|
| For more details, refer to the [FFMPEG Documentation](https://www.ffmpeg.org/download.html#build-linux). |
|
|
| --- |
|
|
| ## Install VMAF |
|
|
| To enable video quality validation, install **VMAF** by following the steps below to set up a clean virtual environment, install dependencies, and compile the tool. |
|
|
| --- |
|
|
| Clone the VMAF repository into the working root directory of your `vidaio-subnet` package. If the `vidaio-subnet` virtual environment is currently active, deactivate it first: |
|
|
| ```bash |
| git clone https://github.com/vidAio-subnet/vmaf.git |
| cd vmaf |
| ``` |
|
|
| --- |
|
|
| ### Step 1: Set Up a Virtual Environment in VMAF directory |
|
|
| 1. Install `venv` if it’s not already installed: |
| ```bash |
| python3 -m venv vmaf-venv |
| ``` |
|
|
| 2. Activate the virtual environment: |
| ```bash |
| source vmaf-venv/bin/activate |
| ``` |
|
|
| --- |
|
|
| ### Step 2: Install Dependencies |
|
|
| 1. Install `meson`: |
| ```bash |
| pip install meson |
| ``` |
|
|
| 2. Install system dependencies: |
| ```bash |
| sudo apt-get update |
| sudo apt-get install nasm ninja-build doxygen xxd |
| ``` |
| For Ninja, verify whether the package name is `ninja` or `ninja-build` before running the install command. |
|
|
| --- |
|
|
| ### Step 3: Compile VMAF |
|
|
|
|
| 1. Set up the build environment: |
| ```bash |
| cd libvmaf |
| meson setup build --buildtype release -Denable_avx512=true |
| ``` |
|
|
| 2. Optional flags: |
| - Use `-Denable_float=true` to enable floating-point feature extractors. |
| - Use `-Denable_avx512=true` to enable AVX-512 SIMD instructions for faster processing on supported CPUs. |
| - Use `-Denable_cuda=true` to build with CUDA support (requires `nvcc` and CUDA >= 11). |
| **Note:** To enable CUDA successfully, ensure `nvcc` and the CUDA driver are installed. Refer to the [CUDA and NVCC setup guide](miner_setup.md#step-2-install-cuda-and-nvcc). |
| - Use `-Denable_nvtx=true` to enable NVTX marker support for profiling with Nsight Systems. |
| - **Recommendation:** |
| We recommend adding `-Denable_avx512=true` to enhance validation speed. If CUDA is available, include the flag `-Denable_cuda=true` But At present, VMAF does not include support for CUDA integration. |
| |
| 3. Build the project: |
| ```bash |
| ninja -vC build |
| ``` |
|
|
| --- |
|
|
| ### Step 4: Test the Build |
|
|
| Run tests to verify the build: |
| ```bash |
| ninja -vC build test |
| ``` |
|
|
| --- |
|
|
| ### Step 5: Install VMAF |
|
|
| Install the library, headers, and the command-line tool: |
| ```bash |
| ninja -vC build install |
| ``` |
|
|
| --- |
|
|
| ### Step 6: Generate Documentation |
|
|
| Generate HTML documentation: |
| ```bash |
| ninja -vC build doc/html |
| ``` |
|
|
| ### Step 7: Deactivate vmaf-venv, activate project venv |
|
|
| ```bash |
| deactivate |
| cd .. |
| cd .. |
| source venv/bin/activate |
| ``` |
|
|
| ## Running the Validator with PM2 |
|
|
| To run the validator, use the following command: |
|
|
| ```bash |
| pm2 start run.sh --name vidaio_v_autoupdater -- --wallet.name [Your_Wallet_Name] --wallet.hotkey [Your_Hotkey_Name] --subtensor.network finney --netuid 85 --axon.port [port] --logging.debug |
| ``` |
|
|
| ### Parameters: |
| - **`--wallet.name`**: Replace `[Your_Wallet_Name]` with your wallet name. |
| - **`--wallet.hotkey`**: Replace `[Your_Hotkey_Name]` with your hotkey name. |
| - **`--subtensor.network`**: Specify the target network (e.g., `finney`). |
| - **`--netuid`**: Specify the network UID (e.g., `85`). |
| - **`--axon.port`**: Replace `[port]` with the desired port number. |
| - **`--logging.debug`**: Enables debug-level logging for detailed output. |
|
|
| --- |
|
|
