README.md

---
library_name: transformers
tags:
- code
license: mit
datasets:
- iamtarun/python_code_instructions_18k_alpaca
pipeline_tag: text-generation
language:
- en
---

# PyCodeGen 350M

<!-- Provide a quick summary of what the model is/does. -->

This model is finetuned version of [codegen-350M-mono](https://huggingface.co/Salesforce/codegen-350M-mono) by Salesforce trained on python code [dataset](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca) using QLORA method.


## Pretrained model description

[codegen-350M-mono](https://huggingface.co/Salesforce/codegen-350M-mono)

Codegen-350M-mono comes from the family of autoregressive models for program synthesis developed by Salesforce. 
This model was first trained on ThePile dataset which is 825.18 GiB English text corpus.
It was then adapted to generate code by training on a set of GitQuery with source codes.
Finally model has been adapted to the Python language by training on the BigPython dataset.


## Training Data

[python_code_instructions_18k_alpaca](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca)

The dataset contains problem descriptions and code in python language. 
This dataset is taken from sahil2801/code_instructions_120k, which adds a prompt column in alpaca style.

## Intended uses

The model can be used to generate python code that solves task with optionally given input data.


## Example of usage

```py
from transformers import AutoModelForCausalLM, AutoTokenizer


model = AutoModelForCausalLM.from_pretrained('chincyk/PyCodeGen')
tokenizer = AutoTokenizer.from_pretrained('chincyk/PyCodeGen')

instruction = "Write a python class that represents a calculator, then use it to add two numbers."
input = "a = 5, b = 2"

prompt = f"""
    ### Instruction:
    Use the Task below and the Input given to write the Response, which is a programming code that can solve the Task.

    ### Task:
    {instruction}

    ### Input:
    {input}
    
    ### Response:
    """

input_ids = tokenizer(prompt, truncation=True, return_tensors="pt")['input_ids']
output = model.generate(input_ids=input_ids, max_length=200)

print(tokenizer.decode(output[0], skip_special_tokens=True))

```

## Training parameters

BitsAndBytes:
- load_in_4bit: True,
- bnb_4bit_quant_type: nf4,
- bnb_4bit_use_double_quant: True,
- bnb_4bit_compute_dtype: torch.bfloat16

LoraConfig:
- r: 32,
- lora_alpha: 16,
- target_modules: all-linear,
- lora_dropout: 0.1,
- bias: none,
- task_type: CASUAL_LM

Finetuning:
- num_epochs: 15
- train_batch_size: 4
- eval_batch_size: 8
- gradient_accumulation_steps: 8
- learning_rate: 3e-4
- weight_decay: 0.01
- lr_scheduler_name: cosine
- num_warmup_steps: 190