Fine-tune on model response only?
Hello,
It looks like when running SFTTrainer on a falcon model for example, the dataset gets pre-processed into a packed dataset where it's essentially a dataset of constant length samples. If not packing, it trims all the samples to max_seq_length passed to the trainer. With this, the supervised fine-tuning is essentially fine-tuning on a next token prediction task on the text, which makes sense for causal LM.
However, for instruction type datasets, where you want to map user queries to some kind of output, I have a prompt that looks like this
### Instructions
You are an assistant that does this and that.
### Examples
User:
Assistant:
### User
<input query>
### Assistant
<output>
I want to fine-tune falcon to perform well on this task. And I only care about the model produced <output>s from the user queries. I simply pre-process my dataset to generate training samples that include the entire prompt and all the <input_query>ies and their corresponding <output>s.
With SFTTrainer the model will simply be fine-tuning on the entire text, including the instructions and examples. This seems weird, since you will be fine-tuning the model over and over again on the same prompt, where the actual desired output is only at the end (<output>). How do you guys deal with this? Wouldn't it make more sense to compute the loss only on the output of the model? And is there a way to do this?
