Upload model

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,42 @@
+{
+  "_name_or_path": "/home/matthias/phd/artificial_tasks/meta_adapters/models/w_fsts_pretrain_s4_32_hf_ft",
+  "architectures": [
+    "SIPFinetuningModel"
+  ],
+  "auto_map": {
+    "AutoModel": "modeling_sip_finetune.SIPFinetuningModel",
+    "AutoModelForSeq2SeqLM": "configuration_sip_finetune.SIPFinetuningModelConfig"
+  },
+  "classifier_dropout": 0.0,
+  "d_ff": 3584,
+  "d_kv": 64,
+  "d_model": 1472,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "gradient_checkpointing": false,
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "sip_finetune",
+  "num_decoder_layers": 4,
+  "num_examples": 32,
+  "num_heads": 6,
+  "num_layers": 12,
+  "num_precomputed_examples": 400,
+  "pad_token_id": 0,
+  "prefix_length": 50,
+  "prefix_max_init_length": 70,
+  "random_selection": true,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tie_word_embeddings": false,
+  "tokenizer_class": "ByT5Tokenizer",
+  "torch_dtype": "float32",
+  "transformers_version": "4.38.1",
+  "use_cache": true,
+  "vocab_size": 384
+}

configuration_sip_finetune.py

@@ -0,0 +1,21 @@
+
+from transformers import T5Config
+
+class SIPFinetuningModelConfig(T5Config):
+    model_type = "sip_finetune"
+    
+    def __init__(self,
+                 num_examples: int = 32,
+                 prefix_length: int = 50,
+                 random_selection: bool = True,
+                 # don't change these unless you change what the prefix of the model is initialized with:
+                 prefix_max_init_length: int = 70,
+                 num_precomputed_examples: int = 400,
+                 **kwargs):
+        # These are all about the initialization of the prefix.
+        self.num_examples = num_examples
+        self.prefix_length = prefix_length
+        self.random_selection = random_selection
+        self.prefix_max_init_length = prefix_max_init_length
+        self.num_precomputed_examples = num_precomputed_examples
+        super().__init__(**kwargs)

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "decoder_start_token_id": 0,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.38.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7450e9144e3d83b9b6f67ce2701ca8d544664d37ba89a80c4f8f9b3139f16480
+size 1363731112

modeling_sip_finetune.py

@@ -0,0 +1,102 @@
+import torch
+from transformers import AutoTokenizer, PretrainedConfig, T5Config, PreTrainedModel, T5ForConditionalGeneration, \
+    AutoModelForSeq2SeqLM
+    
+from typing import Optional, List, Callable, Mapping, Any, Union
+import os
+
+from .configuration_sip_finetune import SIPFinetuningModelConfig
+
+
+class SIPFinetuningModel(PreTrainedModel):
+    config_class = SIPFinetuningModelConfig
+
+    def __init__(self, config: SIPFinetuningModelConfig):
+        super().__init__(config)
+
+        self.model = T5ForConditionalGeneration(config)
+
+        # Initialize the prefix with NaNs.
+        self.register_buffer("prefix_init_tensor", torch.zeros(config.num_precomputed_examples, config.prefix_max_init_length, config.d_model))
+
+        # There are two cases: (1) we initialize the model after SIP-pretraining, i.e. the tunable prefix is not set
+        # and (2) the model has been fine-tuned on downstream data, and hence there is meaningful data in the tunable prefix
+
+        # Initialize the prefix with NaNs. If we initialize from SIP-pretraining, this will not be overwritten by a custom version of from_pretrained
+        # if we initialize after fine-tuning, the NaNs will be overwritten anyway.
+
+        self.prefix_embedding = torch.nn.Parameter(torch.nan + torch.zeros((1, self.config.prefix_length, self.config.d_model)))
+        self.prefix_has_been_initialized = False
+
+    def _initialize_prefix(self):
+        prefix_init_tensor = self.prefix_init_tensor
+        if self.config.random_selection:
+            # randomize selection of FSTs to average for initialization the prefix.
+            prefix_init_tensor = prefix_init_tensor[torch.randperm(prefix_init_tensor.shape[0]), :, :]
+
+        prefix_init_tensor = prefix_init_tensor[:self.config.num_examples, :self.config.prefix_length,
+                             :]  # shape (num ex, prefix length, d model)
+        self.prefix_embedding.data.copy_(prefix_init_tensor.mean(dim=0, keepdims=True))
+
+    @classmethod
+    def from_pretrained(
+        cls,
+        pretrained_model_name_or_path: Optional[Union[str, os.PathLike]],
+        *model_args,
+        **kwargs,
+    ):
+        model = super(SIPFinetuningModel, cls).from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
+        if torch.all(model.prefix_embedding.isnan()):
+            model._initialize_prefix()
+        return model
+
+
+    def prepare_input(self, kwargs):
+        """
+        Prepends the prefix to the given input.
+        :param kwargs:
+        :return:
+        """
+        input_ids = kwargs["input_ids"]
+
+        embedded_inputs = self.model.get_input_embeddings()(input_ids)
+
+        batch_size = input_ids.shape[0]
+
+        prefix = torch.repeat_interleave(self.prefix_embedding, batch_size, 0) #shape (batch, prefix length, embed dim)
+
+        kwargs = dict(kwargs)
+
+        embedded_inputs = torch.cat([prefix, embedded_inputs], dim=1)  # shape (batch, prefix + seq length, embed dim)
+
+        del kwargs["input_ids"]
+        kwargs["inputs_embeds"] = embedded_inputs
+
+        if "attention_mask" in kwargs:
+            ones = torch.ones((batch_size, self.config.prefix_length), device=embedded_inputs.device, dtype=kwargs["attention_mask"].dtype)
+            input_mask = torch.cat([ones, kwargs["attention_mask"]], dim=1)
+            kwargs["attention_mask"] = input_mask
+
+        return kwargs
+
+    def forward(self, **kwargs):
+        return self.model(**self.prepare_input(kwargs))
+
+    def generate(self, **kwargs):
+        return self.model.generate(**self.prepare_input(kwargs))
+
+
+    def get_optimizer(self, optimizer: Callable[..., torch.optim.Optimizer], prefix_lr:float = 1.0, **kwargs) -> torch.optim.Optimizer:
+        """
+        Return an optimizer that uses a different learning rate (typically higher) for the prefix than for the rest of the model.
+        """
+
+        prefix_params = []
+        other_params = []
+        for name, param in self.named_parameters():
+            if name == "prefix_embedding":
+                prefix_params.append(param)
+            else:
+                other_params.append(param)
+        return optimizer(params=[{"params": prefix_params, "lr": prefix_lr}, {"params": other_params}], **kwargs)
+