Update README.md

README.md

@@ -11,4 +11,109 @@ tags:
 - ciscoAITeam
 - Cyber
 - CTI
----
+---
+
+
+# SecureBERT 2.0 Base Model
+
+SecureBERT 2.0 is a **domain-specific transformer model** built on top of ModernBERT, optimized for cybersecurity tasks. It produces contextualized embeddings for technical text and code, enabling applications such as masked language modeling, semantic search, named entity recognition, vulnerability detection, and code analysis.
+
+Cybersecurity data is highly technical, heterogeneous, and rapidly evolving. SecureBERT 2.0 leverages domain-specific pretraining to capture complex, jargon-heavy, and context-dependent language. It integrates natural language text from threat reports, blogs, technical documentation, and source code, providing superior representations for tasks requiring deep understanding of both language and code.
+
+---
+
+## Model Details
+
+- **Architecture:** ModernBERT  
+- **Base Model:** answerdotai/ModernBERT-base  
+- **Pipeline Task:** fill-mask  
+- **Max Sequence Length:** 1024 tokens  
+- **Language:** English  
+- **License:** Apache-2.0  
+
+---
+
+## Pretraining
+
+### ModernBERT Architecture
+ModernBERT introduces extended attention and hierarchical encoding to handle long documents, structured text, and source code efficiently. It supports hybrid tokenization for both natural language and code, enabling multi-modal reasoning and long-range dependency modeling—critical for cybersecurity tasks.
+
+### Pretraining Dataset
+SecureBERT 2.0 was pretrained on a large and diverse corpus, approximately **13.6B text tokens** and **53.3M code tokens**, over **13× larger** than the original SecureBERT. The dataset includes:
+
+| Dataset Category | Description |
+|-----------------|-------------|
+| Seed corpus | High-quality curated security articles, reports, and technical blogs |
+| Large-scale web text | Open web content filtered for cybersecurity relevance |
+| Reasoning-focused data | Security-oriented QA and reasoning datasets |
+| Instruction-tuning data | Procedural and instructional texts for cybersecurity workflows |
+| Code vulnerability corpus | Annotated open-source code focused on vulnerabilities |
+| Cybersecurity dialogue data | Security conversations, Q&A, and analyst workflows |
+| Original baseline dataset | Data from the first SecureBERT for continuity |
+
+#### Dataset Statistics
+
+| Dataset Category | Code Tokens | Text Tokens |
+|-----------------|------------|------------|
+| Seed corpus | 9,406,451 | 256,859,788 |
+| Large-scale web text | 268,993 | 12,231,942,693 |
+| Reasoning-focused data | -- | 3,229,293 |
+| Instruction-tuning data | 61,590 | 2,336,218 |
+| Code vulnerability corpus | 2,146,875 | -- |
+| Cybersecurity dialogue data | 41,503,749 | 56,871,556 |
+| Original baseline dataset | -- | 1,072,798,637 |
+| **Total** | 53,387,658 | 13,623,037,185 |
+
+---
+
+### Pretraining Objectives and Strategies
+- **Masked Language Modeling (MLM):** Random tokens in text and code are masked for prediction. Code-specific tokens (identifiers, operators) are masked to improve program understanding.  
+- **Microannealing Curriculum:** Gradually introduces diverse datasets, balancing high-quality and challenging data for optimal learning.  
+- **Optimization:** AdamW optimizer, learning rate of 5e-5, weight decay 0.01, MLM probability 0.10, 20 epochs, per-GPU batch size of 16 across 8 GPUs.  
+
+---
+
+## Performance Evaluation
+
+SecureBERT 2.0 was evaluated on masked language modeling tasks across objects (nouns), actions (verbs), and code tokens:
+
+| Top-n | Objects (Nouns) | Verbs (Actions) | Code Tokens |
+|-------|-----------------|----------------|-------------|
+| 1     | 56.20%          | 45.02%         | 39.27%      |
+| 2     | 69.73%          | 60.00%         | 46.90%      |
+| 3     | 75.85%          | 66.68%         | 50.87%      |
+| 4     | 80.01%          | 71.56%         | 53.36%      |
+| 5     | 82.72%          | 74.12%         | 55.41%      |
+| 10    | 88.80%          | 81.64%         | 60.03%      |
+
+
+
+This figure presents a comparative study of SecureBERT 2.0, SecureBERT, and ModernBERT on the masked language modeling (MLM) task. This shows SecureBERT 2.0 outperforms both the original SecureBERT and generic ModernBERT, particularly in code understanding and domain-specific terms.
+![image](https://cdn-uploads.huggingface.co/production/uploads/661030b81d2d202e24567c37/o0hxtirn-LV_omHsnBXhd.png)
+
+## Usage
+```bash
+pip install transformers
+```
+
+Load and use
+```python
+from transformers import AutoModelForMaskedLM, AutoTokenizer
+
+model_name = "CiscoAITeam/SecureBERT2.0-base"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForMaskedLM.from_pretrained(model_name)
+
+# Example masked sentence
+text = "The malware exploits a vulnerability in the [MASK] system."
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+
+# Get predictions
+predicted_token_id = outputs.logits.argmax(-1)
+predicted_word = tokenizer.decode(predicted_token_id[0])
+print(predicted_word)
+
+
+```