Update README.md

README.md

@@ -2,119 +2,102 @@
 license: apache-2.0
 language:
 - en
-base_model:
-- answerdotai/ModernBERT-base
-pipeline_tag: fill-mask
-library_name: transformers
 tags:
+- fill-mask
+- transformers
+- safetensors
+- modernbert
 - cybersecurity
 - ciscoAITeam
-- Cyber
+- code
 - CTI
+datasets:
+- custom
+library_name: transformers
+pipeline_tag: fill-mask
+model-index:
+- name: SecureBERT2.0-base
+  results: []
 ---
 
+# Model Card for CiscoAITeam/SecureBERT2.0-base
 
-# SecureBERT 2.0 Base Model
-
-[SecureBERT 2.0](https://arxiv.org/pdf/2510.00240) is a **domain-specific transformer model** built on top of ModernBERT, optimized for cybersecurity tasks. It produces contextualized embeddings for technical text and code, enabling applications such as masked language modeling, semantic search, named entity recognition, vulnerability detection, and code analysis.
-
-Cybersecurity data is highly technical, heterogeneous, and rapidly evolving. SecureBERT 2.0 leverages domain-specific pretraining to capture complex, jargon-heavy, and context-dependent language. It integrates natural language text from threat reports, blogs, technical documentation, and source code, providing superior representations for tasks requiring deep understanding of both language and code.
+SecureBERT 2.0 Base is a domain-specific transformer model optimized for cybersecurity tasks. It extends the ModernBERT architecture with cybersecurity-focused pretraining to produce contextualized embeddings for both technical text and code. SecureBERT 2.0 supports tasks like masked language modeling, semantic search, named entity recognition, vulnerability detection, and code analysis.
 
 ---
 
 ## Model Details
 
-- **Architecture:** ModernBERT  
-- **Base Model:** answerdotai/ModernBERT-base  
-- **Pipeline Task:** fill-mask  
-- **Max Sequence Length:** 1024 tokens  
+### Model Description
+
+SecureBERT 2.0 Base is designed for **deep contextual understanding of cybersecurity language and code**. It leverages domain-specific pretraining on a large, heterogeneous corpus covering threat reports, blogs, documentation, and codebases, making it effective for reasoning across natural language and programming syntax.
+
+- **Developed by:** Cisco AI Team  
+- **Model type:** Transformer (ModernBERT architecture)  
 - **Language:** English  
-- **License:** Apache-2.0  
+- **License:** Apache 2.0  
+- **Finetuned from model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base)
 
----
+### Model Sources
 
-## Pretraining
-
-### ModernBERT Architecture
-ModernBERT introduces extended attention and hierarchical encoding to handle long documents, structured text, and source code efficiently. It supports hybrid tokenization for both natural language and code, enabling multi-modal reasoning and long-range dependency modeling—critical for cybersecurity tasks.
-
-### Pretraining Dataset
-SecureBERT 2.0 was pretrained on a large and diverse corpus, approximately **13.6B text tokens** and **53.3M code tokens**, over **13× larger** than the original SecureBERT. The dataset includes:
-
-| Dataset Category | Description |
-|-----------------|-------------|
-| Seed corpus | High-quality curated security articles, reports, and technical blogs |
-| Large-scale web text | Open web content filtered for cybersecurity relevance |
-| Reasoning-focused data | Security-oriented QA and reasoning datasets |
-| Instruction-tuning data | Procedural and instructional texts for cybersecurity workflows |
-| Code vulnerability corpus | Annotated open-source code focused on vulnerabilities |
-| Cybersecurity dialogue data | Security conversations, Q&A, and analyst workflows |
-| Original baseline dataset | Data from the first SecureBERT for continuity |
-
-#### Dataset Statistics
-
-| Dataset Category | Code Tokens | Text Tokens |
-|-----------------|------------|------------|
-| Seed corpus | 9,406,451 | 256,859,788 |
-| Large-scale web text | 268,993 | 12,231,942,693 |
-| Reasoning-focused data | -- | 3,229,293 |
-| Instruction-tuning data | 61,590 | 2,336,218 |
-| Code vulnerability corpus | 2,146,875 | -- |
-| Cybersecurity dialogue data | 41,503,749 | 56,871,556 |
-| Original baseline dataset | -- | 1,072,798,637 |
-| **Total** | 53,387,658 | 13,623,037,185 |
+- **Repository:** [https://huggingface.co/CiscoAITeam/SecureBERT2.0-base](https://huggingface.co/CiscoAITeam/SecureBERT2.0-base)  
+- **Paper:** [arXiv:2510.00240](https://arxiv.org/abs/2510.00240)
 
 ---
 
-### Pretraining Objectives and Strategies
-- **Masked Language Modeling (MLM):** Random tokens in text and code are masked for prediction. Code-specific tokens (identifiers, operators) are masked to improve program understanding.  
-- **Microannealing Curriculum:** Gradually introduces diverse datasets, balancing high-quality and challenging data for optimal learning.  
-- **Optimization:** AdamW optimizer, learning rate of 5e-5, weight decay 0.01, MLM probability 0.10, 20 epochs, per-GPU batch size of 16 across 8 GPUs.  
+## Uses
 
----
+### Direct Use
 
-## Performance Evaluation
+- Masked language modeling for cybersecurity text and code  
+- Embedding generation for semantic search and retrieval  
+- Code and text feature extraction for downstream classification or clustering  
+- Named entity recognition (NER) on security-related entities  
+- Vulnerability detection in source code
 
-SecureBERT 2.0 was evaluated on masked language modeling tasks across objects (nouns), actions (verbs), and code tokens:
+### Downstream Use
 
-| Top-n | Objects (Nouns) | Verbs (Actions) | Code Tokens |
-|-------|-----------------|----------------|-------------|
-| 1     | 56.20%          | 45.02%         | 39.27%      |
-| 2     | 69.73%          | 60.00%         | 46.90%      |
-| 3     | 75.85%          | 66.68%         | 50.87%      |
-| 4     | 80.01%          | 71.56%         | 53.36%      |
-| 5     | 82.72%          | 74.12%         | 55.41%      |
-| 10    | 88.80%          | 81.64%         | 60.03%      |
+Fine-tuning for:
+- Threat intelligence extraction  
+- Security question answering  
+- Incident analysis and summarization  
+- Automated code review and vulnerability prediction
 
+### Out-of-Scope Use
 
+- Non-English or non-technical text  
+- General-purpose conversational AI  
+- Decision-making in real-time security systems without human oversight  
 
-This figure presents a comparative study of SecureBERT 2.0, SecureBERT, and ModernBERT on the masked language modeling (MLM) task. This shows SecureBERT 2.0 outperforms both the original SecureBERT and generic ModernBERT, particularly in code understanding and domain-specific terms.
-![image](https://cdn-uploads.huggingface.co/production/uploads/661030b81d2d202e24567c37/o0hxtirn-LV_omHsnBXhd.png)
+---
+
+## Bias, Risks, and Limitations
+
+The model reflects biases in the cybersecurity sources it was trained on, which may include:
+- Overrepresentation of certain threat actors, technologies, or organizations  
+- Inconsistent code or documentation quality  
+- Limited exposure to non-public or proprietary data formats  
+
+### Recommendations
+
+Users should evaluate outputs in their specific context and avoid automated high-stakes decisions without expert validation.  
 
-## Usage
-```bash
-pip install transformers
-```
+---
+
+## How to Get Started with the Model
 
-Load and use
 ```python
 from transformers import AutoModelForMaskedLM, AutoTokenizer
 
 model_name = "CiscoAITeam/SecureBERT2.0-base"
-
 tokenizer = AutoTokenizer.from_pretrained(model_name)
 model = AutoModelForMaskedLM.from_pretrained(model_name)
 
-# Example masked sentence
 text = "The malware exploits a vulnerability in the [MASK] system."
 inputs = tokenizer(text, return_tensors="pt")
 outputs = model(**inputs)
 
-# Get predictions
 predicted_token_id = outputs.logits.argmax(-1)
 predicted_word = tokenizer.decode(predicted_token_id[0])
 print(predicted_word)
-
-
-```
-SecureBERT 2.0 outperforms both the original SecureBERT and generic ModernBERT, particularly in code understanding and domain-specific terms.
+```