Article
We Fact-Checked Our Own Security Dataset. Here's What We Fixed.
scthornton
• • 1perfecxion.ai PRO
perfecXion
AI & ML interests
Security
Recent Activity
upvoted an article 8 days ago
We Fact-Checked Our Own Security Dataset. Here's What We Fixed. reacted to scthornton's post with 🧠 8 days ago
SecureCode update: we went back and fact-checked our own security dataset and corrected what didn't hold up.
The original claim was "complete incident grounding, every example ties to a documented CVE." An adversarial re-audit found that it was overstated: many CVEs were misattributed, and many "incidents" were representative scenarios carrying invented statistics. So we fixed it.
- Grounding: re-verified every reference. Removed 802 misattributed CVEs on the web side, corrected or honestly relabeled the incident narratives, and confirmed the AI/ML conversation CVEs are real (EchoLeak CVE-2025-32711, EmailGPT CVE-2024-5184, and others).
- Fix-correctness: reviewed whether each "secure" example actually eliminates the vulnerability. Removed 28 that did not (a "secure" secret scanner whose entropy check always returned zero, an Angular example still using bypassSecurityTrustHtml, and more).
- Leakage: re-split so near-duplicates stay on one side. Test contamination went from 11.6% to zero.
- Viewer, schema, and metadata: rebuilt as parquet under a shared schema. All three viewers are live.
- Models: retrained the whole family on the corrected data so the fix reaches the weights, not just the cards. Now ten open models (3B to 26B), including two new Gemma 4 variants, refreshed locally on a DGX Spark GB10. The paper (arXiv:2512.18542) was revised to match.
Counts moved from 2,185 to 2,372 unified (web 1,625 + AI/ML 747). A slightly smaller, fully-checked dataset beats a larger one you have to take on faith. Full writeup and links in the article.
Datasets: scthornton/securecode, scthornton/securecode-web, scthornton/securecode-aiml
reacted to scthornton's post with ❤️ 8 days ago
SecureCode update: we went back and fact-checked our own security dataset and corrected what didn't hold up.
The original claim was "complete incident grounding, every example ties to a documented CVE." An adversarial re-audit found that it was overstated: many CVEs were misattributed, and many "incidents" were representative scenarios carrying invented statistics. So we fixed it.
- Grounding: re-verified every reference. Removed 802 misattributed CVEs on the web side, corrected or honestly relabeled the incident narratives, and confirmed the AI/ML conversation CVEs are real (EchoLeak CVE-2025-32711, EmailGPT CVE-2024-5184, and others).
- Fix-correctness: reviewed whether each "secure" example actually eliminates the vulnerability. Removed 28 that did not (a "secure" secret scanner whose entropy check always returned zero, an Angular example still using bypassSecurityTrustHtml, and more).
- Leakage: re-split so near-duplicates stay on one side. Test contamination went from 11.6% to zero.
- Viewer, schema, and metadata: rebuilt as parquet under a shared schema. All three viewers are live.
- Models: retrained the whole family on the corrected data so the fix reaches the weights, not just the cards. Now ten open models (3B to 26B), including two new Gemma 4 variants, refreshed locally on a DGX Spark GB10. The paper (arXiv:2512.18542) was revised to match.
Counts moved from 2,185 to 2,372 unified (web 1,625 + AI/ML 747). A slightly smaller, fully-checked dataset beats a larger one you have to take on faith. Full writeup and links in the article.
Datasets: scthornton/securecode, scthornton/securecode-web, scthornton/securecode-aiml
Organizations
upvoted an article 8 days ago
reacted to scthornton's post with 🧠❤️ 8 days ago
Post
2647
SecureCode update: we went back and fact-checked our own security dataset and corrected what didn't hold up.
The original claim was "complete incident grounding, every example ties to a documented CVE." An adversarial re-audit found that it was overstated: many CVEs were misattributed, and many "incidents" were representative scenarios carrying invented statistics. So we fixed it.
- Grounding: re-verified every reference. Removed 802 misattributed CVEs on the web side, corrected or honestly relabeled the incident narratives, and confirmed the AI/ML conversation CVEs are real (EchoLeak CVE-2025-32711, EmailGPT CVE-2024-5184, and others).
- Fix-correctness: reviewed whether each "secure" example actually eliminates the vulnerability. Removed 28 that did not (a "secure" secret scanner whose entropy check always returned zero, an Angular example still using bypassSecurityTrustHtml, and more).
- Leakage: re-split so near-duplicates stay on one side. Test contamination went from 11.6% to zero.
- Viewer, schema, and metadata: rebuilt as parquet under a shared schema. All three viewers are live.
- Models: retrained the whole family on the corrected data so the fix reaches the weights, not just the cards. Now ten open models (3B to 26B), including two new Gemma 4 variants, refreshed locally on a DGX Spark GB10. The paper (arXiv:2512.18542) was revised to match.
Counts moved from 2,185 to 2,372 unified (web 1,625 + AI/ML 747). A slightly smaller, fully-checked dataset beats a larger one you have to take on faith. Full writeup and links in the article.
Datasets: scthornton/securecode, scthornton/securecode-web, scthornton/securecode-aiml
The original claim was "complete incident grounding, every example ties to a documented CVE." An adversarial re-audit found that it was overstated: many CVEs were misattributed, and many "incidents" were representative scenarios carrying invented statistics. So we fixed it.
- Grounding: re-verified every reference. Removed 802 misattributed CVEs on the web side, corrected or honestly relabeled the incident narratives, and confirmed the AI/ML conversation CVEs are real (EchoLeak CVE-2025-32711, EmailGPT CVE-2024-5184, and others).
- Fix-correctness: reviewed whether each "secure" example actually eliminates the vulnerability. Removed 28 that did not (a "secure" secret scanner whose entropy check always returned zero, an Angular example still using bypassSecurityTrustHtml, and more).
- Leakage: re-split so near-duplicates stay on one side. Test contamination went from 11.6% to zero.
- Viewer, schema, and metadata: rebuilt as parquet under a shared schema. All three viewers are live.
- Models: retrained the whole family on the corrected data so the fix reaches the weights, not just the cards. Now ten open models (3B to 26B), including two new Gemma 4 variants, refreshed locally on a DGX Spark GB10. The paper (arXiv:2512.18542) was revised to match.
Counts moved from 2,185 to 2,372 unified (web 1,625 + AI/ML 747). A slightly smaller, fully-checked dataset beats a larger one you have to take on faith. Full writeup and links in the article.
Datasets: scthornton/securecode, scthornton/securecode-web, scthornton/securecode-aiml
- 8 replies
updated 3
models 4 months ago
upvoted an article 4 months ago
Article
IntentGuard: Building a Production-Grade Vertical Intent Classifier for LLM Safety
perfecXion
• • 2 published an article 4 months ago
Article
IntentGuard: Building a Production-Grade Vertical Intent Classifier for LLM Safety
perfecXion
• • 2 reacted to their post with 👍 5 months ago
Post
2573
# IntentGuard: Open-Source Vertical Intent Classifiers for LLM Guardrails
Three models published to the Hub:
- [perfecXion/intentguard-finance]( perfecXion/intentguard-finance)
- [perfecXion/intentguard-healthcare]( perfecXion/intentguard-healthcare)
- [perfecXion/intentguard-legal]( perfecXion/intentguard-legal)
DeBERTa-v3-xsmall fine-tuned for three-way classification: **allow**, **deny**, or **abstain**. ONNX + INT8 quantized, under 80MB, p99 <30ms on CPU. Margin-based thresholds (not argmax) — uncertain queries route to clarification instead of forcing a guess.
**Eval results (adversarial test sets, ~470-480 examples per vertical):**
| Vertical | Accuracy | Legit-Block Rate | Off-Topic-Pass Rate |
|----------|----------|------------------|---------------------|
| Finance | 99.6% | 0.00% | 0.00% |
| Healthcare | 98.9% | 0.00% | 0.98% |
| Legal | 97.9% | 0.00% | 0.50% |
Apache 2.0. Full pipeline + Docker configs on [GitHub](https://github.com/perfecxion-ai/intentguard).
Feedback welcome on domain coverage, adversarial robustness, and multilingual demand.
Three models published to the Hub:
- [perfecXion/intentguard-finance]( perfecXion/intentguard-finance)
- [perfecXion/intentguard-healthcare]( perfecXion/intentguard-healthcare)
- [perfecXion/intentguard-legal]( perfecXion/intentguard-legal)
DeBERTa-v3-xsmall fine-tuned for three-way classification: **allow**, **deny**, or **abstain**. ONNX + INT8 quantized, under 80MB, p99 <30ms on CPU. Margin-based thresholds (not argmax) — uncertain queries route to clarification instead of forcing a guess.
**Eval results (adversarial test sets, ~470-480 examples per vertical):**
| Vertical | Accuracy | Legit-Block Rate | Off-Topic-Pass Rate |
|----------|----------|------------------|---------------------|
| Finance | 99.6% | 0.00% | 0.00% |
| Healthcare | 98.9% | 0.00% | 0.98% |
| Legal | 97.9% | 0.00% | 0.50% |
docker run -p 8080:8080 ghcr.io/perfecxion/intentguard:finance-latest
curl -X POST http://localhost:8080/v1/classify \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "What are current mortgage rates?"}]}'Apache 2.0. Full pipeline + Docker configs on [GitHub](https://github.com/perfecxion-ai/intentguard).
Feedback welcome on domain coverage, adversarial robustness, and multilingual demand.
posted an update 5 months ago
Post
2573
# IntentGuard: Open-Source Vertical Intent Classifiers for LLM Guardrails
Three models published to the Hub:
- [perfecXion/intentguard-finance]( perfecXion/intentguard-finance)
- [perfecXion/intentguard-healthcare]( perfecXion/intentguard-healthcare)
- [perfecXion/intentguard-legal]( perfecXion/intentguard-legal)
DeBERTa-v3-xsmall fine-tuned for three-way classification: **allow**, **deny**, or **abstain**. ONNX + INT8 quantized, under 80MB, p99 <30ms on CPU. Margin-based thresholds (not argmax) — uncertain queries route to clarification instead of forcing a guess.
**Eval results (adversarial test sets, ~470-480 examples per vertical):**
| Vertical | Accuracy | Legit-Block Rate | Off-Topic-Pass Rate |
|----------|----------|------------------|---------------------|
| Finance | 99.6% | 0.00% | 0.00% |
| Healthcare | 98.9% | 0.00% | 0.98% |
| Legal | 97.9% | 0.00% | 0.50% |
Apache 2.0. Full pipeline + Docker configs on [GitHub](https://github.com/perfecxion-ai/intentguard).
Feedback welcome on domain coverage, adversarial robustness, and multilingual demand.
Three models published to the Hub:
- [perfecXion/intentguard-finance]( perfecXion/intentguard-finance)
- [perfecXion/intentguard-healthcare]( perfecXion/intentguard-healthcare)
- [perfecXion/intentguard-legal]( perfecXion/intentguard-legal)
DeBERTa-v3-xsmall fine-tuned for three-way classification: **allow**, **deny**, or **abstain**. ONNX + INT8 quantized, under 80MB, p99 <30ms on CPU. Margin-based thresholds (not argmax) — uncertain queries route to clarification instead of forcing a guess.
**Eval results (adversarial test sets, ~470-480 examples per vertical):**
| Vertical | Accuracy | Legit-Block Rate | Off-Topic-Pass Rate |
|----------|----------|------------------|---------------------|
| Finance | 99.6% | 0.00% | 0.00% |
| Healthcare | 98.9% | 0.00% | 0.98% |
| Legal | 97.9% | 0.00% | 0.50% |
docker run -p 8080:8080 ghcr.io/perfecxion/intentguard:finance-latest
curl -X POST http://localhost:8080/v1/classify \
-H "Content-Type: application/json" \
-d '{"messages": [{"role": "user", "content": "What are current mortgage rates?"}]}'Apache 2.0. Full pipeline + Docker configs on [GitHub](https://github.com/perfecxion-ai/intentguard).
Feedback welcome on domain coverage, adversarial robustness, and multilingual demand.
published 3
models 5 months ago
upvoted an article 5 months ago
Article
AI Coding Assistants Keep Shipping Vulnerable Code -- Here's What We're Doing About It
scthornton
• • 3 reacted to scthornton's post with 👀🚀 5 months ago
Post
1984
# SecureCode Dataset Family Update: 2,185 Security Examples, Framework-Specific Patterns, Clean Parquet Loading
Hey y'all,
Quick update on the SecureCode dataset family. We've restructured things and fixed several issues:
**What changed:**
- The datasets are now properly split into three repos: [unified]( scthornton/securecode) (2,185), [web]( scthornton/securecode-web) (1,378), [AI/ML]( scthornton/securecode-aiml) (750)
- All repos now use Parquet format --
- SecureCode Web now includes 219 framework-specific examples (Express, Django, Spring Boot, Flask, Rails, Laravel, ASP.NET Core, FastAPI, NestJS)
- Data cards have been corrected and split sizes fixed
**Why it matters:**
With AI-generated code accounting for 60%+ of some codebases (Checkmarx 2025), security training data is more important than ever. Every example in SecureCode is grounded in a real CVE with 4-turn conversations that mirror actual developer-AI workflows.
If you're working on code generation models, I'd love to hear how you're approaching the security angle. Are there vulnerability categories or frameworks you'd like to see covered?
Paper: [arxiv.org/abs/2512.18542](https://arxiv.org/abs/2512.18542)
Hey y'all,
Quick update on the SecureCode dataset family. We've restructured things and fixed several issues:
**What changed:**
- The datasets are now properly split into three repos: [unified]( scthornton/securecode) (2,185), [web]( scthornton/securecode-web) (1,378), [AI/ML]( scthornton/securecode-aiml) (750)
- All repos now use Parquet format --
load_dataset() just works, no deprecated loading scripts- SecureCode Web now includes 219 framework-specific examples (Express, Django, Spring Boot, Flask, Rails, Laravel, ASP.NET Core, FastAPI, NestJS)
- Data cards have been corrected and split sizes fixed
**Why it matters:**
With AI-generated code accounting for 60%+ of some codebases (Checkmarx 2025), security training data is more important than ever. Every example in SecureCode is grounded in a real CVE with 4-turn conversations that mirror actual developer-AI workflows.
If you're working on code generation models, I'd love to hear how you're approaching the security angle. Are there vulnerability categories or frameworks you'd like to see covered?
Paper: [arxiv.org/abs/2512.18542](https://arxiv.org/abs/2512.18542)
liked 3
datasets 5 months ago
Convert to Parquet format (fixes datasets >= 4.0.0 compatibility) Fixes https://huggingface.co/datasets/scthornton/securecode-v2/discussions/2 - Removes deprecated loading script - Uses modern Parquet format - Aligns schema across all splits - Compatible with datasets >= 4.0.0
1
#3 opened 6 months ago
by perfecXion