davidberenstein1957 (David Berenstein)

posted an update about 2 months ago

Post

2525

🚨 Phare LLM benchmark V2: Reasoning models don't guarantee better security

Read the full blog here: https://huggingface.co/blog/davidberenstein1957/phare-llm-benchmark-v2

reacted to anakin87's post with 🧠 3 months ago

Post

2883

LLMs can leak their post-training data (RL included) 💧

New interesting paper on this topic from Google DeepMind: Extracting alignment data in open models (2510.18554)

It's known that Language Models memorize data that can be extracted via prompting.

In this paper, the authors investigate this aspect:
- using open models, where prompting can be fully customized by the user, including special tokens.
- focusing on open-source models like Olmo, where full training data is available.

📤 How do they extract data?

During post-training (like SFT), new tokens such as <|user|> are introduced.

The authors hypothesize prompting the model with these tokens can make it output its alignment data (remember Magpie?).

For example, for SFT, their extraction prompt is <|endoftext|><|user|>.

📏 Evaluating memorization

The authors compare each sampled example with the original data using vector search with embedding similarity.

They find that many outputs are semantically very similar to the original data, even if the exact words differ.

Traditional string-matching algorithms underestimate memorization by 10x.

🔁 What about RL?

Surprisingly, the same technique works to extract data from Reinforcement Learning (PPO/GRPO) phases.

This is counter-intuitive because the RL objective is not designed to increase sequence likelihoods (unlike SFT).

Practical limitation: in this case, extraction relies on using the initial part of the training prompt, which is not generally public.

📈 Are the extracted data effective for post-training?

Both in SFT and RL, the extracted data can be used to fine-tune models to similar performance to the originals.

The authors suggest that model distillation, where a stronger model is used to drive the training of a weaker one, may be a form of indirect training on the original dataset.

posted an update 7 months ago

Post

612

Announcing RealPerformance, a dataset of functional issues of language models that mirrors failure patterns identified through rigorous testing in real LLM agents

https://huggingface.co/blog/davidberenstein1957/realperformance-llm-business-compliance

posted an update 7 months ago

Post

406

🚨 LLMs recognise bias but also reproduce harmful stereotypes: an analysis of bias in leading LLMs

I've written a new entry in our series on the Giskard, BPIFrance and Google Deepmind Phare benchmark(phare.giskard.ai).

This time it covers bias: https://huggingface.co/blog/davidberenstein1957/llms-recognise-bias-but-also-produce-stereotypes

Previous entry on hallucinations: https://huggingface.co/blog/davidberenstein1957/phare-analysis-of-hallucination-in-leading-llms

1 reply

·

posted an update 8 months ago

Post

1768

I created a collection of FLUX.1 models but 4x faster PrunaAI/flux1-but-4x-faster-66c0b7340836dd7a55e9c0ea

reacted to sayakpaul's post with 🚀🤗 9 months ago

Post

2941

Diffusers supports a good variety of quantization backends. It can be challenging to navigate through them, given the complex nature of diffusion pipelines in general.

So, @derekl35 set out to write a comprehensive guide that puts users in the front seat. Explore the different backends we support, learn the trade-offs they offer, and finally, check out the cool space we built that lets you compare quantization results.

Give it a go here:
https://lnkd.in/gf8Pi4-2

2 replies

·

posted an update 9 months ago

Post

1098

Good answers are not necessarily factual answers: an analysis of hallucination in leading LLMs

https://huggingface.co/blog/davidberenstein1957/phare-analysis-of-hallucination-in-leading-llms

posted an update 10 months ago

Post

2276

🔥 Announcing FLUX-Juiced: The Fastest Image Generation Endpoint (2.6x faster)!

Optimisations are widely applied and can reduce inference time, but their impact on quality often remains unclear, so we decided to challenge the status quo and create our own optimised version of FLUX.1[dev] called FLUX-juiced.

Blog: https://huggingface.co/blog/PrunaAI/flux-fastest-image-generation-endpoint

posted an update 10 months ago

Post

1744

🧑‍🏫 I wrote a brief blogpost to give An Introduction to AI Model Optimization Techniques!

URL: https://huggingface.co/blog/PrunaAI/introduction-to-ai-model-optimization-techniques

posted an update 10 months ago

Post

1410

RealHarm: A Collection of Real-World Language Model Application Failure

I'm David from Giskard, and we work on securing your Agents.
Today, we are launching RealHarm: a dataset of real-world problematic interactions with AI agents, drawn from publicly reported incidents.

Check out the dataset and paper: https://realharm.giskard.ai/

posted an update 11 months ago

Post

2113

🚨 New Bonus Unit: Tracing & Evaluating Your Agent! 🚨

Learn how to transform your agent from a simple demo into a robust, reliable product ready for real users.

UNIT: https://huggingface.co/learn/agents-course/bonus-unit2/introduction

In this unit, you'll learn:
- Offline Evaluation – Benchmark and iterate your agent using datasets.
- Online Evaluation – Continuously track key metrics such as latency, costs, and user feedback.

Happy testing and improving!

Thanks Langfuse team!

posted an update 11 months ago

Post

2438

🔥 Text2SQL, explore and share any data analysis!

🤗 Hugging Face - Dataset Studio is an amazing new feature.

🚀 Start yourself: https://huggingface.co/datasets/fka/awesome-chatgpt-prompts/viewer/default/train

📺 YouTube: https://youtu.be/5LUZq7MHolA?feature=shared

reacted to their post with 🚀🔥 11 months ago

Post

4268

🥊 Epic Agent Framework Showdown! Available today!

🔵 In the blue corner, the versatile challenger with a proven track record of knowledge retrieval: LlamaIndex!

🛑 In the red corner, the defender, weighing in with lightweight efficiency: Hugging Face smolagents!

🔗 URL:

agents-course

We just published the LlamaIndex unit for the agents course, and it is set to offer a great contrast between the smolagents unit by looking at

- What makes llama-index stand-out
- How the LlamaHub is used for integrations
- Creating QueryEngine components
- Using agents and tools
- Agentic and multi-agent workflows

The team has been working flat-out on this for a few weeks. Supported by Logan Markewich and Laurie Voss over at LlamaIndex.

Who won? You decide!

posted an update 11 months ago

Post

4268

🥊 Epic Agent Framework Showdown! Available today!

🔵 In the blue corner, the versatile challenger with a proven track record of knowledge retrieval: LlamaIndex!

🛑 In the red corner, the defender, weighing in with lightweight efficiency: Hugging Face smolagents!

🔗 URL:

agents-course

We just published the LlamaIndex unit for the agents course, and it is set to offer a great contrast between the smolagents unit by looking at

- What makes llama-index stand-out
- How the LlamaHub is used for integrations
- Creating QueryEngine components
- Using agents and tools
- Agentic and multi-agent workflows

The team has been working flat-out on this for a few weeks. Supported by Logan Markewich and Laurie Voss over at LlamaIndex.

Who won? You decide!

posted an update 11 months ago

Post

3058

🫸 New release to push vector search to the Hub with vicinity and work with any serialisable objects.

🧑‍🏫 KNN, HNSW, USEARCH, ANNOY, PYNNDESCENT, FAISS, and VOYAGER.

🔗 Example Repo: minishlab/my-vicinity-repo

reacted to jsulz's post with ❤️➕🚀 12 months ago

Post

3173

Toward the end of last year, the Xet team provided an inside look into the foundations of how we plan to enable rapid experimentation and iteration for the AI builders on the Hub: https://huggingface.co/blog/from-files-to-chunks

But it turns out chunks aren't all you need!

Our goal is to bring:
🚀 Faster uploads
⏬ Speedy downloads
💪 All without sacrificing your workflow

To do that, we need the infrastructure and system and design to back it up. As we prepare to roll out the first Xet-backed repositories on the Hub, we wrote up a post explaining the nitty gritty details of the decisions that bring this to life https://huggingface.co/blog/from-chunks-to-blocks

Complete with an interactive visualization that shows the power of deduplication in action - taking a 191GB repo to ~97GB and shaving a few hours off upload speeds.

The darker each block in the heatmap, the more we dedupe, the less we have to transfer. Clicking on a file's blocks shows all other files that share blocks.

Check it out and explore for yourself! xet-team/quantization-dedup

David Berenstein PRO

AI & ML interests

Recent Activity

Organizations

David Berenstein PRO

AI & ML interests

Recent Activity

Organizations

davidberenstein1957's activity