Update README.md with tool-use support for huggingface and vLLM

README.md

@@ -78,6 +78,102 @@ completion = client.chat.completions.create(
 )
 ```
 
+## With Tool-use
+To enable tool-use, load, tokenize, and generate as the examples above, but additionally define a set of tools and pass them to the chat template.
+
+**Using transformers**
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("llm360/k2-v2-instruct", device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained("llm360/k2-v2-instruct")
+
+prompt = "Explain why the derivative of sin(x) is cos(x)."
+messages = [
+    {"role": "system", "content": "You are K2, a helpful assistant created by Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) Institute of Foundation Models (IFM)."},
+    {"role": "user", "content": prompt}
+]
+
+# Define list of tools
+tools = [
+    {
+        "type": "function",
+        "name": "get_horoscope",
+        "description": "Get today's horoscope for an astrological sign.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "sign": {
+                    "type": "string",
+                    "description": "An astrological sign like Taurus or Aquarius",
+                },
+            },
+            "required": ["sign"],
+        },
+    },
+]
+
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+    reasoning_effort="high" # Or "medium"/"low"
+    tools=tools
+)
+
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=200)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+**Using vLLM**
+Serve the model with tool-use enabled and parser set.
+
+```
+vllm serve LLM360/K2-V2-Instruct --enable-auto-tool-choice --tool-call-parser hermes --tensor-parallel-size 8 --port 8000 
+```
+
+```
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="key"
+)
+
+# Define list of tools
+tools = [
+    {
+        "type": "function",
+        "name": "get_horoscope",
+        "description": "Get today's horoscope for an astrological sign.",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "sign": {
+                    "type": "string",
+                    "description": "An astrological sign like Taurus or Aquarius",
+                },
+            },
+            "required": ["sign"],
+        },
+    },
+]
+
+completion = client.chat.completions.create(
+    model="LLM360/K2-V2-Instruct",
+    messages = [
+        {"role": "system", "content": "You are K2, a helpful assistant created by Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) Institute of Foundation Models (IFM)."},
+        {"role": "user", "content": "Explain why the derivative of sin(x) is cos(x)."}
+    ],
+    extra_body={
+        "chat_template_kwargs": {"reasoning_effort": "high"},
+    },
+    tools=tools,
+)
+```
+
+
 ---
 
 ## **Evaluation Summary**