RAG-Enhanced LLM Queries

RAG in Menatic AI is a two-step process: retrieve relevant chunks with /api/rag/search, then inject them as context when calling the chat completions API.

Step 1 — Search for Relevant Chunks

curl -X POST https://<domain>/api/rag/search \
  -H "X-API-Key: <project-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How do I configure GPU scheduling?",
    "limit": 5
  }'

With a user JWT, include project_id:

curl -X POST https://<domain>/api/rag/search \
  -H "Authorization: Bearer <jwt>" \
  -H "Content-Type: application/json" \
  -d '{
    "project_id": "<uuid>",
    "query": "How do I configure GPU scheduling?",
    "limit": 5
  }'

Parameters

Field	Required	Default	Description
query	Yes	—	The question or topic to search for
project_id	Conditional	—	Required when using a user JWT; inferred from project token
limit	No	5	Number of chunks to return (minimum 1)

Response

{
  "results": [
    {
      "id": "point-uuid",
      "score": 0.91,
      "text": "GPU scheduling is configured per-notebook by selecting a GPU count in the resource preset. The cluster must have nodes labeled with nvidia.com/gpu...",
      "document_id": "3f8a1b2c-...",
      "chunk_index": 4,
      "name": "gpu-guide.pdf"
    },
    {
      "id": "point-uuid-2",
      "score": 0.84,
      "text": "To allocate a GPU to a batch job, set the gpu field in the resource configuration when submitting the job...",
      "document_id": "3f8a1b2c-...",
      "chunk_index": 7,
      "name": "gpu-guide.pdf"
    }
  ]
}

score is the cosine similarity between the query embedding and the chunk embedding (0 to 1 — higher is more relevant).

---

Step 2 — Inject Context and Call the LLM

Take the retrieved text fields and build a system message that instructs the model to answer using only the provided context.

curl -X POST https://<domain>/api/v1/chat/completions \
  -H "X-API-Key: <project-token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:8b",
    "messages": [
      {
        "role": "system",
        "content": "Answer the user question using only the context below. If the answer is not in the context, say you do not know.\n\n--- CONTEXT ---\nGPU scheduling is configured per-notebook by selecting a GPU count in the resource preset...\nTo allocate a GPU to a batch job, set the gpu field in the resource configuration..."
      },
      {
        "role": "user",
        "content": "How do I configure GPU scheduling?"
      }
    ]
  }'

---

Complete Example (Python, Jupyter Notebook)

This is the recommended pattern for RAG in notebooks, combining both steps with the pre-injected environment variables:

import os
import requests
from openai import OpenAI

base_url = os.environ["FTN_BASE_URL"]
api_key  = os.environ["FTN_API_KEY"]
headers  = {"X-API-Key": api_key, "Content-Type": "application/json"}

def rag_query(question: str, model: str = "llama3:8b", top_k: int = 5) -> str:
    # Step 1: retrieve relevant chunks
    search_resp = requests.post(
        f"{base_url}/api/rag/search",
        headers=headers,
        json={"query": question, "limit": top_k},
    )
    search_resp.raise_for_status()
    results = search_resp.json().get("results", [])

    if not results:
        context = "No relevant documents found."
    else:
        context = "\n\n".join(
            f"[{r['name']}, chunk {r['chunk_index']}]\n{r['text']}"
            for r in results
        )

    # Step 2: call the LLM with retrieved context
    client = OpenAI(
        base_url=f"{base_url}/api/v1",
        api_key=api_key,
    )
    response = client.chat.completions.create(
        model=model,
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a helpful assistant. Answer the user's question using "
                    "only the context below. If the answer is not present in the "
                    "context, say you do not know.\n\n"
                    f"--- CONTEXT ---\n{context}"
                ),
            },
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content


# Usage
answer = rag_query("How do I configure GPU scheduling?")
print(answer)

---

Streaming RAG Responses

Pass stream=True to get a streaming response:

response = client.chat.completions.create(
    model="llama3:8b",
    messages=[
        {"role": "system", "content": f"Use this context:\n\n{context}"},
        {"role": "user",   "content": question},
    ],
    stream=True,
)

for chunk in response:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

---

Tuning Retrieval

Consideration	Recommendation
Relevance threshold	Filter results by score — discard chunks below ~0.6 to avoid injecting irrelevant context
Context length	Keep total context under ~3 000 tokens for smaller models. Use fewer chunks or truncate long ones.
limit	Start with 3–5 chunks. Increase if answers are incomplete; decrease if the model gets confused by too much context.
Query wording	The search query should be the question, not keywords. Full sentences embed better.

# Example: filter by score threshold
results = [r for r in results if r["score"] >= 0.65]

---

Semantic Search Without LLM

/api/rag/search is also useful on its own — for example, to power a document search UI or to check whether a topic is covered in your knowledge base before deciding whether to call the LLM.

results = requests.post(
    f"{base_url}/api/rag/search",
    headers=headers,
    json={"query": "VLAN configuration limits", "limit": 3},
).json()["results"]

for r in results:
    print(f"{r['score']:.2f}  [{r['name']}]  {r['text'][:120]}...")