Mastering the Art of Prompting LLMs for RAG

Previously published on Nuclia.com. Nuclia is now Progress Agentic RAG.

Retrieval Augmented Generation (RAG) has become a cornerstone technology for building powerful, context-aware AI applications. By connecting Large Language Models (LLMs) to external knowledge bases, RAG overcomes the limitations of pre-trained models, reducing hallucinations and providing more accurate, grounded answers. But getting the best results from your RAG pipeline isn’t just about retrieval; it’s also crucially about how you prompt the LLM with the retrieved context.

This post, inspired by a recent session from the Nuclia RAG Academy, delves into the vital role of prompting in RAG and shares expert tips on crafting prompts that unlock the full potential of your system. Whether you’re exploring RAG-as-a-Service platforms or building your own, mastering prompting is key.

Why Prompting is Essential in Your RAG Pipeline

As highlighted in the Nuclia RAG Academy session featuring the Nuclia Lead Machine Learning Engineer and “prompting master,” Carmen Iniesta, prompting sits at a critical juncture in the RAG pipeline. After the user asks a question and the relevant context is retrieved from your data source (like NucliaDB), both the user’s question and the retrieved context are sent to the LLM. The prompt acts as the instruction manual for the LLM, guiding how it should use the provided context to generate the final answer.

Without effective prompting, even the most relevant retrieved information might not be used correctly by the LLM. The key reasons prompting matters in RAG are:

1. Avoiding Hallucinations: This is perhaps the most critical role. A well-crafted prompt explicitly instructs the LLM to only use the information provided in the context, preventing it from fabricating details or drawing upon its general knowledge when the context is insufficient.
2. Generating Custom Outputs: Prompting allows you to define the desired behavior, personality, tone and style of the LLM response. Want a friendly, concise answer or a detailed, formal one? Prompting makes it possible.
3. Setting Up Constraints and Rules: You can use prompts to enforce specific rules, such as prohibiting the mention of certain topics or requiring a particular format (like bullet points or JSON).

Prompting Goes Beyond Just the Final Answer in RAG-as-a-Service

One insightful point from the Nuclia RAG Academy session is that in sophisticated RAG-as-a-Service offerings like the Nuclia platform, prompting opportunities exist at multiple stages of the pipeline, not just when generating the final answer. This multi-point prompting allows for fine-grained control over the entire RAG process:

• LLM System Prompt: Defining the overall role and behavior of the primary LLM.
• Query Rephrasing: Prompting an LLM to rephrase or optimize the user’s initial query for better retrieval results.
• Data Extraction/Indexing: Using visual LLMs with specific prompts to guide the extraction of data from complex document formats (like tables or templates) during the indexing phase.
• Agent Prompts: In systems using AI agents (like the Nuclia Labeler, Generator, etc.), each agent can have its own specific prompt to define its task and behavior within the larger pipeline.

Understanding these different points allows for more powerful customization within your RAG-as-a-Service deployment.

Building Effective Prompts: A Structured Approach

So, how do you write prompts that actually work? The Nuclia RAG Academy session provided a solid framework and valuable advice:

1. Embrace Iteration and Experimentation: Don’t expect perfection on the first try. Prompting is an iterative process of drafting, testing and refining based on the results.
2. Ground Your Prompt: Before writing, be crystal clear about what you want the LLM to achieve. Write down your goals, desired behavior, restrictions and examples.
3. Be Clear, Specific and Concise: LLMs lack human intuition. Use unambiguous language. Formatting like Markdown or XML can help structure instructions.
4. Structure Your Prompt: A well-structured prompt is easier for the LLM to understand. A common structure includes:
   • Intro: Define the LLM role and the task (e.g., “You are a RAG system tasked with answering questions…”).
   • Context: Clearly delineate where the retrieved information is provided.
   • Input Data: Clearly delineate the user’s question.
   • Steps/Rules: Provide specific instructions, constraints or examples.
   • Output/Extra Considerations: Define the desired output format or any final instructions.
5. Provide Guidance on Format and Tone: Explicitly state how you want the answer formatted (e.g., “Answer in bullet points”) and the desired tone (e.g., “Use a friendly and informative tone”).
6. ALWAYS Provide the Context: In a RAG prompt, the retrieved context is paramount. Ensure it’s clearly presented to the model.

Avoiding Common RAG Prompting Mistakes

The session highlighted frequent errors users make:

• Not Clearly Defining Context and Question: Forgetting to wrap the {CONTEXT} and {QUESTION} placeholders (or whatever syntax you use) in clear delimiters means the model doesn’t know which part is the context and which is the question. It will likely ignore the context.
• Not Explicitly Restricting to Context: Simply saying “Answer using the context” is often not enough to prevent hallucinations. You must explicitly state rules like “Only use information explicitly mentioned in the context. Do not add outside knowledge.”

The correct approach involves both clearly marking the context/question and adding explicit instructions to stick only to the provided context.

Expert Tips from the Nuclia Team

A few extra tips shared by Carmen Iniesta:

• Fixed Prompt + Injectable Rules: For complex, customizable tasks (like data extraction), consider having a robust base prompt and allowing users to inject specific rules or parameters to tailor the behavior.
• Use Structured Output (JSON, etc.): If you need predictable, machine-readable output, instruct the LLM to respond in a structured format like JSON.
• Keep Track of Your Prompts: Maintain a record of prompts that worked well and those that didn’t. This builds a valuable knowledge base for future prompt design.
• Don’t Hesitate to Ask for Help: Platforms like ours have experts ready to assist you in optimizing your prompts.

Resources to Get Started

Feeling ready to dive in? The Nuclia RAG Academy recommends checking out prompting guides from leading AI companies like Anthropic and OpenAI, as well as resources like promptingguide.ai. Some companies even offer prompt generator tools that can give you a starting point based on your desired task.

Conclusion: Prompting is Your Lever for RAG Excellence

Prompting is not a trivial add-on for RAG. It’s a fundamental skill that directly impacts the accuracy, reliability and usability of your system. By understanding where and how to apply prompts, following best practices for structure and clarity, and being explicit with your instructions (especially regarding context usage), you can significantly enhance the performance of your RAG applications.

Whether you’re leveraging the power of RAG-as-a-Service or building custom solutions, investing time in mastering prompt engineering for RAG is an investment in better, more trustworthy AI. Explore platforms that offer flexible prompting options at various pipeline stages and keep experimenting, as the Nuclia platform does. Your journey to perfect RAG outputs starts with the right prompt.