Building AI Knowledge Agents for Slack: A Guide to Efficient RAG Systems

Summary: Retrieval-Augmented Generation (RAG) systems are becoming increasingly popular for creating internal knowledge bots within companies, helping employees quickly find answers from diverse documents and resources. This article provides insights into building such a bot, focusing on the tools, techniques, and architecture involved, as well as cost considerations and time investment. What is RAG and Agentic RAG? RAG combines information retrieval and generative capabilities to fetch relevant data from multiple documents and use it to enhance the AI's response to user queries. This ensures accurate and contextually appropriate answers. Agentic RAG takes this a step further by allowing the large language model (LLM) to actively decide where to retrieve information from, improving the bot's adaptability and efficiency. Tools like LlamaIndex facilitate this process, though simpler setups are viable for basic needs. Technical Stack and Deployment Options The system involves using event-driven architecture with serverless functions to minimize costs. Two popular choices are AWS Lambda and Modal. Modal, while newer and offering cheaper CPU pricing, has shown some instability on the free tier. For storing and retrieving embeddings, vector databases such as Weaviate, Milvus, pgvector, Redis, and Qdrant are used. Qdrant and Milvus provide generous free tiers, making them cost-effective for small-scale implementations. Cost and Time to Build Building a RAG bot involves engineering hours, cloud costs, embedding costs, and LLM API calls. Setting up a minimal framework is relatively quick, but the majority of time is spent on data preprocessing, including chunking documents and attaching metadata. Cloud costs are minimal due to serverless functions, but vector databases can become costly with larger datasets. Embedding costs are generally low, with OpenAI's text-embedding-3-small being a budget-friendly option. LLM API calls, especially with more advanced models, can significantly impact costs. Using cheaper models like GPT-4-mini or Gemini Flash 2.0 keeps the monthly budget under $50 for moderate usage. Document Chunking and Metadata Chunking documents correctly is crucial to maintain context and improve retrieval accuracy. Poorly structured data can lead to irrelevant results. The author uses Docling for PDF chunking and a custom web crawler for web pages, attaching metadata like URLs, headings, and page numbers to each chunk. Summarizing content with an LLM and assigning higher authority to summaries helps prioritize relevant information during retrieval. Agent Architecture and User Experience The agent system connects to various tools, each containing different types of data from the vector database. The author prefers a single agent to simplify management. A preliminary LLM function assesses the need to run the agent, enhancing user experience by reducing initial wait times. LlamaIndex's FunctionAgent is used to set up the system, with tools like onboarding_tool, public_docs_tool, and access_links_tool providing specific data sources. The system sends updates back to Slack to keep users informed of the agent's progress, using an event stream to communicate each step. Retrieval Techniques Hybrid search techniques, combining dense and sparse vector models, improve the bot's ability to handle both exact and fuzzy searches. Dense vectors excel at finding similar content, while sparse vectors are precise for keyword-based searches. Deduplication and re-ranking further refine the results, filtering out irrelevant chunks before the LLM generates a response. These techniques are straightforward to implement but add complexity and can introduce additional latency. Key Focus Areas Despite the availability of sophisticated tools and techniques, most of the time and effort are spent on: 1. Prompting: Crafting effective system prompts for the LLM to ensure accurate and appropriate responses. 2. Reducing Latency: Optimizing the system to respond within 8 to 13 seconds, typical for corporate tools. Cold starts and LLM latency are significant challenges. 3. Document Ingestion: Programmatically ingesting and chunking documents, especially when dealing with unstructured data. This requires careful consideration to maintain context and relevance. Advanced Features and Future Improvements To further enhance the bot, features like caching, continuous data updates, and long-term memory can be implemented. Caching query embeddings speeds up retrieval, while periodic re-embedding strategies update the database with new or changed information. Long-term memory, facilitated by fetching Slack conversation history, helps the agent maintain context over multiple interactions. However, caution is advised to prevent overwhelming the LLM with unnecessary context. Evaluations and Industry Insights Industry insiders recommend using frameworks for rapid prototyping but eventually rewriting core logic to minimize LLM calls and reduce overhead. This ensures better control over latency and cost. While frameworks like LlamaIndex offer valuable abstractions, they can sometimes oversimplify user queries, leading to less contextual responses. Evaluating, monitoring, and implementing guardrails are also essential for maintaining the bot's reliability and accuracy. The author, an experienced developer in the AI field, is keen to explore more advanced topics like agentic memory, evaluation, and sophisticated prompting in future articles. For those interested in following his work, he can be found on his website and LinkedIn.