The constraint. Atacana needed a chatbot that could answer complex questions over thousands of proprietary medical news reports and drug discovery updates added every day. Cloud-based LLMs were not an option because of confidentiality requirements, so the system had to run completely offline on local servers.
What we built. A retrieval-augmented generation (RAG) system on a Mixtral 8x7B model with 8-bit quantization, which reduced the memory footprint and processing time. LangChain orchestrates retrieval from a vector database that ingests the new reports daily.
The result. Cited answers in 5 to 7 seconds on local hardware, with zero data exposure to the public internet.