portable-offline-ai-research-assistant

How to Build a Portable Offline AI Research Assistant Using LLMs and Elasticsearch

Build a Portable Offline AI Assistant for Your PDFs/Ebooks

Learn how to run a local LLM on Windows using llama.cpp, search ebook libraries with Elasticsearch, and build a portable AI research assistant using Python.

Imagine having thousands of PDFs/ebooks on your computer and being able to ask questions about their contents in plain English. Instead of opening books one by one, searching through chapters, and manually collecting information, you could enter a question and receive a consolidated summary based on your own library.

For example, you might ask, "What do my astrology books say about Indira Gandhi's birth chart?" An AI research assistant could search your indexed books, retrieve relevant passages, and prepare a report with source references.

This is possible by combining a local Large Language Model (LLM), Elasticsearch, Python, and a collection of ebooks. With the right configuration, the system can run locally on Windows without sending your research queries or book contents to a cloud AI service.

This guide explains the architecture, the tools required, and the steps involved in creating a portable AI research assistant that can be moved between compatible Windows computers.

What Is a Portable Offline AI Research Assistant?

A portable AI research assistant is an application that combines a locally running language model with a searchable knowledge collection. Unlike a conventional chatbot that relies on a cloud service, a local LLM can generate responses directly on your computer.

The term portable means that the application and its required components can be stored on an external SSD or USB drive and configured to run from different drive letters without hardcoded paths.

The system has three main components:

• Local LLM: Understands questions and generates summaries.

• Elasticsearch: Searches the indexed book collection and retrieves relevant documents.

• Python application: Connects the search engine to the language model and presents the final answer.

Together, these components can form a private research assistant for ebooks, technical documentation, personal reference libraries, and other searchable collections.

Understanding the Technology Stack

1. Local LLM: Qwen3 8B GGUF

The language model is responsible for reading the retrieved information and producing a structured response.

One option is Qwen3 8B in GGUF format, using the Q4_K_M quantization. Quantization reduces the model's memory requirements by representing its numerical values with fewer bits.

The model can be run using llama.cpp, an inference engine designed to execute supported language models on local hardware.

For a CPU-based setup, this avoids requiring a dedicated GPU. The trade-off is that response generation may be slower than GPU-accelerated inference.

Resources:

• llama.cpp: https://github.com/ggml-org/llama.cpp

• Qwen3 8B GGUF: https://huggingface.co/Qwen/Qwen3-8B-GGUF

The Q4_K_M model file is approximately 5 GB, although the exact size depends on the particular model file. The running application also requires memory for the model's working data and context.

A laptop with 16 GB or 32 GB of RAM can provide room to experiment with this configuration, subject to its CPU performance and other running applications.

2. Elasticsearch: Searching Your Book Collection

Elasticsearch is a search and analytics engine that can index documents and retrieve matching results efficiently.

For an ebook research assistant, Elasticsearch can store searchable fields such as book titles, authors, chapter names, extracted text, and other metadata.

Suppose your library contains hundreds of astrology books. A query about Indira Gandhi could search for references to her name, birth details, planetary placements, and related astrological terms.

However, there is an important distinction: an index containing only book metadata cannot search every sentence inside the books. To retrieve information from the actual contents, the text must also be extracted and indexed.

Before connecting an LLM, inspect the existing index mappings and sample documents to determine which fields are available.

Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

3. Python: Connecting Search and AI

Python provides a convenient way to connect Elasticsearch to a local language model.

The application performs four basic tasks:

1. Accepts a question from the user.

2. Searches Elasticsearch for relevant documents or passages.

3. Sends the retrieved information to the local LLM.

4. Displays a summary based on the supplied sources.

The Python application can also include a simple web interface, command-line interface, or desktop interface.

The official Elasticsearch Python client is available here:

https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/index.html

How the System Works: Retrieval-Augmented Generation

This approach is commonly called Retrieval-Augmented Generation, or RAG.

RAG combines information retrieval with language generation. Rather than asking the model to answer a question entirely from its pretrained knowledge, the application retrieves relevant information from a separate knowledge collection and supplies it as context.

Consider the question:

"Summarize all references to Indira Gandhi's birth chart in my astrology library."

The application first searches the indexed books for matching information. It then collects relevant passages, along with available book titles, authors, and page references.

These passages are supplied to the language model with instructions to summarize the material, identify differences between sources, and avoid inventing unsupported details.

The final report might contain sections covering birth details, planetary positions, yogas, dasha periods, and interpretations discussed by the books. These sections should only be populated where the retrieved sources support them.

If the available passages do not establish a birth detail or astrological interpretation, the application should identify that gap instead of presenting an unsupported answer.

Setting Up the Portable Environment

A practical implementation can be developed in stages.

Step 1: Create a dedicated application folder

Create a folder for the AI application on your external drive. For example:

PortableLLM/
    llama.cpp/
    models/
        qwen3-8b-q4_k_m.gguf
    app/
        main.py
    start-model.bat
    start-assistant.bat

The folders separate the model, runtime, application code, and launch scripts.

The actual filenames may vary depending on the downloaded model and llama.cpp release.

Step 2: Download the model and inference engine

Download a compatible Windows CPU build of llama.cpp and extract it into the llama.cpp folder.

Download the selected GGUF model file into the models folder.

The initial downloads require internet access. Once the necessary files are available locally, model inference can run offline.

Step 3: Use relative paths

Portable applications should not assume that an external drive will always have the same drive letter.

In Python, the pathlib module can derive file locations relative to the script:

from pathlib import Path

APP_DIR = Path(__file__).resolve().parent

MODEL_DIR = APP_DIR.parent / "models"
MODEL_PATH = MODEL_DIR / "qwen3-8b-q4_k_m.gguf"

if not MODEL_PATH.exists():
    raise FileNotFoundError(
        f"Model not found: {MODEL_PATH}"
    )

print(f"Using model: {MODEL_PATH}")

This example assumes the script is saved inside the app folder shown above.

The same principle should be applied to configuration files, logs, and any local data directories.

Step 4: Start the local model server

llama.cpp provides a server executable that exposes an HTTP API for local applications.

A batch file can launch the server using the model's path relative to the batch file:

@echo off
setlocal

set "ROOT=%~dp0"
set "MODEL=%ROOT%models\qwen3-8b-q4_k_m.gguf"
set "SERVER=%ROOT%llama.cpp\llama-server.exe"

if not exist "%SERVER%" (
    echo llama-server.exe not found.
    pause
    exit /b 1
)

if not exist "%MODEL%" (
    echo Model file not found.
    pause
    exit /b 1
)

"%SERVER%" -m "%MODEL%" -ngl 0 -c 8192 ^
    -host 127.0.0.1 -port 8080

pause

This example requests CPU inference and binds the server to the local loopback interface. The exact command-line options should be checked against the selected llama.cpp build.

After the server starts, a local Python application can communicate with it through its HTTP API.

Step 5: Connect Python to Elasticsearch

The application can use the Elasticsearch Python client to search an existing index.

A simplified example is shown below:

from elasticsearch import Elasticsearch

es = Elasticsearch("http://localhost:9200")

response = es.search(
    index="your-book-index",
    query={
        "match": {
            "content": "Indira Gandhi birth chart"
        }
    },
    size=10
)

for hit in response["hits"]["hits"]:
    print(hit["_source"])

This is an illustrative example. The index name and content field must be replaced with the actual names found in your Elasticsearch mapping.

If the existing index contains only metadata, the application will need access to the ebook contents and a suitable extraction and indexing process before full-text research will work.

Also, localhost:9200 assumes Elasticsearch is running on the same computer. A fully portable deployment must include a compatible Elasticsearch installation and its data, or connect to an Elasticsearch instance that is already running.

Improving the Quality of Research Results

A basic keyword search is a useful starting point, but comprehensive research requires more than finding a few matching documents.

Search across multiple terms

A query about a birth chart may use different terminology across books. The application can search for the person's name alongside relevant concepts, retrieve the matching passages, and remove duplicates.

Retrieve passages rather than entire books

Sending entire books to the LLM is usually impractical because the model has a finite context window.

Instead, the application should retrieve relevant passages and send a manageable collection to the model. Long research tasks can be processed in batches, followed by a final synthesis of the intermediate results.

Preserve source references

Each retrieved passage should retain its source metadata. Depending on what the index contains, this could include the book title, author, chapter, page number, and document identifier.

The generated report should distinguish direct statements from a book, summaries of several sources, and interpretations produced by the model.

Consider semantic search

Keyword search may miss relevant passages that express the same idea using different words.

Semantic search can help by retrieving passages based on similarity of meaning. It typically requires generating embeddings and storing them in a vector-capable search index.

This can be introduced after basic Elasticsearch retrieval is working. It is not a prerequisite for the initial prototype.

Can the Entire System Run From a USB Drive?

Yes, a portable deployment is possible, but it requires more than copying a model file onto a USB drive.

The external drive may contain the application, model, inference runtime, configuration, and optionally the Elasticsearch installation and index data.

There are practical considerations:

• A USB SSD is generally more suitable than a basic thumb drive for frequent reads and writes.

• Model loading may take longer on slower storage.

• The target computer must have compatible hardware and sufficient RAM.

• Elasticsearch requires its own runtime and data-directory configuration.

• Python dependencies and native libraries must be packaged or otherwise made available on the target machine.

• A portable launcher should detect missing components and report clear errors.

For a simpler first version, the model and Python application can be portable while Elasticsearch runs as a service on the current PC. A fully self-contained deployment can be considered after the search and summarization workflow has been validated.

Security and Privacy Considerations

Running an LLM locally can help keep research material on your computer, provided the application does not send it to external services.

Bind local inference servers to 127.0.0.1 unless network access is explicitly required. Avoid exposing the model or Elasticsearch API to untrusted networks.

If the book library contains copyrighted material, use the system in accordance with the applicable rights and licence terms. A private search and summarization workflow does not automatically grant permission to redistribute entire books or substantial extracts.

A portable offline AI research assistant can turn a large ebook library into a searchable knowledge resource.

By combining a local Qwen3 GGUF model, llama.cpp, Elasticsearch, and Python, it is possible to build a system that searches indexed books and generates source-based summaries without relying on a cloud LLM for inference.

The most important first step is ensuring that the books' actual text is indexed and that search results preserve their source information. Once retrieval is reliable, the local LLM can turn those results into structured research reports.

Start with a small working prototype, validate the quality of its answers against the original books, and then package the components for portable use.

Similar Posts : portable-offline-ai-research-assistant,

See Also:AI Assistant