Offline NanoChat: Running a Local AI Assistant on the Jetson Orin Nano

A step-by-step build log for deploying a fully offline chat assistant on the A608 carrier board, using Ollama, TinyLlama, and a lightweight Flask front end.

Introduction

Most "AI assistant" demos quietly depend on a cloud API somewhere in the background. This build strips that dependency away entirely. NanoChat is a small, self-contained chat application that runs a quantized language model directly on a Jetson Orin Nano mounted on an A608 carrier board — no internet connection required, even while the model is answering questions. This post walks through the full setup, from verifying the board to disconnecting the network and confirming the assistant still responds.

A608 Carrier Board with Jetson Orin Nano: The Hardware Behind NanoChat

The Jetson Orin Nano is a compact AI computing module from NVIDIA, built for edge AI, robotics, and computer vision workloads. It packs an ARM CPU alongside an NVIDIA Ampere-architecture GPU into a small, low-power form factor, giving it enough compute to run quantized language models and vision workloads locally, without relying on a cloud server. It's commonly used in robots, drones, and embedded AI devices where power budget and physical space are limited but on-device intelligence is still required.

The A608 is a carrier board designed to host the Jetson Orin NX/Orin Nano module and expose its interfaces for real-world use. On its own, the Jetson module needs a carrier board to actually be usable — it provides the power input, networking, storage, and I/O connectors around the compute module. The A608 specifically offers two Gigabit Ethernet ports, M.2 slots for SSD storage and WiFi/4G/5G connectivity, multiple USB 3.2 ports, CSI camera connectors, and function connectors like CAN, I2C, SPI, and UART — making it well suited for robotics, drone, and edge AI video analytics projects where rich connectivity is needed alongside compact size.

Together, the Jetson Orin Nano module and the A608 carrier board form the hardware base for this project — compact and power-efficient enough to run entirely offline, while still offering the I/O needed to expand into robotics or sensor-driven use cases later.

Ollama, TinyLlama & Flask: The Software Behind NanoChat

NanoChat is built around three layers working together on a single embedded board. Ollama handles model serving and inference locally, so no round trip to an external API is ever made. TinyLlama is the language model itself — small enough to run comfortably within the Orin Nano's memory and compute budget while still producing coherent, useful answers. A thin Flask web application sits on top, giving the whole thing a simple browser-based chat interface at 127.0.0.1:7860.

The entire pipeline — model weights, runtime, and web server — lives on the device itself. That's what makes it possible to unplug the network cable partway through testing and keep chatting without interruption.

Step-by-Step Implementation

The implementation follows ten sequential steps, moving from environment verification through to the final offline test.

Step 1: Verify the Jetson environment

Before installing anything, confirm the Jetson environment is healthy — JetPack version, available storage, and that the board boots cleanly on the A608 carrier.

# Check Jetson / JetPack environment
jetson_release

Step 2: Install the Ollama runtime

Install Ollama, which will handle downloading and serving the language model locally on the device.

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

Step 3: Download the TinyLlama model

With Ollama installed, pull the TinyLlama model. This is the last step that requires an internet connection — once the weights are cached locally, the model is available offline from that point forward.

# Pull the TinyLlama model through Ollama
ollama pull tinyllama

Step 4: Create a project workspace

Create a dedicated project folder to keep the NanoChat application and its files organized and separate from other projects on the board.

# Create the project workspace
mkdir nanochat && cd nanochat

Step 5: Create a Python virtual environment

Set up an isolated Python virtual environment so NanoChat's dependencies stay separate from the system Python installation.

# Create and activate a virtual environment
python3 -m venv venv
source venv/bin/activate

Step 6: Install all required packages

With the virtual environment active, install the Python packages the Flask app needs to run and communicate with the local Ollama server.

# Install required packages
pip install flask requests

Step 7: Create the NanoChat Flask application

The Flask app is intentionally minimal: a single input box, a send button, and a route that forwards each question to the local Ollama server running TinyLlama, then renders the reply back into the page.

from flask import Flask, request, render_template_string
import requests

app = Flask(__name__)
history = []

# Point at the local Ollama server — no external host involved
OLLAMA_URL = "http://localhost:11434/api/generate"

@app.route("/", methods=["GET", "POST"])
def chat():
    if request.method == "POST":
        question = request.form["question"]
        response = requests.post(OLLAMA_URL, json={
            "model": "tinyllama",
            "prompt": question,
            "stream": False
        })
        answer = response.json()["response"]
        history.append((question, answer))
    return render_template_string(PAGE, history=history)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=7860)

Step 8: Run the NanoChat application

Start the Flask app from inside the activated virtual environment. This launches the local web server on port 7860.

# Run the NanoChat application
python3 app.py

Step 9: Open NanoChat in the browser and start a chat

With the server running, open 127.0.0.1:7860 in the on-device browser to load the NanoChat interface and begin chatting with the local TinyLlama model.

# Open in the browser
xdg-open http://127.0.0.1:7860

Step 10: Disconnect the internet connection and try the chat

The final and most important step is the offline test: disconnect the network entirely and send another message. If TinyLlama keeps replying, the assistant is genuinely offline — not just idle while quietly reaching out to a remote API.

Why It Works Offline

• Model weights (TinyLlama) are pulled once and cached locally by Ollama — no repeated downloads.
• Inference runs entirely through the local Ollama server on localhost:11434, never leaving the device.
• The Flask front end and the model server both run on the same board, so there's no external hop for the UI either.
• Once the venv and packages are installed, the whole stack starts from local disk with no package-manager calls at runtime.

Interface & Sample Output

The browser page below shows the NanoChat interface right after the Flask app starts — a plain "Offline AI Assistant" title, an input box, and a Send button, served locally at 127.0.0.1:7860.

Two example exchanges confirm the assistant responds sensibly to technical questions — one about UART serial communication, and one comparing Jetson and Orin Nano platforms — both answered entirely by the local TinyLlama model.

Conclusion

Ten steps is all it takes to turn a Jetson Orin Nano on an A608 carrier board into a self-contained AI assistant. The build stays deliberately simple — Ollama for serving, TinyLlama for the model, Flask for the UI — but the payoff is real: an assistant that keeps working when the network doesn't. For edge deployments where connectivity can't be guaranteed, that's the whole point.