Getting Started with LattePanda Mu Ultra (your first LLM)

Embedded Linux Guides
1

This is a page about an out of box experience with the LattePanda Mu Ultra and setting up an LLM

Docs:

Availability

Kits (These include Mu Ultra Compute Module) :

Power Supplies:

Basic Requirements

  • HDMI Montior
  • USB Keyboard/Mouse
  • Network
  • NVME Storage
  • USB Flash drive with Debian debian-13.7.0-amd64-netinst.iso

image
image1402×908 257 KB

image
image1920×1078 428 KB

image
image1920×1306 460 KB

First Boot (BIOS (DEL key gets into bios)

image
image1402×908 257 KB

Check bios version: (2026/04/30 is the latest today: LattePanda-Mu-Ultra/Softwares/BIOS/DFLT at main · LattePandaTeam/LattePanda-Mu-Ultra · GitHub)

image
image2492×378 94.9 KB

I’m setting this up in a server rack, so enable AC Power Loss: (Power On) so i never have to hit the power button:

image
image2182×708 193 KB

Secure boot is disabled (by default), I’ll be running Linux.

image
image2158×678 125 KB

Make sure to Save Changes and Reset:

image
image2450×536 177 KB

Bios force Debian (UEFI) boot:

image
image1920×714 117 KB

Proxmox PCI passthru:

cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=8086:64a0,8086:643e
softdep xe pre: vfio-pci
softdep intel_vpu pre: vfio-pci

lspci -nnk

00:02.0 VGA compatible controller [0300]: Intel Corporation Lunar Lake [Intel Arc Graphics 130V / 140V] [8086:64a0] (rev 04)
        DeviceName: Onboard - Video
        Subsystem: Intel Corporation Device [8086:0000]
        Kernel driver in use: vfio-pci
        Kernel modules: xe

00:0b.0 Processing accelerators [1200]: Intel Corporation Lunar Lake NPU [8086:643e] (rev 04)
        DeviceName: Onboard - Other
        Subsystem: Intel Corporation Device [8086:8086]
        Kernel driver in use: vfio-pci
        Kernel modules: intel_vpu

Booting Debian 13:

image
image1920×1080 89.2 KB

Debian Package set, keeping it light with SSH server / standard system utilities:

image
image1920×1080 136 KB

First boot:

image
image1920×1080 149 KB

Install common tools and docker:

sudo apt update ;\
sudo apt-get install btop docker-compose docker.io \
local-apt-repository nvtop pipx tree

Configure docker, video and render permissions

Add your user to docker groups:

sudo usermod -aG docker $USER

Add your user to video and render groups

sudo usermod -aG video,render $USER

Install huggingface_hub[cli] so we can download models from huggingface

pipx install huggingface_hub[cli] ;\
pipx ensurepath ;\
source ~/.bashrc

and reboot

sudo reboot

llama.cpp (vulkan) on GPU

Download model:

hf download ggml-org/gemma-4-E2B-it-GGUF \
--include "*Q4_0*" \
--local-dir ~/llama-cpp/models/gemma-4-E2B-it-GGUF

Configure .env

MODELS_DIR=./models

# GPU Vulkan Settings
GPU_VULKAN_MODEL=gemma-4-E2B-it-GGUF/gemma-4-E2B-it-Q4_0.gguf
GPU_VULKAN_PORT=8080

Configure docker-compose.yml

services:
  # --- OPTION 1: VULKAN (GPU) ---
  llama-gpu-vulkan:
    image: ghcr.io/ggml-org/llama.cpp:server-vulkan
    container_name: llama-gpu-vulkan
    profiles: ["gpu-vulkan"]
    restart: unless-stopped
    ports: ["${GPU_VULKAN_PORT}:8080"]
    devices: ["/dev/dri/renderD128:/dev/dri/renderD128", "/dev/dri/card0:/dev/dri/card0"]
    volumes: ["${MODELS_DIR}:/models:ro"]
    environment:
      - LLAMA_ARG_MODEL=/models/${GPU_VULKAN_MODEL}
      - LLAMA_ARG_N_GPU_LAYERS=99
      - LLAMA_ARG_CTX_SIZE=4096
      - LLAMA_ARG_HOST=0.0.0.0
      - LLAMA_ARG_PORT=8080

Review

voodoo@lattepanda-mu-ultra-226v:~$ tree -a llama-cpp/
llama-cpp/
├── docker-compose.yml
├── .env
└── models
    └── gemma-4-E2B-it-GGUF
        ├── .cache
        │   └── huggingface
        │       ├── CACHEDIR.TAG
        │       ├── download
        │       │   ├── gemma-4-E2B-it-Q4_0.gguf.lock
        │       │   ├── gemma-4-E2B-it-Q4_0.gguf.metadata
        │       │   ├── mtp-gemma-4-E2B-it-Q4_0.gguf.lock
        │       │   └── mtp-gemma-4-E2B-it-Q4_0.gguf.metadata
        │       ├── .gitignore
        │       └── trees
        │           └── b4243c156154b6dca9324415f8c7ccc098b4aed1.json
        ├── gemma-4-E2B-it-Q4_0.gguf
        └── mtp-gemma-4-E2B-it-Q4_0.gguf

7 directories, 11 files

Start llama-gpu-vulkan

docker compose --profile gpu-vulkan up

image
image2070×1356 236 KB

Stop llama-gpu-vulkan

docker compose down

NPU (currently broken with llama-server)

Configure NPU for openvino

install compute-runtime

Grab latest: Releases · intel/compute-runtime · GitHub

For example: Releases · intel/compute-runtime · GitHub

mkdir neo ; \
cd neo ;\
wget https://github.com/intel/intel-graphics-compiler/releases/download/v2.41.5/intel-igc-core-2_2.41.5+22716_amd64.deb ;\
wget https://github.com/intel/intel-graphics-compiler/releases/download/v2.41.5/intel-igc-opencl-2_2.41.5+22716_amd64.deb ;\
wget https://github.com/intel/compute-runtime/releases/download/26.35.39758.10/intel-ocloc_26.35.39758.10-0_amd64.deb ;\
wget https://github.com/intel/compute-runtime/releases/download/26.35.39758.10/intel-opencl-icd_26.35.39758.10-0_amd64.deb ;\
wget https://github.com/intel/compute-runtime/releases/download/26.35.39758.10/libigdgmm12_22.10.0_amd64.deb ;\
wget https://github.com/intel/compute-runtime/releases/download/26.35.39758.10/libze-intel-gpu1_26.35.39758.10-0_amd64.deb

Copy *.deb files to local-apt-repository

sudo mkdir -p /srv/local-apt-repository ;\
sudo cp -v *.deb /srv/local-apt-repository

Then install

sudo apt update
sudo apt-get install intel-igc-core-2 \
intel-igc-opencl-2 intel-ocloc intel-opencl-icd \
libigdgmm12 libze-intel-gpu1 ocl-icd-libopencl1

Build linux-npu-driver

sudo apt update ;\
sudo apt-get install build-essential ccache \
cmake git git-lfs libtbb-dev libyaml-cpp-dev \
ocl-icd-opencl-dev opencl-headers python3

Clone linux-npu-driver

git clone https://github.com/intel/linux-npu-driver ;\
cd linux-npu-driver ;\
git submodule update --init --recursive

Build npu user-space

cmake -B build -S .
cmake --build build --parallel $(nproc)

Install the driver in the system

sudo cmake --install build

Reload the intel_vpu module to load new firmware

sudo rmmod intel_vpu ;\
sudo modprobe intel_vpu