Local model chore: read a photo with a vision model, on-device

The Stack

Tested Against

Prerequisites

• A laptop with ~8GB RAM (gemma3:4b is about 3.3GB)

Steps

1. 1

   Read a fixture image on a real vision model (offline)

   Install Ollama, pull the vision-capable gemma3:4b, build a small receipt image, and ask the model to read the total via the local API. After the download this runs offline. CI runs exactly this and checks the vision model returned a real answer (the number it reads is fenced).

   command -v ollama >/dev/null 2>&1 || curl -fsSL https://ollama.com/install.sh | sh
   curl -s http://localhost:11434/api/version >/dev/null 2>&1 || (ollama serve >/tmp/ollama-serve.log 2>&1 &)
   for i in $(seq 1 60); do curl -s http://localhost:11434/api/version >/dev/null 2>&1 && break; sleep 1; done
   ollama pull gemma3:4b >/dev/null 2>&1
   python3 -m venv .venv
   .venv/bin/pip install -q pillow
   .venv/bin/python - <<'EOF'
   import base64, json, urllib.request
   from PIL import Image, ImageDraw
   
   img = Image.new("RGB", (480, 220), "white")
   d = ImageDraw.Draw(img)
   for pos, line in [((20, 30), "CAFE RECEIPT"), ((20, 90), "Latte      4.50"),
                     ((20, 120), "Muffin     3.25"), ((20, 170), "TOTAL      7.75")]:
       d.text(pos, line, fill="black")
   img.save("receipt.png")
   
   b64 = base64.b64encode(open("receipt.png", "rb").read()).decode()
   payload = json.dumps({
       "model": "gemma3:4b",
       "prompt": "Read this receipt. What is the total amount?",
       "images": [b64],
       "stream": False,
   }).encode()
   req = urllib.request.Request("http://localhost:11434/api/generate", data=payload,
                               headers={"Content-Type": "application/json"})
   out = (json.load(urllib.request.urlopen(req, timeout=720)).get("response") or "").strip()
   assert out, "vision model returned an empty response"
   print(f"chore4 OK: gemma3:4b read the image and returned {len(out)} chars offline (quality fenced)")
   EOF

2. 2

   Point it at your own photos (the quality step, not checked by CI)

   In LM Studio or Open WebUI with a vision model, attach a real receipt, label, or handwritten note and ask 'what is the total' or 'type out this text.' What it reads is non-deterministic, so CI never claims the specific value.