Step 12 of 16Day 20 · Sat 14 NovWeek 3Phase A · notebooks

NVIDIA vision helpers

Test whether NVIDIA's vision models can speed up labelling. Cosmos Reason 2 2B checks your labels by looking at the images; NanoOWL draws first-draft boxes from a text prompt.

Runs on: Jetson terminal (servers) + JupyterLab (client)
Notebooks: 62_nvidia_vision.ipynb
Creates: reports/label_check.json, data/raw/detect/prelabels/*.txt, reports/vision_helpers.md
Time: Day 20

Where Where to implement

Cosmos Reason 2 2B

Same llama.cpp build as Step 11
Server on port 8081. On 16 GB it can run next to Nemotron (about 7 GB together); watch jtop.

NanoOWL

Follow the Jetson AI Lab tutorial (container or source install)
Call it from the notebook to write YOLO pre-labels

Do Do it

1Start Cosmos Reason 2 2B

Cosmos Reason 2 is NVIDIA's vision-language model: it takes images or video plus text and reasons about them. The 2B version runs comfortably on your 16 GB, even next to Nemotron.

Jetson terminal

tmux new -s vlm             # a separate window; Nemotron can keep running in "llm"
~/llama.cpp/build/bin/llama-server -hf Kbenkhaled/Cosmos-Reason2-2B-GGUF:Q8_0 -c 8192 \
  --host 127.0.0.1 --port 8081

This is the GGUF build the Jetson AI Lab page points to for smaller Jetsons. -hf downloads it, plus its vision projector, on first run.

Optional on 16 GB: Cosmos Reason 2 8B. Jetson AI Lab lists about 18 GB for its default 8B build, so on your board only try a 4-bit GGUF, with everything else stopped. Compare its label-check agreement with the 2B model before deciding it's worth the memory.

2Ask questions about your images

Notebook cellnotebooks/62_nvidia_vision.ipynb

import base64, json, re
from pathlib import Path
from openai import OpenAI
ROOT = Path.home() / "sutraai"
vlm = OpenAI(base_url="http://127.0.0.1:8081/v1", api_key="not-needed")

def ask_image(path, question):
    b64 = base64.b64encode(Path(path).read_bytes()).decode()
    r = vlm.chat.completions.create(model="cosmos", temperature=0, messages=[{"role": "user", "content": [
        {"type": "text", "text": question},
        {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}}]}])
    text = r.choices[0].message.content
    return re.sub(r"<think>.*?</think>", "", text, flags=re.S).strip()     # drop the reasoning trace if present

img = sorted((ROOT / "data/raw/detect/images").glob("*.jpg"))[0]
print(ask_image(img, "Are this person's eyes open or closed? Answer with one word: open or closed."))

3Check your labels automatically

Disagreements between the model and your labels are where labelling mistakes hide. Review them by hand. Don't overwrite labels automatically.

Notebook cellnotebooks/62_nvidia_vision.ipynb

LAB = ROOT / "data/raw/detect/labels"; CLASSES = ["eye_open", "eye_closed", "yawn", "no_yawn"]
rows = []
for p in sorted((ROOT / "data/raw/detect/images").glob("*.jpg"))[:60]:
    lab = LAB / (p.stem + ".txt")
    ours = {CLASSES[int(l.split()[0])] for l in open(lab) if l.strip()} if lab.exists() else set()
    if not ours & {"eye_open", "eye_closed"}: continue
    says = ask_image(p, "Are this person's eyes open or closed? Answer with one word: open or closed.").lower()
    model_label = "eye_closed" if "closed" in says else "eye_open"
    rows.append({"image": p.name, "ours": sorted(ours), "model": model_label, "agree": model_label in ours})
disagree = [r for r in rows if not r["agree"]]
json.dump(rows, open(ROOT / "reports/label_check.json", "w"), indent=2)
print(f"checked {len(rows)} images, {len(disagree)} disagreements to review")

4NanoOWL: first-draft boxes from text

NanoOWL is NVIDIA's TensorRT-optimized version of the OWL-ViT open-vocabulary detector. On JetPack 6.2.1 the easiest route is the jetson-containers image for L4T r36.4, which already has TensorRT 10.3 and the right PyTorch inside. Follow the Jetson AI Lab NanoOWL tutorial for the full steps (it builds the image-encoder engine):

Jetson terminal

git clone https://github.com/dusty-nv/jetson-containers ~/jetson-containers
bash ~/jetson-containers/install.sh
jetson-containers run -v ~/sutraai:/sutraai $(autotag nanoowl)      # autotag picks the r36.4 image for your L4T 36.4.7

Inside the container, turn its boxes into YOLO pre-labels:

Notebook cellnotebooks/62_nvidia_vision.ipynb (inside the NanoOWL environment)

from PIL import Image
from nanoowl.owl_predictor import OwlPredictor          # API per the NanoOWL README; check it if names differ

predictor = OwlPredictor("google/owlvit-base-patch32", image_encoder_engine="data/owl_image_encoder_patch32.engine")
prompts = ["an open eye", "a closed eye", "an open mouth yawning", "a closed mouth"]   # same order as CLASSES
text_enc = predictor.encode_text(prompts)

out_dir = ROOT / "data/raw/detect/prelabels"; out_dir.mkdir(parents=True, exist_ok=True)
for p in sorted((ROOT / "data/raw/detect/images").glob("*.jpg")):
    image = Image.open(p).convert("RGB"); W, H = image.size
    out = predictor.predict(image=image, text=prompts, text_encodings=text_enc, threshold=0.1)
    lines = []
    for box, label in zip(out.boxes.tolist(), out.labels.tolist()):
        x0, y0, x1, y1 = box
        lines.append(f"{int(label)} {(x0+x1)/2/W:.4f} {(y0+y1)/2/H:.4f} {(x1-x0)/W:.4f} {(y1-y0)/H:.4f}")
    (out_dir / f"{p.stem}.txt").write_text("\n".join(lines))

Import the pre-labels into Label Studio (YOLO import), correct them, and time yourself against labelling from scratch.

5Decide what goes into Month 2

Filereports/vision_helpers.md

| helper | what it did | time per 50 images | errors found / introduced | keep for Month 2? |
|---|---|---|---|---|
| Cosmos Reason 2 2B label check | | | | |
| NanoOWL pre-labels | | | | |

Test Test it

Notebook cellnotebooks/62_nvidia_vision.ipynb (last cell)

assert len(rows) >= 20, "check at least 20 labelled images"
assert (ROOT / "reports/label_check.json").exists()
print(f"✅ label check ran: agreement {sum(r['agree'] for r in rows) / len(rows):.0%}")

Pass: Cosmos answers about real images, disagreements are listed and reviewed by hand, NanoOWL pre-labels are written (or the setup is logged as blocked), and the decision table is filled in.

Done Done when

Cosmos Reason 2 2B served on port 8081; memory checked in jtop
Label check run on 20+ images; disagreements reviewed by hand
NanoOWL set up via the tutorial and pre-labels generated
Pre-labels corrected in Label Studio; time saved measured
vision_helpers.md decision table filled in

If it breaks

Symptom	Fix
Out of memory loading Cosmos	Stop Nemotron or any training kernel; lower `-c` to 4096.
Server ignores the image	The vision projector (mmproj) didn't load: check the server log, update llama.cpp, rerun with `-hf`.
Long answers instead of one word	Keep `temperature=0`; the reasoning trace is stripped by the regex; parse for "open" or "closed".
Container won't start or can't see the GPU	Check `docker info \| grep -i runtime` lists `nvidia`; rerun `install.sh`; let `autotag` pick the image for L4T r36.4 rather than typing a tag.