Step 14 of 16Day 22 · Mon 16 NovWeek 4Phase B · web
Notebooks to an ml_core package
Turn sutra_core into an installable package with a command-line entry point and tests. FastAPI will call this package: no notebook code gets copied into the web app.
Where Where to implement
Package
~/sutraai/packages/ml_core/src/ml_core/- Move (don't rewrite) the files from
sutra_core/
Tests
packages/ml_core/tests/- Golden MFCC vectors + a tiny smoke dataset
Do Do it
1Create the layout and move the code
cd ~/sutraai
mkdir -p packages/ml_core/src/ml_core packages/ml_core/tests
git mv sutra_core/*.py packages/ml_core/src/ml_core/
ls packages/ml_core/src/ml_core/ # __init__.py data.py dsp.py models.py train.py export.py profile.py device_sim.py detect.py run.py[project]
name = "ml-core"
version = "0.1.0"
description = "SutraAI pipeline engine: input → DSP → learning → export"
requires-python = ">=3.10,<3.11" # matches the cp310 Jetson wheels on JetPack 6.2.1
# torch 2.8.0 / torchvision 0.23.0 / onnxruntime-gpu 1.23.0 are NOT listed: they come from the Jetson
# jp6/cu126 index (Step 02) and stay pinned by constraints-jp621.txt (PIP_CONSTRAINT in the venv)
dependencies = ["numpy==1.26.4", "pyyaml", "timm", "onnx", "scikit-learn", "librosa", "pandas", "thop", "ai-edge-litert"]
[project.scripts]
sutra-run = "ml_core.run:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/ml_core"]2A proper command-line entry point
The web worker will run this command and read its JSON lines, the same lines you printed in every training loop.
import argparse
def main():
p = argparse.ArgumentParser(prog="sutra-run", description="Train + export one SutraAI pipeline")
p.add_argument("config", help="path to pipeline.yaml")
p.add_argument("--data", help="dataset folder (overrides 'dataset' in the YAML)")
p.add_argument("--out", help="folder for models/, reports/, exports/", default=None)
a = p.parse_args()
cfg_path = a.config
if a.data: # write a copy of the config pointing at the uploaded data
cfg = yaml.safe_load(open(a.config)); cfg["dataset"] = str(Path(a.data).resolve())
cfg_path = Path(a.out or ".") / "pipeline.resolved.yaml"
Path(cfg_path).parent.mkdir(parents=True, exist_ok=True); yaml.safe_dump(cfg, open(cfg_path, "w"))
run(cfg_path, a.out)
if __name__ == "__main__":
main()In load_timeseries, use Path(cfg["dataset"]) directly when it's an absolute path, and only prefix ROOT when it's relative: root = Path(cfg["dataset"]); root = root if root.is_absolute() else ROOT / root.
3Install in editable mode and update imports
cd ~/sutraai && source .venv/bin/activate
pip install -e packages/ml_core
grep -rl "sutra_core" notebooks packages | xargs sed -i 's/sutra_core/ml_core/g'
sutra-run --help4Tests that protect what you proved
import glob, os, numpy as np
from ml_core.dsp import mfcc
GOLD = os.path.expanduser("~/sutraai/exports/golden")
def test_mfcc_matches_golden_vectors():
files = glob.glob(f"{GOLD}/mfcc_*.npz")
assert files, "no golden vectors found: run notebook 31 first"
for f in files:
d = np.load(f)
np.testing.assert_allclose(mfcc(d["x"]), d["y"], rtol=1e-5, atol=1e-5)import json, subprocess, sys, os
def test_audio_pipeline_runs_end_to_end(tmp_path):
cfg = os.path.expanduser("~/sutraai/configs/audio.yaml")
out = subprocess.run([sys.executable, "-m", "ml_core.run", cfg, "--out", str(tmp_path)],
capture_output=True, text=True, timeout=3600)
assert out.returncode == 0, out.stderr[-2000:]
done = [json.loads(l) for l in out.stdout.splitlines() if l.startswith("{") and '"event": "done"' in l]
assert done and done[-1]["test_acc"] > 0.5pip install pytest && cd ~/sutraai && pytest packages/ml_core/tests -qTest Test it
cd ~/sutraai
sutra-run configs/audio.yaml --out /tmp/sutra_try | tail -n 3 # last line: {"event": "done", ...}
pytest packages/ml_core/tests -q # all greenPass:
sutra-run reproduces the NB90 audio score (± about 1 point), prints one JSON line per epoch plus a final "done" line, and the tests pass.Done Done when
- packages/ml_core installed with
pip install -e - Notebooks import
ml_coreand still pass Restart and Run All sutra-run CONFIG --data DIR --out DIRworks- Golden-vector test and CLI smoke test pass
- Committed
If it breaks
| Symptom | Fix |
|---|---|
| pip tries to install a CPU torch | torch must not be in dependencies, and echo $PIP_CONSTRAINT must print the constraints-jp621.txt path (reactivate the venv if it's empty). |
ModuleNotFoundError: ml_core in a notebook | Restart the kernel after pip install -e; remove old sys.path.insert lines. |
| Golden test fails | Someone changed mfcc(). That's exactly what the test is for: revert, or regenerate the vectors on purpose and note why. |