CompCode Academy

Lesson 57 of 65 · Free to read

Day 57 ΓÇö Running SLMs locally

Month 3 · Week 12 — Small Language Models & AI Automation with n8n Audience: Final-year students & early-career engineers (0–3 years) Format: Self-paced study + optional live clinic · Suggested time: 4–6 focused hours Key tools: Ollama, LM Studio

Overview

Run SLMs locally with Ollama/LM Studio, understand quantization, serve an API, and route hard tasks to frontier models.

Learning objectives

By the end of this session, you can:

  • Explain the core ideas behind: Running SLMs locally: Ollama / LM Studio, quantization, and serving a local API; routing tasks between an SLM and an LLM.
  • Apply the ideas using Ollama, LM Studio on a realistic task (capstone, internship, or first-job style)
  • Produce the exercise deliverable with clear success criteria
  • Identify common failure modes and how to recover
  • Capture at least one reusable prompt, rule, or checklist for your personal library
  • Decide what must remain human-reviewed for this class of work

Core concepts

Quantization

Smaller weights → faster/cheaper locally with quality trade-offs. Know your RAM budget.

Router pattern

Classifier/SLM first; escalate on low confidence.

Tooling for today

Primary tools: Ollama, LM Studio. Prefer official docs when APIs change. When a model invents a symbol, verify against source or docs immediately.

Practice posture

You are a final-year or early-career engineer pairing with a very fast teammate (the model): clear intent, grounded context, inspect consequential changes, keep a paper trail (commits, prompts, decisions).

Process diagram

Watch & learn

Study these free videos before or during the exercise. Each embed is chosen to reinforce today's objectives.

Course video

Open on YouTube
Ollama Tutorial for Beginners (2026) — Local models, API, and practical workflow.

Course video

Open on YouTube
Learn Ollama in 10 Minutes — Install and run local LLMs quickly.

Worked example / case study

Scenario

Run quantized SLM; expose API; build helper that falls back to frontier model.

Step-by-step

  1. Clarify the outcome in one sentence tied to the exercise below.
  2. Gather grounding context (files, docs, schema, acceptance checks).
  3. Run the primary tool(s): Ollama, LM Studio.
  4. Produce the smallest vertical slice that proves the idea.
  5. Verify with a command, test, screenshot, or side-by-side comparison.
  6. Write down one steering note — what you told the AI when it went wrong.

Sample prompt you can adapt

Role: Senior engineer mentoring an early-career developer (pairing with me).
Task: Running SLMs locally: Ollama / LM Studio, quantization, and serving a local API; routing tasks between an SLM and an LLM.
Context: Final-year / early-career project (capstone, internship, or junior role).
Constraints:
- Match existing project patterns
- No new dependencies unless justified
- Include tests or a verification checklist
- Call out assumptions explicitly
Output: step plan first, then code/files, then verification commands.

Hands-on exercise

Primary task. Run a quantized SLM locally with Ollama, expose it over an API, and build a small private/offline coding helper that falls back to a frontier model for hard tasks.

Coaching notes

Log which model answered each request.

Steps

  1. Set up or open the workspace required by: Ollama, LM Studio.
  2. Skim today's core concepts and watch the video(s) before prompting.
  3. Execute the task in small commits (or saved artifacts if no-code).
  4. Fill the success checklist below.
  5. Note one surprise and one process improvement for tomorrow.

Success criteria

  • Deliverable matches the exercise description
  • You can explain every AI-generated change in plain language
  • Verification ran (tests, manual check, or published URL)
  • At least one pitfall from this lesson was consciously avoided
  • Artifact saved (repo commit, doc, or published link)

Timebox

Suggested: 90–150 minutes of focused build time after study.

Templates & checklists

AssetPurpose
Prompt stubReuse today's structure (role → task → constraints → format)
Verification checklistTests / manual QA / security skim
Decision logWhat you delegated vs kept human

Copy-paste prompt skeleton

## Goal
Running SLMs locally: Ollama / LM Studio, quantization, and serving a local API; routing tasks between an SLM and an LLM.

## Context files
- (list paths)

## Constraints
- Stack: (fill)
- Do not: (secrets, drive-by refactors)

## Definition of done
- Run a quantized SLM locally with Ollama, expose it over an API, and build a small private/offline coding helper that falls back to a frontier model for hard tasks.

Daily reflection (3 minutes)

  • What did AI do well?
  • Where did it confidently fail?
  • What rule will you add to your cookbook/rules file?

Common pitfalls

  • Models too big for your machine thrashing disk
  • No fallback path
  • Exposing local API to the LAN unintentionally

Key takeaways

  • Local is a superpower for private code
  • Routing saves money
  • Ops matters even on localhost
  • Tools (Ollama, LM Studio) change; the judgment skills transfer.

Resources

  • Official docs for: Ollama, LM Studio
  • Course cookbook: keep prompts in a versioned Markdown file in your repo
  • Prior days this week — skim objectives if you skipped ahead

Self-paced lesson — Mastering AI-Assisted Development & Vibe Coding — Day 57 of 65.

New free lessons and course updates. Unsubscribe from any email.