SpeakLab

Explain, learn, explain again

A session starts with a technical prompt. The first recording is an explanation off the top of your head, with reference material hidden. You work with what you know, including the parts you struggle to put into words.

SpeakLab reviews the explanation against the topic's reference concepts, separating what you explained accurately, partly explained, missed, or got wrong. It also offers communication feedback on structure, clarity, examples, and transitions, tied to passages from the transcript.

Those gaps shape the learning session. Short reference sections focus on the concepts that need attention, and a local AI tutor can answer questions using the references and your first attempt. Then the notes disappear and you record again, explaining the topic in your own words while trying to apply the feedback on both content and delivery.

Both recordings, transcripts, and assessments remain available for comparison. Review dates bring the topic back for another unprepared explanation later, so practice extends beyond the immediate second attempt.

Hearing how you communicate

Alongside the content review, SpeakLab analyzes speaking rate, pauses, relative loudness, pitch, and detected filler phrases. You can replay measured passages and listen for places where a change in pace, a pause, or a clearer transition could help an idea land.

I kept acoustic measurements separate from the model's interpretation of the explanation. NumPy and Praat measure the audio; the language model works with the transcript, references, and a summary of those measurements. The feedback gives you specific things to listen for and try, without turning “more variation” or “fewer pauses” into a universal goal.

A fully local AI pipeline

The React and TypeScript frontend records audio in the browser and sends it to a Python FastAPI backend. The server saves the recording, decodes it for analysis, and sends it through transcription, acoustic analysis, and model review.

whisper.cpp handles speech recognition, and Ollama serves the language model for assessment and tutoring. The AI workload runs entirely on my own hardware, without hosted LLM or speech APIs. Transcription runs on the CPU to leave GPU memory for the language model; quantized weights and a bounded context window keep that model within the available memory.

I also built checks around the model's output. Assessments follow a JSON schema validated with Pydantic, and supporting evidence must point to an actual transcript excerpt. The backend resolves those references to the original words. Invalid responses get one repair attempt, and proposed factual corrections receive another review. These checks make the feedback traceable while leaving room to question the model's judgment.

Running it on a server

I deployed the app on a Windows PC inside a dedicated WSL2 Linux environment. FastAPI serves the built frontend, and systemd manages the application, Ollama, and whisper.cpp as separate services. The PC holds the models and practice data; my laptop runs the browser.

I use SSH for remote administration and a tunnel for browser access from the laptop. Key authentication and a pinned server key establish the connection, and a background supervisor reconnects after interruptions. Forwarding the app to localhost also gives the browser the secure context needed for microphone recording.

Tailscale connects the devices privately, and Tailscale Serve provides HTTPS access from my phone, including over cellular. The application and model services listen only on loopback, with explicit browser-origin checks at the app boundary. This lets me reach the server away from home while keeping its services off the public internet.

Keeping practice recoverable

SQLite stores sessions, transcripts, assessments, tutor conversations, and review dates, with audio kept as separate files. A single background worker processes recordings, and audio is saved before analysis begins. If a service fails, the recording remains available and the failed analysis can be retried.

Reanalysis preserves previous feedback versions and the model used, so I can revisit an explanation as the system changes. Recording, playback, reference material, and review scheduling also work when the AI services are unavailable.

Ongoing work

I'm refining the topic material, interface, and feedback through practice. The next step is more individualized vocal coaching that connects suggestions to specific moments in a recording, alongside continued checks of the model's factual and communication feedback.