Romily Cocking made Video Transcriber, an open source tool that takes a video, transcribes the audio to text in a markdown file, and clips images every n seconds. It uses local models, so is great for transcribing private presentations and narrated sketch notes.

I attempted to write about Video Transcriber when I first used it nine months ago. But for me, improvisation and explanation don't mix very often, in the moment. I picked it back up to help me draft blog posts, starting from me talking with a virtual flip chart in hand. I am writing all my words organically again. Some typed, some translated from handwriting or voice notes and now images and words from a video.

Instead of using AI to write my words, I am using AI to build me small tools to get my words out. These tools may or may not contain AI themselves. In today's episode, I used a Speech-to-Text model together with a deterministic computer vision library. I only want to record an image (a 'slide' in the domain of Video Transcriber) whenever I stop dragging the canvas around, or stop drawing for a bit. The vision library allows deterministic code to do that. It can be used instead of the existing functionality in Video Transcriber that creates a slide every 15 or 30 seconds.

A regular pace does not seem to be a good fit for when I am sketching. I either sketch a lot, or talk a lot, but rarely at the same time. Extending Video Transcriber to take that into account was easier done than thought ;-). I am still surprised I can now do this kind of thing without deeply understanding the underlying technology.

This post hinges on three things:

• Using local AI to work with audio and video, and help me write
• Using a coding agent to modify someone else's tool to taste
• The challenge of publishing blog posts at the moment I have just done the thing

It became a bit bigger than I imagined, and is less structured then when I was drafting. I did not plan to do stopped-motion detection, I swear! I hope you enjoy.

Before we go on, a small sample of Video Transcribers' work. This one has timestamps to go with the text, it can also output whole paragraphs at once.

You may notice it says "Santola" in the screenshot. That should be Centola. With a large Speech-To-Text model running on my laptop, this was about the only mistake. Phonetically they are the same, and I may need to work on my pronounciation. This is of course easily fixed by using search and replace.

Using local AI to work with audio and video, help me write

What is Video Transcriber for?

It is originally intended to make youtube videos easier to skim and refer back to. Reading text is a lot faster than listening to words. If you missed something, you can quickly glance back in text, whereas in video you have to scroll back and wait for the moment that you missed - it is only obvious in hindsight where that was.

Different tools for different purposes

For youtube videos I now use Gemini Notebook, because it already has the transcriptions. I do miss the visuals with it sometimes, and want to replace it with an open notebook that runs with local models when I want to. I hadn't used video transcriber in a while, but that changed last week.

Getting words on the page

What I am interested now is a more effective Brain to Blog flow. I have many ideas that I want to explain better to others, and get feedback on, but getting words and pictures on paper is a struggle sometimes. I understand, friction is part of the learning, and I do not wish to remove all of it. But I have recently been Sketch noting again. Narrating while drawing does help me to get ideas on to paper and in audio. Talking after drawing also works. The second and third talk-throughs are usually more concise. It does help to sharpen the vocabulary.

I picked Video Transcriber back up to transcribe sketch notes. I sketch on my tablet, and make a screen recording with audio as I do it. Then transcribe it on a laptop and get a markdown file with images for significant moments out. It takes about half the time of the length of the video to do the transcription. For instance, on my laptop it transcribed a 6 minutes chat in 3 minutes, at the highest quality settings. No special hardware like fancy graphics cards required. I did explore graphics cards and wrote about that further down.

Baby you can code while I write

While I was drafting this blog post, trying to explain while I did not write about it nine months ago, I set my coding agent to work on a wish. A 6 and a half sketch noting video yielded 81 'slides' (the systems metaphor is a transcriber for a presentation with slides). It would be nice if it only made a slide once I stopped dragging the infinite canvas around, or when I stopped writing for a bit.

I gave the coding agent a brief prompt, and the video I made to draft a book review of Damon Centola's Change: How to make big things happen, together with output from Video Transcriber that made a slide every 15 seconds, for a total of 81 slides. With access to tools (ffmpeg, the OpenCV library, Python), and my question, the coding agent (Pi with GLM 5.3 flash in this case) did most of the work independently. It used Romily's demo video from the repository for quick tests, and also used my book review video to compare the resulting markdown. It did some iterations on it to find good settings, and explained three settings to me, which I had it save in a document for the future.

After it more or less worked, I found that the text occasionally contained repeated phrases. The model had spotted this as well, and suggested to remove silence from the video before processing. It had found something that could do it, so I let it rip.

Getting words out to the world

I already invested in making it easier to put images in notes and blog posts, and export notes to blog posts. That is paying off already; it is the kind of work that compounds. I have less friction in details, so I can focus on the stories I want to tell. Now the work is in making it a habit to finish and publish posts, as the drafts (with images :-) ) were piling up.

The challenge of publishing blog posts at the moment I have just done the thing

Why I didn't write about it nine months ago

So why didn't I write about Video Transriber earlier? I sometimes find it hard to explain what I did right after I did it. Especially when I am improvising.

Romily asked if I wanted to contribute something back, at the time, say a blog post or a pull request. I struggled with that. As a practice, Will It Blend is great to get something to work, and the flow feels great, as it is very improvisational. Being able to quickly vibe code even a small improvement and use it is fantastic. A downside of it, is that I often can’t easily explain what happened afterwards, not even with the session transcript, as the coding agent records it, in hand.

Adopting someone else's tools nine months ago

Looking back at the nine month old commits now, it was me using `uv` instead of `pip` - I standardized on uv as it felt easier and seems more modern and faster. But I am not a heavy python user like Romily, so I did not want to push my preferences at the time, and I couldn’t quickly see how to have both co-exist in the README.

The other one was support for my AMD graphics cards through ROCM. It looks now like it also works with the GPU build in to my laptop, fancy external GPU optiona.. I hand’t done anything with ROCM at the time, so was unsure how to integrate it into video-transcriber.

Am I going to use Video Transcriber more now?

It might well be, that once I have the technical solution, is that my flow of narrating improves to the point where I can just take the screenshots myself, or prefer to select SVGs from the source and drag them in at the appropriate place in the blog post.

I guess this is what happpened with my first use of Video Transcriber. I liked having videos readable and easy to skim, but found that a notebook works better for me when I want to ask questions and go deeper.

Automation is great, but not the solution for everything.

In the meantime, I would recommend checking Video Transcriber out. It is small and focus, and looks like a good starting point for experimenting with transcription and getting information out of videos. My coding agent is humming in the background, see if if we can do something clever with motion detection to reduce the number of generated slides.

See you next time!

Further Reading

In Will It Blend I went into more detail in how I adjust other peoples' software to my needs.

Last year I wrote for QWAN on how I see my evolution of knowledge and tool use, in the age of Large Language models: Stochastic to Deterministic. I do more with language and other kinds of models than before. But I still turn things I repeat into tools. Writing is still hard, but removing accidental friction, like adding images to this blog from my writing and publishing flow keeps paying off.

I published my fork of Video Transcriber, in the spirit of "Great artists ship". It is make it run, with tests. I must admit, I let my agent ignore the instructions to write one test at a time, red-green-refactor style. I find that interesting to do occasionally, especially as a workshop, to 'feel' a model up close, or when I guide a team through entangled legacy code. In this case, I wanted the outcome, and that goes faster with Synthetic TDD. Express intent, set up feedback loops, try it out, and do another round.