Local e-ink handwriting recognition with on-device VLMs
For decades, I've carried a Field Notes notebook and a pen. I mainly used them to capture ideas on the go, but it would also be great to sketch out a diagram, or to journal a little bit, especially while traveling.
Fast note-taking via the iPhone's action button has alleviated a lot of my need for quickly capturing ideas. But nothing can replace pen and paper for long form stream-of-consciousness writing or diagramming.
I wanted to give my writing a digital life alongside the rest of my notes. So a year ago, I bought an A5 e-ink writer called Supernote. I really like it so far: it's a good size, input latency is reasonable, and the overall writing experience is fine. The device provides real-time text recognition on-device and a modest cloud syncing service. I've been using their unofficial API to sync notes and bring them into my Obsidian inbox. But then something happened...
Book: The Crisis of Narration by Byung-Chul Han
Book: Livewired by David Eagleman
Book: Monkey by Arthur Waley
The pursuit of frictionless capture
The most fruitful moments for contemplation are often the least conducive to capture. I've been working on reducing friction in capturing thoughts, feelings, and ideas wherever I may be and whatever I may be doing. At the same time, I want to stay in the moment and not get distracted by the act of capturing itself.
In addition, I want to retain control over all of this data. It should be stored in human-friendly plaintext format. It should touch as few servers as possible and come to rest on a surface that I control.
I am excited about all three of the capture methods I'm about to share with you. Three questions to whet your appetite:
- What if you could freewrite with pen on paper and have the salient bits magically show up in my digital note corpus?
- What if you could use your locked smartphone to type a note without ever being distracted by its contents?
- What if you could dictate notes while walking, running, or riding a bike without even requiring a smartphone?
To motivate this, watch this quick summary of the scenarios above, and read on for some tech details for how it all works.
[フレーム]Book: Hard to Be a God by The Strugatskys
Book: Piranesi by Susanne Clark
Book: Secret of our Success by Joseph Henrich
Book: Math from Three to Seven by A. K. Zvonkin
Book: Abundance by Ezra Klein and Derek Thompson
Book: VALIS by Phillip K. Dick
Book: The Maniac by Benjamín Labatut
Large and sometimes Oppressive Language Models (LOLs)
Authoritarian governments have latched onto open-source LLMs like Llama to craft their own models, complete with censorship rules. In places like Russia and China, these censors manifest as an abrupt cutoff, where a stream of text is replaced with an uncanny canned response like "I'd better keep quiet". In this post I'm interested in probing for subtler, more insidious manipulations potentially present in language models controlled by authoritarian governments. How might these warped filters influence entire populations? Are they already shaping our collective understanding in ways we barely notice?
Book: A Peace to End All Peace by David Fromkin
Horsing around with invention.cards
This is the first in a series of vignettes based on observations I captured while creating invention.cards, a visual chronology of science and discovery. In this series I hope to explore bits of the history of science and technology I found fascinating ("that's funny...") while reading and digesting Asimov's chronology and also examine the limitations of source material. First, let's take a really thin slice of human ingenuity: horse-related inventions and discoveries.
Invention & Discovery Cards work complete
I'm pleased to have completed transforming Asimov's Chronology of Science and Discovery into a deck of Magic Cards. Over five years later, all 1477 entries from Asimov's encyclopedia are now represented as illustrated cards on https://invention.cards. The website is rendered based on this master spreadsheet which I compiled with the help of AI and manually vetted. Since AI hallucinations can safely be ignored, and I am infallible, I declare victory!