In a nutshell
Evesdrop is an experimental AI-powered picture frame, built using custom electronics, 3D printing and code, that listens to ambient conversation and turns it into art. Speech is transcribed and summarised using local AI models, stripped of personal information, then turned into an image using generative AI and displayed on a six-colour e-ink screen. Sometimes the result is surprisingly on point. Sometimes it’s completely unhinged.
Initial concept and AI realisation
Many years ago, one of the first electronic projects I undertook was an e-ink based picture frame that displayed a sequence of static comic book pages. After showing it to a friend, we started to discuss how it would be cool if the picture was more dynamic, and potentially based on current events, such as news feeds or even a conversation near-by. We theorised that you might be able to capture the audio and stream it to a server where it would be converted to text, and the text used to do an image search and display relevant images.
Years later while sitting in a presentation from one of our directors discussing AI, it occurred to me that the new capabilities that AI offered could help realise this idea perfectly.
The plan
The system functions as an end-to-end pipeline: audio is captured via a microphone, processed locally to text, summarized into thematic descriptions, and finally transformed into an image by a cloud-based generative model. This design balances the need for privacy—by keeping the initial speech-to-text processing on-device—with the demanding compute requirements of generative image AI.
The initial challenges:
- Figure out a way to capture audio and convert it to text (ideally on device).
- Convert the text into some sort of summary description.
- Generate an image using a generative AI image model, with possible configurable parameters.
- Display the image on an e-ink screen.
To run everything I needed to find a device that could potentially run local models as I was aware that having a device that records everything discussed in a room could have some privacy concerns. As well as having enough power to run local models, it needed connectivity options to drive the e-ink screen and be able to fit in a small form-factor. I ended up grabbing an OrangePi (similar to a Raspberry Pi), with a fair bit of RAM to start my testing.
Software
I chose to write the software pipeline as a NodeJS application, using event streams to pipe input and outputs from each step. I used Ollama to install and run the inference locally, which was super easy to set up.
Speech to Text
The first step was to figure out how to capture speech and transcribe it to text. Speech-to-text software has been around for decades, but new AI models have made it far simpler and more robust. OpenAI has a very popular model called Whisper available in many sizes, and this seemed to work straight away. After connecting a USB microphone to the Pi, I was able to get a stream of text coming from the model.
Evesdrop’s summary and description
Having a large pile of conversation fragments is great, but often it’s a bit incoherent and sometimes broken as the sound conditions in a space changes. To make the words useful they needed to be summarised into topic / themes so they could be better described. For this, I chose IBM's Granite4 model—selecting specifically one of the smaller parameter size models allowing it to perform summarization and extraction tasks effectively while running locally on the low powered hardware.
Generating the image
Using the description of the conversation, a prompt is formulated to be passed on to a cloud-based text-to-image model. Up until this point, all of the AI models are run on the device. The prompt generated is intentionally abstract and includes explicit instructions to remove personally identifiable information (PII), ensuring that personal details are never passed to 3rd parties.
To begin with, I started using OpenAIs DALL-E, but soon after Google released their Nano Banana model, which was a substantial step up in quality and coherence. Part of the prompt contains a variable style declaration that can be toggled by the user using buttons on the device. This switches between:
- classical portrait oil painting
- Ukiyo-e painting
- cyberpunk digital art
- art deco poster
Displaying the image
E-ink displays (like those found in many e-readers), have a few interesting characteristics that make them suitable for this type of project. For one, they aren’t back-lit, which means they aren’t as distracting compared to regular bright LCD displays. And second, once you’ve loaded an image it will persist without needing any power, making it ideal for something that runs on batteries.
Since my first e-ink project, which was a simple black and white image, e-ink screens have evolved with six colour versions. Six is better than two but not great compared to what humans see. To display the full colour image it needs to be resized and reduced to a six colour palette via some dithering.
Hardware
Form concepting
Thinking about what the device might look like, I imagined it as something small enough to fit on a shelf or a coffee table. A fairly minimal appearance like a picture frame, with the ability to free-stand. With this in mind I started generating concept images using Nano Banana, narrowing down the form to something I thought looked interesting. One of the images had a picture frame fused with the base which I initially rejected as a defect, but after thinking about it, realised that it would create an interesting design feature and provide a nice way to combine the frame (which I planned on buying off-the-shelf), and the base which I would be 3D printing.
CAD modelling
With a concept in hand, I began modelling the design in FreeCAD. The challenge was to come up with a way to include the e-ink frame, picture frame, Orange Pi board, batteries, some mechanism for turning it on/off, and possibly a way to change some of the input parameters to the generation process.
After a few iterations, I managed to reduce it down to a few basic components. A base that housed most of the components and allowed the frame to be inserted into the top, and some nice clunky mechanical radio buttons for choosing the image style. A backing frame for the screen to hold it in place and route the cables internally and some button caps.
Build and construction
Once modelled I 3D printed the parts and iterated for fitment. I didn’t want the device to look 3D printed, so I experimented with various filler and epoxy coatings, sanding and painting to give it a semi gloss finish.
Construction involved screwing in all the components and making a wiring harness to connect them all together. I managed to source some nice tactile radio buttons, that gave a satisfying clunk when switching the styles.
Output
It’s fun to see the images pop up on the device and try to guess at the topics that it’s pulled out of the air. Sometimes there are fairly obvious themes and depictions, other times you’re left guessing at what the discussions might have been. An interesting phenomenon is that often people will stand around the device discussing the current image. It creates a delightful feedback loop where the conversation about the art becomes the art itself.
