Heads up: Our Ideas Factory has been refreshed, levelled up, and grown-up into Alphero Intelligence. Some of our old posts are pretty cool tho'. Check this one out.
- We love taking the latest technology, mashing it up and seeing how it could be used to make life better for people - basically putting the purpose behind our R&D
- We took Apple’s Sound Recognition AI and explored it through a health lens, asking what challenges it could solve
- After much concepting and technical investigation, we can see big potential for the future…
When we first did this exploration in 2021, Apple’s AI platform had just been released with sound detection capability - an accessibility feature which can be leveraged for other purposes. This feature is programmed to decipher critical sounds like a baby crying or glass smashing - and then alert the user to what’s happening around them; it’s particularly helpful for people with hearing loss.
But with over 300 pre-trained sounds on board, it can also pick up other more subtle cues, like a person sighing, a cup of tea being poured or some cutlery clinking. The latest version even allows for customisation - so that it can be programmed to recognise sounds that are specific to you, like your own personal doorbell.
Mashing things up
This got us thinking: What are the wider possibilities for this type of technology?
Next minute, we got the team together for an #ideasfest. We asked a group of designers and developers to look at this software with a health lens and explore what problems it could solve.
They started mashing up ideas around a range of health issues from breathing to sleeping to taking medications, then throwing these concepts to the developers and saying ‘Ok, how would we do this technically?’
Enter the dragon
One of the sounds we considered in the early stage was breathing. How might sound detection be useful here? We landed on asthma and followed that thought to consider kids and the tricky business of helping them use inhalers.
New Zealand has a high prevalence of asthma in kids aged 2-14, so we wanted to explore whether sound detection could be used to monitor their breathing patterns and make it easier for them to use their inhalers.
We came up with a Puff the Magic Dragon concept, where the system could translate the sound of the child’s breathing into a dragon breathing fire - fun and helpful!
The software could check for the sound and duration of the child’s inhalation and feedback with encouraging messages “4 more puffs, see if you can do even better!”
We knew a similar technique had been used in a popular kids’ app Balloonimals. That was purely for entertainment purposes - but could it also be helpful for asthma? The dev team tested the idea using the device’s microphone - looking to detect the force and length of the exhale, to eventually measure how effectively a child could breathe in and blow out. They found that while the software could recognise the sound of breathing, it couldn’t assess the length of the breath - so not there yet.
Next we moved on to another pressing issue for the team: could sound monitoring be used to help elderly or disabled relatives remain independent in their own homes for longer?
The sweet sound of home
The issue of how to care for an ageing population is going to be a big one in the future. Not only will care facilities be unaffordable for some but, costs aside, many elderly people would much prefer to stay at home for as long as they can. The same is true for people living with disabilities but wanting to retain the freedom to live independently.
A few of us already have family members in this boat…so we were keen to ideate around it.
Detecting daily routines
We began by exploring the idea that sound detection could be used to illustrate patterns in behaviour that relate to a person’s wellbeing - and show what is normal (and not normal) for that person.
We thought about the key indicators that could be monitored, such as whether our rellies were rising at the same time each day - or suddenly staying in bed longer; If the sound of a front door opening and closing could reveal that they’re getting outdoors regularly; and whether the sounds of food preparation would show that they’re eating a proper meal - versus simply opening a packet of biscuits.
Key changes in routine could be flagged with a family member or designated carer so that they could check in.
Some of the younger team members came up with some novel solutions: No sound of breakfast being made? Then Uber Eats is triggered to turn up with pancakes!
Note: Our CEO and Delivery Manager couldn’t see this one going over well with their elderly folks.
The serious stuff
The AI could also detect more urgent concerns like if the person were to have a fall, then help could be alerted immediately. Or it could regularly monitor for specific sounds of ill health such as coughing, sneezing and sniffling. A carer could request a health check through the app and see when these sounds are happening and what else is occurring at those times i.e. does coughing occur more frequently at meal times and could this be a choking issue that needs to be checked?
The daily dose
Along with general healthy routines we considered another crucial element of staying independent: being able to keep up with daily medications. We looked at the possible sound activations that could indicate whether this was (or wasn’t) happening, at a set time each day, eg the sound of a pill case rattling, whether food preparation was occurring for medications that needed to be taken at meal times, and whether there were any interruptions to the routine that could cause a person to forget this process - like a knock at the door or any other kind of distraction.
Someone even pondered whether a robot vacuum could come around with the pills…. No idea's a bad one when you're ideating.
The future
Overall we felt there were very real possibilities in this area - but there were also ethical issues to ponder. Would people be comfortable with technology operating this level of surveillance in their lives? We talked to our rellies about this and found that in most cases, maintaining privacy was of far less concern as they aged, particularly if it meant the freedom to stay at home instead of moving into a care facility. This may not be the case for everyone…
Importantly, our investigations showed us that you don’t necessarily need lots of extra doodads - simple noise detection could already start to solve some common problems and make a positive difference in the lives of older generations.
A bit of background
The purpose of these sessions is not necessarily to come up with a complete solution to a known problem - but more importantly to create a micro opportunity for our team to do some out-of-the-box thinking.
More than just trying out some software, we get to mix it up, merge ideas and concepts and take it to a new level; prototyping something completely new and different.
During this process we stretch our minds beyond everyday practical considerations and move into the zone of ‘what if?’
Often the discoveries we make in these sessions find their way into the work we do for clients. Not only does it build our creative approach to problem solving - but it expands what we know about the capabilities of a range of new technology.