Fun times! How to quantify the productivity impact of using gen AI tools for digital delivery? On the positive we've been rolling with AI tooling etc for coding for a while with some devs and projects. And feedback is now "I literally can't imagine doing my job without it". Followed by mildly blank looks when it comes to a request to quantify the benefit. There is a lot of noise in this space and it can be hard to unpick what's real (or realistic).
So we ran an experiment.... went analog with a stack of post-its by each dev's laptop. Every time they go to use gen AI to complete a task they have to stop, write the task down, estimate doing it without AI. Then record the time and effort (and evaluate the outcome) of using AI instead.
This is a journey. From a headline insights perspective: the devs reckon they are getting at least a 20% improvement in coding productivity just from using the IDE (think autocomplete on steroids etc). On top of that there are a raft of tasks the team use it for as standard: great for troubleshooting and knowledge retrieval, explaining other people's code, writing utility scripts, converting natural language into code or technical artefacts, bootstrapping and set-up, translating from one code language to another, creating a curl command to create a typescript function. Anything repetitive and/or involving data: things with repeating a pattern (crack one of 10 things and it will do the next 9), writing complex SQL queries, common table expressions, data inserts, data transformations (create schema). The impact of these tasks is in addition to the 20% eg some tasks that might've taken a day, now take 20 minutes.
The gotcha is that our most experienced folks have been getting the most out of it at this stage (and you still need a human in the driver's seat!) There are occasional hallucinations suggesting approaches that aren't technically possible; or changing functionality without letting you know. It's our seniors who pick this up fast.
What we'll be doing next is doing a baseline analysis of defects across multiple clients and then starting to analyse whether there is also a correlation with quality.
Team Design is next off the block for analysis. They are already using a raft of tools for everything from heuristic reviews to analysis of user research. Plus MidJourney etc. We get access the gen AI through Figma shortly so will report back in due course.