August
16
Tags
Leadership hack 025 – quantative data tells you what, qualitative tells you why
We now have access to too much information rather than too little. Making great decisions requires an understanding of the limitations of data and the importance of analysis and dialogue.
Modern digital products have no little limits on data you can collect. Here are a few examples:
- How quickly a user moves a slider when selecting a loan size (famously used by Wonga to determine risk and therefore rate)
- How quickly a user enters their data, how many mistakes they make and where they make them
- Frequency and depth of use of your product
- From an IP address, you can find their location and often the company they work for
- From the device/OS you can determine technical competence, income level, education level
Turing data into understanding required analysis and dialogue
Data by itself will not solve help you build better products or make better decisions. Organising data can transform it into information. Analysis turns information into knowledge. Knowledge, when combined with domain expertise, allows insight and over time wisdom.
Importantly, moving from data to understanding requires humans. Several high-profile failures when AI has been used poorly (see below), demonstrate that humans will need to remain involved for the foreseeable future.
- Microsoft’s Tay chatbot turned racist in just a day
- Google censored image searches for “gorilla” and “chimp” because it showed photos of African-Americans
- Facebook researchers found two AI-driven chatbots had developed their own secret language
It is important to have a dialogue around AI. All stakeholders need to be involved, and preferably customers. You will need to check for biases in the data scientists and the data. As far as you can, you need to make it clear what data an AI uses and how it arrives at a decision.
The limitations of data
Quantitative vs qualitative. While data is incredibly useful, the data only tells you where to look and what has happened, it will not explain why. To move beyond the what and understand why you need to sit with customers and see how they are using your product.
| Quantitive |
Qualitative |
| Structured data (however this is changing with advanced analytics) | Unstructured data |
| Result is numerical | Result is non-numerical |
| Results generated by instruments or machines (including software) | Results generated by humans |
I will use an example to demonstrate the difference.
We needed to build an onboarding flow for a new financial product. In the UK there are a number of terms for a person’s surname, these include surname, last name and family name. We found that we had a high number of failures when trying to establish someone’s identity because people put the wrong name in the box marked surname. It was only through a lot of user testing, we found that this term confused a lot of people, for example ‘I don’t know what surname is’. We switched to the ‘last name’, and this solved the problem.
Getting the right data. It is easy to use the data you have, rather than the data you need. Being focused on the decisions you need to make, will allow you to get the data you need to monitor and alert critical areas of your business. For example:
| Example | Monitoring | Alert | Why this is important | Decision to take |
| Tactical | Uptime of customer onboarding application | No new successful onboarding in last five minutes (between 1000 and 2200) | · Impact to the business of lost customers and revenue
· Reputational damage undermines trust in the brand · Regulatory implications should this continue beyond 24 hrs |
P2 alert sent to CC1 distribution list, new channel spun up on Slac with logs. On-call Dev to investigate and inform the incident management team |
| Operational | New customer acquisition | Number of new active customers drops 10% | · Regulatory implications should this continue beyond 24 hrs | Analysis of current and competitor digital marketing (ad-words, PPC) to determine if changes are needed, analyse sales funnel to determine if there is a specific page/step causing higher than normal drop-off |
| Strategic | Change in digital channel mix (desktop, mobile, voice, VR/AR) | Voice or AR/VR amount to more than 10% of financial interactions in the UK market | · We do not have a voice or AR/VR offering
· We may not be missing a key market segment · We could miss out on industry inflexion point
|
Create a new channel pilot. Investigate the rate of change.
Investigate change in company strategy, structure, processes, tools and people |
Data is incredibly valuable, if used well and if you understand the limitations. Here are the top tips I learnt:
- Have a clear reason for data and make that clear to everyone (including customers) – data needs the effort to turn it into understanding and this must inform a decision otherwise it is wasted
- Dialogue with stakeholders can help prevent biases in both the data and decisions
- Use quantitative data to tell you where to look and qualitative data to understand why
Pingback: Product leadership part 1 - making great decisions - getting better every day