August 16

Tags

Leadership hack 025 – quantative data tells you what, qualitative tells you why

Posted by on August 16, 2018 in Blog | 1 Comment

We now have access to too much information rather than too little.  Making great decisions requires an understanding of the limitations of data and the importance of analysis and dialogue.

Modern digital products have no little limits on data you can collect.  Here are a few examples:

  • How quickly a user moves a slider when selecting a loan size (famously used by Wonga to determine risk and therefore rate)
  • How quickly a user enters their data, how many mistakes they make and where they make them
  • Frequency and depth of use of your product
  • From an IP address, you can find their location and often the company they work for
  • From the device/OS you can determine technical competence, income level, education level

 

Turing data into understanding required analysis and dialogue

Data by itself will not solve help you build better products or make better decisions.  Organising data can transform it into information.  Analysis turns information into knowledge.  Knowledge, when combined with domain expertise, allows insight and over time wisdom.

Importantly, moving from data to understanding requires humans.  Several high-profile failures when AI has been used poorly (see below), demonstrate that humans will need to remain involved for the foreseeable future.

It is important to have a dialogue around AI.  All stakeholders need to be involved, and preferably customers.  You will need to check for biases in the data scientists and the data.  As far as you can, you need to make it clear what data an AI uses and how it arrives at a decision.

 

The limitations of data

Quantitative vs qualitative.  While data is incredibly useful, the data only tells you where to look and what has happened, it will not explain why.  To move beyond the what and understand why you need to sit with customers and see how they are using your product.

Quantitive

Qualitative
Structured data (however this is changing with advanced analytics) Unstructured data
 Result is numerical  Result is non-numerical
Results generated by instruments or machines (including software) Results generated by humans

I will use an example to demonstrate the difference.

We needed to build an onboarding flow for a new financial product.  In the UK there are a number of terms for a person’s surname, these include surname, last name and family name.  We found that we had a high number of failures when trying to establish someone’s identity because people put the wrong name in the box marked surname.   It was only through a lot of user testing, we found that this term confused a lot of people, for example ‘I don’t know what surname is’.  We switched to the ‘last name’, and this solved the problem.

Getting the right data.  It is easy to use the data you have, rather than the data you need.  Being focused on the decisions you need to make, will allow you to get the data you need to monitor and alert critical areas of your business.  For example:

Example Monitoring Alert Why this is important Decision to take
Tactical Uptime of customer onboarding application No new successful onboarding in last five minutes (between 1000 and 2200) ·    Impact to the business of lost customers and revenue

·    Reputational damage undermines trust in the brand

·    Regulatory implications should this continue beyond 24 hrs

 P2 alert sent to CC1 distribution list, new channel spun up on Slac with logs.  On-call Dev to investigate and inform the incident management team
Operational New customer acquisition Number of new active customers drops 10% ·       Regulatory implications should this continue beyond 24 hrs Analysis of current and competitor digital marketing (ad-words, PPC) to determine if changes are needed, analyse sales funnel to determine if there is a specific page/step causing higher than normal drop-off
Strategic Change in digital channel mix (desktop, mobile, voice, VR/AR) Voice or AR/VR amount to more than 10% of financial interactions in the UK market ·    We do not have a voice or AR/VR offering

·    We may not be missing a key market segment

·    We could miss out on industry inflexion point

 

 Create a new channel pilot.  Investigate the rate of change.

Investigate change in company strategy, structure, processes, tools and people

 

Data is incredibly valuable, if used well and if you understand the limitations.    Here are the top tips I learnt:

  • Have a clear reason for data and make that clear to everyone (including customers) – data needs the effort to turn it into understanding and this must inform a decision otherwise it is wasted
  • Dialogue with stakeholders can help prevent biases in both the data and decisions
  • Use quantitative data to tell you where to look and qualitative data to understand why