SonderMind Product Manager Project (post #17)

Overview

This project is based on SonderMind’s new product manager position. I amhoping that this will give you some insight into my ability to discover a combination of creative and data based insights into product management as it applies to SonderMind’s goal of dramatically improving behavioral healthcare.

The product enhancement that I focus on in this project is a machine learning based chatbot that interacts with new users throughout the onboarding process. The chatbot could perform a fairly detailed intake to further improve the matching while also making the sign-up process more interactive and engaging. An additional value that the chatbot could provide is addressing the fact that therapy causes fear for many people. If SonderMind could ease that fear, more people would reach out for help. A chatbot that has a back and forth exchange with the user could be a much better experience than the current sign-up process. If this experience improves the users motivation to go to therapy, then significantly more users would be converted to customers.

SonderMind Current Product

SonderMind is a two-sided platform for clients and mental health professionals (therapists). SonderMind quickly connects customers with a local, in-network therapist who understands their needs. They match them based on specialty, availability, location, treatment approaches, insurance, and more.

SonderMind also makes it easier for mental health providers to focus on providing great care, not running a business. SonderMind can help therapists grow their private practice, attract more clients, manage billing, and get credentialed with major payors. 

According to their website, their network has over 400 mental health professionals who cover a wide range of mental health issues https://www.sondermind.com/faq.

SonderMind is looking to discover, define, and create solutions that drive SonderMind’s ability to re-design behavioral healthcare. 

SonderMind Technology

I went through the sign-up process. On the homepage of the website, you can click on “match with a therapist.” From there, you can choose from several options as to why you are seeking support. Those options are anxiety, depression, relationships, health and wellness, addiction, need medication, children and general support. There is also a “details for your therapist” text box in which you can provide further information. I picked anxiety. 

I was then asked whether or not this was a life-threatening situation. I checked no. Then I was asked to enter a zip code. After that, days and times that do not work for me. SonderMind uses this to connect you with a therapist with the same availability. Next was insurance benefits, followed by name/birthdate/gender/email/phone. I was asked if I preferred to be contacted about matches through text or email and had to check that I had read HIPPA/Disclosure Agreement, Terms of Service, and Privacy Policy. I was told that I would receive an email that would provide further details in regard to setting up my account. 

The link sent me to their portal. In the portal you create a password and enter your mailing address, insurance information, and credit card information. 

Within an hour, I received an email about being matched with two therapists and had the ability to view their profile. The therapists clearly had a focus on helping those with anxiety. I received a phone call from both therapists within 24 hours.

Competition

Two Chairshttps://www.twochairs.com.

Not a true competitor at this time as Two Chairs is focused on the Bay Area in northern California while SonderMind is focused on the Denver area. However, they represent an interesting comparison and are both likely to continue to scale. They offer a similar matching model, and likely similar technology, but their process does have some differences. One big difference is that there is a consult clinician who does a 50 minute interview with new clients. Before this, new clients do fill out a survey that goes over history, preferences and goals. 

The in-person interview allows for deeper conversation around client needs and what they hope to get out of therapy. With data at hand, Two Chairs makes a match with the right therapist and makes a custom treatment plan. 

The consult focuses on symptoms, goals, personality, background, past treatment, medical history, family history and logistics. They offer many forms of therapy, but some of the most popular ones include Acceptance and Commitment Therapy (ACT), Cognitive Behavioral Therapy (CBT), Dialectical Behavior Therapy (DBT), Emotionally Focused Therapy (EFT), and Psychodynamic Therapy. 

Two Chairs has clearly put a lot of resources towards getting the right match. This includes getting style and therapeutic orientation matched up between therapists and clients. Additionally, Two Chairs asks clients about their preferences in terms of level of structure, directiveness, present vs. past focus, and challenge vs. support from their therapist. The matching works well, but there would seem to be a risk of clinicians who have to do these consults being overworked and burning out. 

Therapists of Two Chairs are employees and get paid a salary. 

My Therapist Matchmytherapistmatch.com

The process is to answer their proprietary questionnaire, evaluate matches, and then contact the matches that you like. 

Some questions include whether or not you prefer male or female clinicians, they list 49 potential issues for you to choose from, and then there are several personality questions such as the reasons you buy a car (price, features, safety). 

Psychology Todaypsychologytoday.com

Allows the user to search for therapists. Each therapist has a bio in which they outline their therapeutic focus, their phone number and their email. User can also filter results by issue or insurance carrier. 

My Approach to Improving the Product

The matching technology is a great idea. It seems that both therapists and clients like it. Two Chairs has an intake in which the consulting clinician gets a significant amount of information from the client in an effort to further improve the value of the matching. This should enable the therapist and client to hit the ground running in their first session. However, it also uses up resources and could lead to some burnout for the clinicians. It also is not particularly efficient in terms of getting the client to their first session with their new therapist. 

I think it would be very interesting to see what a machine learning backed chatbot would look like. The chatbot could be much more interactive and engaging than the current process. It could also do the intake (or at least the majority of it) so that a clinician did not have to. 

An additional value that the chatbot could provide is addressing the fact that therapy causes fear for many people. If SonderMind could ease that fear, more people would reach out for help. A chatbot that has a back and forth with the user could be a much better experience than just going through a sign-up process or reading a blog. If, for example, a user wanted therapy due to their anxiety, having the chatbot discuss anxiety with them could make it seem less frightening. Or perhaps the bot could even discuss some of the more common reasons why people avoid therapy. 

It could also be that converting an activated user into a customer happens through a series of interactions with the bot. One of the great values of machine learning is that the chatbot would learn how to respond in a similar way that Netflix learns what movie or show you might want to see next. For example, the bot could conclude that “86% of clients who responded like you prefer psychodynamic therapists. Would you like to look at some of our psychodynamic therapists who work nearby?” 

The metrics that we would be evaluating are change in NPS score, change in optimism levels reported to chatbot, average number of counseling sessions per month, percent of new users who become customers, retained users, and change in percentage of incomplete signups per month.

Customer Development

Customer development is the practice of establishing a continuous and iterative communication line with your customer so that you can come up with ideas, hypotheses, try them out, get feedback, and adopt your product accordingly. 

The key point of customer development is trying to find out if we were talking about building the right thing. I also see customer development as a tool for risk mitigation and opportunity recognition.

Customer interviews help you understand the reasons why customers buy or don’t buy your product. 

Types of Interviews

Here we only would focus on the exploratory and validation interviews. We really want to focus on pain points and validation. 

Exploratory Interview: the most free form. Trying to establish whether or not they have a certain pain point (fearful of going to therapy and therefore not going) and are open to certain solutions (chatbot). Ideally, the people to interview are ones who have registered for SonderMind but have not set up or gone to a therapy session. 

Validation Interview: you have a theory and you want to test it out. These interviews are run in a scientific way. They are hypersensitive to bias. Don’t introduce your theory or idea until the very end. You try to be as objective as possible when describing your idea. See if they talk about this problem on their own. 

Good Questions to Ask

Ask open-ended questions: it gives them the room to give any information they see fit. They talk more.

Don’t ask binary questions: these are questions with only two possible answers. You don’t get any useful information this way. 

Don’t ask hypothetical questions: people don’t really know what they would do in hypothetical situations. You won’t get useful feedback. 

Don’t ask leading questions: these are questions that somehow include the answer. You will influence their answer and bias it. 

Don’t ask questions that might make them lie: don’t put them in an awkward situation. You will never know if their answer is real. 

Minimum Viable Product (MVP) 

A version of a new product that allows a team to collect the maximum amount of validated learning about customers with the least effort. 

Figuring out what your problem/solution set is: while a machine learning based chatbot would be capable of much more, the MVP could be built without any machine learning. As an example, a combination of using textit.in(to build the text message decision tree) and twilio.com(to get a phone number to text from) could be used. This could allow for the text message interaction to take place and to understand whether or not users liked the interaction and whether it improved conversion rates. 

Identifying your assumptions and ranking them: if it is not true, your product will definitely fail. Usually, the riskiest thing to assume is the fact that your customers have the specific problem that you’re trying to solve. If they don’t have that problem, they won’t care about your product. The number one assumption is that going to therapy scares many people and this fear often keeps them from ever going.

Building testable hypotheses around your assumptions: the difference between assumptions and hypotheses is that hypotheses are actionable, they have a target group, an expected outcome, and a strategy to get customers to act in a certain way. 

Basic Hypothesis: we believe activated users will become customers more often because the conversational chatbot makes the signup process more engaging and interactive while also easing their fears about therapy. 

Specific Hypothesis: we believe potential customers have a fear because the perception of therapy is often scarier than reality. If we develop a conversational chatbot that eases those fears, the number of activated users will increase and conversion rates will improve. 

Establishing your minimum criteria for success (MCS): You need to establish a MCS in order to decide whether your product is worth building or not. The MCS gives your experiments clarity and meaning. In order to set your MCS, you need to take into consideration the cost and the reward of building the product. Without having access to data it is impossible for me to come up with precise numbers, but the MCS would be that users who interacted with the chatbot increased conversions (went to at least one therapy session) by x%. 

Prioritization Due to Risk and Difficulty: Generally speaking, companies have to prioritize which assumptions to test, depending on their degree of risk and difficulty.

Risk: how risky are the assumptions for the company or product?

Difficulty: how much effort will you need to make to test the assumptions?

These characteristics can be drawn into a 2x2 table so that high risk and low difficulty are top left, low risk and low difficulty are bottom left, high risk and high difficulty are top right, and low risk and high difficulty are bottom right of the table. The first category to test should be high risk and low difficulty. Second should be high-risk high difficulty.

High Risk/Low Difficulty: The riskiest assumption is that interacting with a chatbot can lower fears about therapy. This can be fairly easily tested with an MVP chatbot that does not have any AI/machine learning capabilities. 

High Risk/High Difficulty: If the MVP is successful, then the next assumption to test is whether a machine learning backed bot could significantly improve the results (user satisfaction and conversion). It is hard to test this without actually building the alpha version.

Low Risk/High Difficulty: Continuing to build out improved versions of the chatbot might require some specific technical skill, but at this point the risk would have been lowered.

Low Risk/Low Difficulty: Forget about fears and just have the bot do a more detailed intake to improve upon the matching. No machine learning necessary, just standardized questions. This may not improve conversion, but could improve matching and therefore improve customer satisfaction and revenue for both therapist and SonderMind. This also could provide value for therapists that they cannot create on their own.

Evaluating and learning from the experiment: MVP experiments will primarily return quantitative data in the form of numbers that indicate behaviors.

We also need qualitative data from customer interviews in order to make the right decision. Qualitative data helps us understand WHY the customer did what they did.

Putting together all the data will help us figure out whether our MVP experiment was successful or if we need to make some changes and run it again. 

HEART Framework 

A 3x5 with happiness, engagement, adoption, retention and task success going down along the y axis and goals, signals and metrics going across the x axis. 

Happiness: how happy is your user?

Engagement: how engaged is your user in the short-term?

Adoption: how many users tried your product?

Retention: are your users returning every single month?

Task Success: what’s the most important thing your users should do with your product and are they doing it?

Goals: what do you want to happen?

Signals: what is the actual thing that you need to measure in order to know that you’re getting closer to your goal?

Metrics: how do you take the goal and signal and express it as an actual metric over time? 

Happiness

Goals: Increased level of optimism about going to therapy

Signals: NPS score or app store rating. Could also have bot ask user about levels of optimism over time

Metrics: Look at change in NPS score or change in optimism levels reported to bot

Engagement

Goals: More counseling sessions

Signals: Number of orders

Metrics: Average # of counseling sessions per month per activated user

Adoption

Goals: Get new customers to their first therapy session

Signals: First therapy session reserved

Metrics: % of new users who become customers

Retention

Goals: Keeping customers

Signals: # of people with a paid session this month

Metrics: Retained users (# of people who with a paid session this month divided by # of those same people who ordered last month

Task Success

Goals: More activated users (they completed sign-up process)

Signals: Incomplete sign-up

Metrics: Percent of incomplete sign-ups per month

Planning with Development Team (good things to remember)

With engineers, explain everything clearly and in detail. Include technical information. 

If something goes wrong, it’s your fault for not providing the correct specs.

When pitching a feature, make sure you have a good idea of where the feature is going to go in the future.

When possible, you should do the work upfront on things like checking logs or looking up data.

Tech debt:something that has to be dealt with later because it wasn’t done right the first time. Engineers hate this. 

Don’t treat engineers like an agency. Don’t design or come up with all the concepts yourself and just hand them the requirements. Let them provide feedback and come up with their own ideas. They will help you.

A/B Tests

Typical tools to use for A/B testing are KISSmetrics, Mixpanel or Optimizely. However, the nature of a machine learning chatbot is that it is constantly optimizing the interaction based on the user’s responses. However, we could and should look at A/B testing the SonderMind website in it’s current form versus the chatbot.  

Customer Feedback

Good places to get feedback are social media (especially Twitter), email, and your blog.

Your blog:look at the people who are commenting. They are probably users and they want to be heard. 

Power Users:these are people who use your product frequently, buy things often, or send messages often. They are more informed about your product and they’re more interested in it, so there’s a good chance that they’ll be interested in talking.

Twitter:look at who is tweeting at you, replying, or sharing your posts. It will be easier to get these people to talk to you.

Cold Emails (from people who have used chatbot)

Be short: the ideal length is 4-7 sentences.

Be personal: mention how you found them and then ask them to talk to you and help with feedback.  

Be valuable: show them that this conversation is valuable to them. People want to feel like they’re helping. Tell them that you value their input. You can offer to incentivize them. 

Bonus: mention that you’re not from sales. Ensure them that you want to fix their problem and that they have valuable information that can help you. 

Once the MVP has been used, we can use two other types of interviews: 

Satisfaction Oriented Interview: find out which parts of the product are good and which are not good. Understand why they are satisfied or unsatisfied. Example questions are something like “what should we stop doing” or “what can we do to do this better for you?”

Efficiency Interview: find out how you can improve your product to better serve its purpose. Find out when they use it and where it is most helpful. “How easy is it for you to use the chatbot?”  “Was there a part of the chatbot interaction that you didn’t like? “Did the chatbot make you feel any better about the prospect of going to therapy? ” 

Roadmap

Customer Development (two weeks): Customer development is the practice of establishing a continuous and iterative communication line with your customer so that you can come up with ideas, hypotheses, try them out, get feedback, and adopt your product accordingly. Customer development is also a tool for risk mitigation and opportunity recognition. 

MVP (1 month): the technology for the MVP would probably not be all that difficult to build. The challenge would be building out the content. It would be necessary to have several ten minute or so interactions for each of the eight options as to why the user is seeking support (anxiety, depression, etc.). I’m going to roughly assume that getting the content together and integrating the text messaging to the SonderMind website takes a month. 

MCS and iterations (one month): we set a minimum criteria for success and see if we can reach it. If so, we look to move on to building the alpha version of the chatbot. If not, we look at the necessary iterations. Iterations are common and require time. 

Alpha version of chatbot (three months): this estimate is completely based on previous work I have done with chatbots. There are also questions as to whether or not this should be a mobile app. If so, we would have reviews and ratings that would provide useful metrics in evaluating the product. A mobile app could also message the user at times they prefer, making it more likely that they continue to interact with SonderMind. Engineers would have more to say about how long this process should be. Here is the basic process:

1.    Develop a lightweight expert systems chat engine that can be generically configured with a conversational sequence. This includes the input format for loading the tutorials. At this point of development it is “headless” and can be strapped with tests.

2.    Develop the actual tutorials and load the engine and test. This part involves the back and forth of proofreading/working out small errors in the data, etc. akin to what often happens with new spreadsheets. This is still “headless.”

3. Integrate the tutorial chat engine to the website and/or mobile app and test.

Major Milestones and Metrics

MVP: change in NPS score, change in optimism levels reported to chatbot, average number of counseling sessions per month, percent of new users who become customers, retained users, and change in percentage of incomplete signups per month. In each of these cases we are comparing chatbot data to the most recent data for the SonderMind website. If this is not available, then we will A/B test it. 

Alpha Version of Chatbot: measure each of the above MVP metrics, but this time for the alpha version that has machine learning. Closely monitor changes in metrics as the chatbot gets smarter. 

Wireframes

I would need a therapist to help me with specific questions, but in general I think the questions would be in categories such as notable psychosocial history, trauma screening, presenting concerns/goals for therapy, previous therapy experience, and schedule and therapist preferences (male/female, structured/unstructured, directive/non-directive, challenging/supportive, and other specific requests).

Below is a basic text message interaction based on someone with issues with his or her anxiety. A real therapist would have to assess the appropriate content and tone. Machine learning could be very useful in that it could recognize the different tones and content that individuals are likely to prefer. It should be mentioned that some people will just want to sign-up, and the bot would be able to take them through that process. Others are considering therapy and not finishing the sign-up process or have not had their first therapy session. This interaction would be aimed at getting them more comfortable with therapy.