Minimum Criteria for Success (post #11)

In my first post about our MVP, we stated what our problem/solution set was, we identified our assumptions and ranked them, and we built a testable hypothesis around our assumptions. Our hypothesis stated that we believe struggling ASVAB students (military entrance exam) need to improve their math skills through numerous repeated customized interactions with the content. If we offer customized micro-learning (learn something in 5 or 10 minutes) through a mobile app, their scores will improve.

Next, we needed to establish a minimum criteria for success (MCS).

Minimum Criteria for Success (MCS)

You need to establish a MCS in order to decide whether your product is worth building or not. The MCS gives your experiments clarity and meaning. In order to set your MCS, you need to take into consideration the cost and the reward of building the product.

As we dug into trying to set our MCS, things got complicated. There were many moving parts. First, some users were going to be using the app in addition to other types of studying (books, tutors, etc.). Second, some users were obviously going to use the app more than other users. Should we be accounting for these differences?

We decided that people were going to use other resources to help them study for the ASVAB. It was going to be impossible to get any reliable data as to the effect of this. So we were just going to assume that on average people were going about studying in a similar way when it came to outside resources.

However, we could get reliable data in terms of how much the app was being used through Google Analytics. We could have used any number of services such as KISSmetrics, but Google Analytics did what we wanted and they were free. So we went with them.

Here was our MCS: among users who have used the app for at least three hours, 50% of them improved their score improved by at least 5%. Our logic with the 50% number was just that it represented the majority. If our MCS was proven true, then we could say that the majority of users who used the app for at least three hours improved their score by at least 5%. We were forced to rely on intuition here. Our competitors had statements about daily and monthly active users, but not so much on improving scores. Sometimes they would guarantee that you improve your score by 100 points or so, but that hardly meant that’s what actually happened. That was more of a marketing ploy. One of the truly great and simultaneously terrible things about doing a startup is that you don’t have anyone to tell you that you are wrong. We asked Rob. He said sounds good. He was the closest thing we had to a board of directors or a boss.

Prioritization Due to Risk and Difficulty

Generally speaking, companies have to prioritize what assumptions to test, depending on their degree of risk and difficulty

Risk: how risky the assumptions are for the company or product

Difficulty: how much effort you need to make to test the assumptions

Usually, these characteristics can be drawn into a 2x2 table so that high risk and low difficulty are top left, low risk and low difficulty are bottom left, high risk and high difficulty are top right, and low risk and high difficulty are bottom right of the table. The first category to test should be high risk and low difficulty. Second should be high-risk high difficulty.

Okay. So our MCS was

among users who have used the app for at least three hours, 50% of them improved their score improved by at least 5%.

The riskiest assumption was that this would work (the app would improve scores). We would be testing that. It wasn’t going to be terribly difficult to test. We really just needed users who would take the pre-test and the post-test. And use the app for three hours. We were pretty sure that getting users to do all of these things would be somewhat frustrating, and so we honestly thought it was high risk/medium difficulty. Close enough.

How to Evaluate and Learn from your Experiment

MVP experiments will primarily return quantitative data, in the form of numbers that indicate behaviors.

You also need qualitative data from customer interviews in order to make the right decision. Qualitative data helps you understand WHY the customer did what they did.

Putting together all the data will help you figure out whether your MVP experiment was successful or if you need to make some changes and run it again.

As you can probably imagine, getting users to go through our three steps (pretest, three hours using the app, post-test) was trying. We did already have some qualified users from our customer development. We ended up with 150 qualified users, and on average their scores improved 8%. The results met our minimum criteria for success, and we received more qualitative feedback from the MVP.

One of the biggest complaints was that people did not know where to start. Some of the topics were clearly more difficult than others. We just had our topics in alphabetical order.

Most people liked the gamification. In fact, they thought there should be more. Daily goals were (we thought) an obvious addition. You could have goals for number of questions answered, number of questions answered correctly, and number of topics mastered.

Given that gamification was well liked, we could order the topics by level. As in there were level 1 topics (the easiest) and level 4 topics (the hardest). We did not want to say that each user had to go through the same process. You should be able to pick which topics you wanted to answer questions from.

Another gamification idea was to add a point animation. So every time the user go the answer correct, a point animation would fly up through the screen. Like in a video game. And it would be accompanied by the sound of a cash register. When I would explain this idea to users, they would usually smile. So it was worth trying.

Some users were pretty annoyed that we had made the missed questions part of the quadrant a dead screen.

Some users were fairly critical of my math videos. I talked too fast. I was boring to listen to. But we also got a few hundred subscribers to our YouTube channel. We didn’t even care all that much about the YouTube channel. Really we just had one so that we could create the videos and embed them into the app. The subscribers made us think that there was some value to the videos, but the production value was not all that great.

Practice tests. ASVAB students definitely wanted practice tests. That were timed. This seemed to go completely against the micro-learning model. But they clearly wanted full length practice tests.