Gradient recently worked with a client launching a new health and wellness brand. Think of this brand like a gym: the more you go, the better the feel, and the more likely you are to come back.  If you mentally substitute “yoga studio” for this client, you would be in the right place. And like a yoga studio, they were considering three types of pricing structures:

  • Walk In
  • Membership
  • Package

The core challenge was to develop a pricing structure when they launch in November. Their core questions included:

  • Should they offer memberships or packages (or both)?
  • What should the per-session price be for each alternative?
  • What additional services should they offer exclusively to members?

This is a common set of questions for many businesses, but especially so for businesses that are just launching and don’t have any experience yet with their product. For these types of projects, there are five phases:

  1. Design: understand what questions the client needs to answer and design an experiment that will produce the necessary results;
  2. Programming: actually build the experiment in a survey platform;
  3. Data collection: get the right people to be experimental subjects and go through the experiment you’ve programmed;
  4. Modeling: estimate the statistical models that explain why people made the choices they did;
  5. Simulation and optimization: simulate how people would behave under a range of scenarios and identify the optimal scenario given a specific objective

Design

The typical approach for a pricing study like this is a conjoint experiment. In a conjoint (considered jointly), you ask people which of a set of alternatives they would choose, and their responses can be analyzed to figure out why they made that choice. For example, if we were doing a conjoint about cars, we might show people a brown Toyota sedan for $20k and a blue Ford pickup for $25k, and ask which one they’d choose

After collecting enough responses from enough people, we could use sophisticated mathematical models to tell us why people were making which choices. Did they like Toyota more than Ford? Pickups more than sedans? How much more is someone willing to pay for a blue car over a brown car? And so on.

We run a lot of conjoint experiments at Gradient. Large, small, simple, complicated — we’ve seen a lot of different types. But this project had a few different demands that led us to take a slightly different modeling approach

The main issue is that, when you think about it, buying a walk-in yoga session and buying a membership are sizeably different purchases. Same goes for buying a package of five or ten sessions. The “price per session” has a very different interpretation for both the customer and the company in each case. So, we felt it didn’t make sense to treat the pricing structure (walk in, membership, or package) as “just another variable” to throw into the conjoint

The second complication is that we wanted to build up the prices in such a way that package prices were always lower (per session) than walk ins, and membership prices were always lower (again, per session) than packages.

Doing this is very difficult without adding some constraints around the how the different levels are displayed. Typically, a conjoint experiment will randomize what’s shown to respondents such that — if each were seeing three options — some might see two walk-ins and a package, and another might see two memberships and a walk-in, and so on. This would lead to a lot of different conditions to check for, and we decided it would be easier to always show one walk in, one package, and one membership.

Combining these two requirements, we realized that if every choice was between options with the same “named” alternatives, we could use some additional tricks in the multinomial logit model that typically aren’t used in conjoint modeling.


What tricks are they and why aren’t they used? We’ll get into the tricks below, but some modeling features aren’t available (or, easily available) because conjoints don’t have named alternatives. Tying back to the car example, you could in theory use “brand” as the name, but that would require that every choice was between the same set of brands. Typically, conjoint designs don’t make that guarantee (in one choice, a respondent may see Ford, Ford, Nissan, and in another they may see Toyota, Ford, Chevrolet).


Now, all of this is technically possible in a standard conjoint — it’s just a bit harder because you have to include interaction terms between different variables. Using named alternatives allowed us to use alternative-specific coefficients so we could model the fact that “price per session” is very different if you’re buying a membership, a package, or just walking in. We could simply include “price” as an alternative-specific coefficient and get separate utility estimates for the impact of price, conditional on the price structure. This would allow us to say “raising the membership price by $X per session is the same as raising the walk in price by $Y per session”, which is very powerful.

Program

Because our design had a few complications we decided we would program the experiment ourselves rather than using an off-the-shelf product like Sawtooth (which we <3, by the way). We dusted off our HTML and Javascript capabilities and wrote the logic directly into a Qualtrics survey. It’s surprisingly easy!

Data collection

Using one of our trusted survey panel providers, we sent the experimental survey to a very specific sample in the geography our client was interested in.  In this case, we got about 250 responses, which is on the low end for studies like this, but our calculations in advance of collecting the data told us the study would be sufficiently powered at 150N.

Model

Once we got the data back, we estimated the model coefficients using the mlogit package in R. At Gradient, we have a love/hate relationship with this package. It is the only package we know of that can handle alternative-specific coefficients and is the best place to start when analyzing conjoint or max-diff data, but it has some severe drawbacks. But, in this case, we got everything working.

The analytical results of conjoint experiments are typically individual-level utilities which are essentially the weights that each respondent places on different components. Going back to the car example, each person would have a different utility for “Toyota” and “red” and so on. What are these magical things? This post can’t get into all of the specifics, but in short they are mainly useful for simulating new scenarios, which we discuss below.


What’s an alternative-specific coefficient? In discrete choice experiments, there are three types of coefficients that are commonly estimated:

A: Alternative-specific variables with generic coefficients

B: Individual-specific variables with alternative-specific coefficients

C: Alternative-specific variables with alternative-specific coefficients

This is a lot. Let’s unpack it.

Say you’re trying to understand why people take the subway, bus, or Uber in a city like New York. Assume you have a dataset and for each datapoint you have 1) the time it would take for each option to complete the journey, 2) the price, and 3) the income of the person taking the journey. And, of course, you have which choice they made.

So, let’s start with time. All things equal, the more time it’ll take to complete the journey under one option, the less likely you are to choose that option. But, not all time is created equal. Time on the bus may be more costly than time on the subway, which is more costly still than time in an Uber. In an Uber you can make calls, stretch out, etc. On a subway you may have to stand, etc. So time, in this case, is an alternative specific variable (it changes with each alternative) but it does not have a generic coefficient. Time should have a coefficient for each of subway, bus, and Uber. So it is type C, an alternative-specific variable with alternative-specific coefficients.

Price, however, is type A. Money is money no matter where or how you spend it. So it is an alternative specific variable (each alternative will cost a different amount) but with a generic coefficient.

Finally, let’s talk income. A person’s income doesn’t change if they take the bus or subway, so we say it’s an individual-specific variable. Why does it have an alternative-specific coefficient? Well, people at different income levels may have different preferences. E.g., a person with a high income may prefer Uber to a bus to an even greater degree than someone with lower income.


Simulate and optimize

The magic of individual-level utilities is that they allow us to estimate what a given respondent would choose among an arbitrary set of alternatives. How? Let’s say the choice is — as above — between a brown Toyota sedan for $20k and a blue Ford pickup for $25k. Thankfully we have this respondents’ utilities for “brown”, “Toyota”, “sedan”, “$20k”, “blue”, “Ford”, “pickup”, and “$25k”. The utility of the first option is the sum of the utilities for brown, Toyota, sedan, and $20k, and the utility of the second option is the sum of blue, Ford, pickup, and $25k. The probability that the person would choose the Toyota over the Ford is estimated as the softmax (or logit) function of the two sums of utilities.

Importantly, there’s nothing special about this specific choice; we could just as easily simulate any combination among the levels that we’ve modeled.

So, let’s say that our client wants to know “assume we offered one walk in, one membership, and one package option, what should they be to maximize our revenue?” We loop through every possible combination of these three options, and model what each respondent would do given the choices we’re considering. Since we know how much revenue is gained when each option is purchased, we can score each combination with an expected revenue.

Similarly, a our client may ask “assume we offered two membership options and a walk-in, but no packages, and we want to optimize market share, not revenue”. Well, the approach is almost the same: loop through every possible combination of two memberships and a walk-in, and model what each respondent would do given the options we’re considering. Only this time, for each scenario, we tally up the percentage of respondents that chose something vs. nothing, and use that as our score for each scenario, as opposed to revenue.

Conclusion

The power of simulation is that it can surface things that you would never be able to uncover otherwise. For example, you may learn that it makes sense to price packages and memberships very far apart, but packages and walk-ins closer together. How could this happen? Don’t you want to drive people to buy packages over walk-ins?

Yes, but, since we’re modeling each respondent individually, the simulation will account for heterogeneity in preferences. There may be a segment of people that don’t have significantly different sensitivities to package vs. walk-in prices, and will choose a package if it’s any cheaper than a walk-in, but there’s also a large segment that is very sensitive to the membership price, and will only choose that if it’s priced very low. It would be very difficult (perhaps impossible) to consider all of the different possibilities of heterogeneous consumer preferences, but a simulation accounts for them all as it models each individuals’ choices.