This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
Barnes &Noble was founded originally 150 years ago by Mr. Barnes and Mr.Noble. It was bought by Leonard Riggio about 50 years ago and now it has 600 stores all across the country. Barnes & Noble uses Alteryx testing macro tools to create analysis of retail store tests and one-sided statistical tests; in unusual test timeframes. With the power of Alteryx AB testing and macros, Barnes & Noble can task a single analyst with managing dozens of tests simultaneously and provide business leaders rapid insights with a solid statistical foundation.
Describe the business challenge or problem you needed to solve
Before Alteryx, retail store testing was inaccurate, confusing, and time-consuming. We have a tremendous range of products that we sell at Barnes & Noble, such as gifts, music, toys and we still sell DVDs. There's a lot of things that we can get involved in testing to drive more profitability and drive repeat visits at Barnes & Noble. We can test if Bestsellers would do well in the back of the store or test where a café would go in the store. But to do some really good tests, we need really good data and results, and then we need to do this very quickly.
We wanted to customize the A/B Analysis and A/B Controls tools to Barnes & Noble, something that could be a lot more modular that we could just plug certain numbers in. We did not have enough data to meet a 95% statistical significance level using the A/B Analysis tool. We wanted to measure the results, not the method. The indexed values from Alteryx A/B Analysis tool were too difficult to show business partners. Last, we needed a method to measure a single store performance.
Describe your working solution
We have a group of stores where we do a change. Then we have a group of stores that we're comparing this change to, where we don't do anything. That measurement between them, that's the analysis that we need to provide to our business partners.
Test Analytic Success Factors
What is a valid comparison?
Control selection for comparison
Correct goals and KPIs
Alteryx A/B Controls and
A/B Treatment tools
What is the effect of the test?
Was the effect measurable?
Was the measured effect caused by the test or by something else?
Alteryx A/B Analysis tool
What happens when it’s over?
Were the results significant?
If not successful: test it again, or drop it?
Roll it out?
Why build a macro?
Enter EDW queries directly
Satisfy requests from business partners for specific data
Single-store results could be provided for deep dives into data Multiple macros covering numerous KPIs
Quick selection of test dates and KPIs
Built once, use again and again
Use R and Python to prototype ideas and then bring that into Alteryx via Macros
Using Alteryx’s built-in R tool and expanding the library with additional packages, we can parameterize the analysis wireframe and make it easier to understand to non-programmers. We decided to use the power of Alteryx Macros to incorporate different R packages into our test and learn workflow as well. We used MarketMatching to determine the closest match from one unit to another group of unit, and CausalImpact to measure the effect of the test.
The Barnes & Noble Testing Suite
Determine the goals and the KPIs of any proposed test with our business partners
Test store selection with Store Operations and Merchandising
Determine length of test and operational parameters
Generate control stores using the BN Control Match Macro and send to store operations
Measure test results with Group and Store Analysis Macros
Present results using the BN Test Presentation Report
We have two test analysis Macros to replace the A/B Analysis tool that's built-in to Alteryx. The first one is our Group Analysis Macro, which assumes that we're going to measure the entire impact of the test on every single store in the test group. If we have five stores that are matched to five control stores each, equaling 25 control stores to this group, then it will calculate the results over that five test store group and the results over the 25 controls store group and make a comparison.
Then our other Macro, the Store Analysis Macro, it's another Batch Macro that iterates over each individual test store comparing only to the five matches. This way we can dig into any outliers or anything that's strange and that's going on between any control stores or any test stores as well. And then we send all the results as CSV files into PowerBI.
Describe the benefits you have achieved
When we started using Alteryx we immediately jumped to hyperspace. It was probably five months from going from the Excel methods to building the workflow and then probably another two or three months building these Macros. We had results within just a couple of minutes rather than several hours in Excel and also we could connect directly to our data warehouse.
At Barnes & Noble, we have some people that know R, some people don't know Python, some people will just use Alteryx. And so we can use R and Python to prototype ideas and then bring that into Alteryx via Macros. And that means that we can share ideas within the team. Everyone on the team can adopt those ideas and be able to leverage them easily.
The entire PowerPoint presentation can be found here.