Hey - I work in design innovation for a very large online travel company (e.g., Expedia, Priceline, Booking). Here are some additional thoughts (1) Don't get obsessed with statistical significance. 95% is an arbitrary number - sometimes you need to make decisions and a 70% probability of one option being better, while not ideal, can be enough to support a decision. (2) Volume! You hear of case studies where people changed a color and conversion went up 20%, but that rarely happens in the real world. With smaller improvements you need A LOT of volume to get statistical significance, so think about what measurement really matters - sometimes there are proxies for conversion that require less traffic (e.g., click through vs. conversion) (3) Tradeoffs and attribution. It's easy to make conclusions like "this element increased conversion - so it's good" but sometimes page elements can improve a desired outcome at the expense of other things. For example, we did an A/B on a hotel page that increased conversion on that specific hotel, so people concluded it must be good, but it came at the expense of cross-selling other hotels.