The one week in 2012 that shaped my career, A/B testing led growth and a culture of bottom-up leadership

It was 2012, I just boarded a plane from Israel to Amsterdam and was sitting nervously thinking about the week to come. I did consulting at the time for a start-up that had a SAAS product allowing companies to watch anonymous recordings of how visitors used their website, and see heatmaps of mouse moves, time spent, scroll depth and more helping to analyze customer behavior and optimize performance based on it. My role was to lead our Enterprise accounts and help their leadership teams to improve their KPIs by using our product's insights to reach their business goals (sell more clothes/hotels/insurance/media/etc).

One of my start-up's customers was Booking.com, they worked with one of our consultants and cancelled their contract at the annual mark. Booking was at the time the most advanced and innovative travel website in the world and all other online businesses looked up to them. In a last attempt to salvage the account, our Enterprise Sales lead convinced Booking to let someone come over for a week to work with their teams and try to change their opinion for free, no strings attached. I felt the heavy weight on my shoulders sitting on that flight being the person who needed to win them back.

I came to Booking's office on Monday morning to meet two of who I discovered to be some of the smartest leaders in online representing UX/UXR and product. The Dutch culture is very similar to the Israeli culture I grew up in - direct, non-apologetic and to the point. I was taken to a 2 hour meeting to start the week where the 2 individuals walked me through how terrible our product, service and overall outcomes were and how we just didn't fit into how they were doing business. While I was sitting uncomfortably, taking notes and nodding, I reminded myself they agreed to have me for a week in their offices, so there was still a chance to change their minds.

At the time, our start-up was one of the very first software's of its kind in the market, and we worked with most fortune 100 companies and all leading travel websites in the world. I was used to sitting in clients' meeting where I'd see many stakeholders debating the ideal design, I was regularly walked through detailed redesigns that were planned quarters out in advance. The concept of waterfall vs agile was well known already and they worked in more of an agile style, but decisions were debated and decided on by leadership and execs, and the notion of fast A/B testing was a tool and not a culture. The most advanced players in the market back then (outside of Facebook/Google/Amazon at the time), ran 5 a/b tests in a quarter, that was considered very fast.

As I stepped onto Booking's huge open space, the 2 individuals walked me through the floor. There was a very large numbers of pods/teams that set in groups of 10, each group had a product manager, designer, data scientist and several engineers. Each one of these groups was responsible for a certain area of the site with specific targets against that area, and each group was running dozens of A/B tests in parallel to meet these goals. Booking was running back in 2012 hundreds of A/B/Multi-variate tests in parallel, today they likely run thousands of experiments at any given point in time. I asked how they ensured all the tests weren't clashing and they walked me through their in-house statistical tools that ensured the experiments are measured independently, and allow them to analyze each experiment in detail.

At Booking, leadership wasn't involved in what the pods did - whether it was a big new feature, full redesign or a small test. There were not a lot of debates or meetings, the pods members scrolled over in their chairs to each other and chatted live on the floor, the goal was just to built and test as fast as possible. It didn't really matter if someone thought the photos of hotels should have a certain layout or another, or whether they should be of certain size or another, and have labels or not, etc; it only mattered how fast they could test it to reach the outcomes they were going for.

Booking established an unheard culture at the time of bottom-up democratic leadership and experimentation. One of it's core principles was "Anyone at the company can test anything—without management’s permission."

Booking made several strategic decisions that allowed it to take over the travel industry by a large margin leaving competitors far behind, culture of experimentation was one of those, and I would say likely the most important one. Booking recognized that unprecedented transformational outcomes would come from incremental growth and speed, and speed comes from giving autonomy, empowering teams to be curious and data driven, setting the right goals, and stepping out. That type of culture, can only come from leadership having no ego, embracing humility and letting data be the only thing that drives decisions in the company.

Today, as a leader in my org, I constantly remind myself - the direction we need to go in should be the right one, the things we need to focus on need to be the right ones, and the metrics we choose to optimize - are critical for success, but what tests the teams run and what design choices they have - is going to be determined fastest by A/B testing and not via debates or guidance - as such - I need to get out of the teams way, give autonomy and create a setup that allows for fast experimentation. It doesn't matter how many tests I've seen across customers we consulted to, and it doesn't matter how many companies I worked in or what I learned - my opinions can be proved right or wrong in a second by real visitors, and they're the only ones that matter.

You might tell yourself - but can't my experience be helpful to provide guidance in the right direction? What if I disagree with the test - won't I save them time by providing feedback early? The answer for me is - no. There's cost to prep for leadership reviews, debates and alignment, if you're heading in a direction of running hundreds of experiments in parallel - and that's what is needed to drive significant growth - you have to empower the teams to move fast and test fast independently. When my team asks me what version we should go with - I usually say - test both, and lets see what wins, regardless of the opinion I have.

That week at Booking's office in 2012 and the partnership we forged in the following years, instilled in me a deep belief in testing that shaped the way I work and lead over a decade letter. It took me a long time to connect the dots on why I felt so strongly about this topic, every time I saw organizations debate tests they should or shouldn't run, I couldn't avoid thinking - how much money is the company going to miss in this direction. I was eventually able to track it back to Booking - once I've seen their method of working - there was no going back.

Booking ended up renewing their contract, in that week I worked together with their team to show the value of adding visual recordings and heatmaps to their toolset to truly understand why an experiment behaves in a certain way - i.e. what members experience in that experiment - that testing data alone doesn't provide.

When I finished writing this blog, I googled to see if anyone wrote a case study on Booking; I came across a Harvard Business Review talking about Booking's culture and how it transformed their business, and I found it to be a good and very close articulation of the experience I had, check it out.

#productmanagement #leadership #growth #productgrowth #datascience

Opinions are my own only!