Founder Stories — About Project ETOPS
The hidden gem in naming our company OpenWater was the instant name generator when it came to new projects.
When we built a system to help our sales team find whales, we quickly came up with the name Orca. The naming continued as we built SONAR, Tugboat, Islands, Hull Swap, and the Ship Yards. I plan to share the story behind each of these tools, along with the OpenWater name; but today is about ETOPS.
What the heck is ETOPS? ETOPS stands for Extended-range Twin-engine Operational Performance Standards. ETOPS is a certification used in the aviation world to help ensure planes don’t crash when they travel across the ocean, just because one engine went out.
Prior to ETOPS, the only planes that were allowed overseas were the gas guzzling 4 engine behemoths. The idea behind ETOPS was that if a plane has two very reliable engines, and each engine was maintained by distinct teams, the likelihood of a two engine failure was so low that the government agencies would deem such flights airworthy.
Each year, in January, the OpenWater platform takes a beating. Hundreds of awards programs around the world have their submission deadlines resulting in millions of entries flowing through our system in a short period of time. This January surge caught us by surprise the first year and managed to knock our systems totally offline for a few hours.
Our solution then was to get the biggest and best server we could afford. Our thought process was that if we could run one server at 10% capacity, we could kick the can down the road until we had a better idea.
By the time January 2014 rolled around, this one server was hitting 30% during peak usage. As each year passed, I breathed a sigh of relief that my solution was holding strong. January 2018, this server peaked at 60% and I began to worry.
Even if we made it through January 2019, I knew by the time 2020 rolled around, it might be too late. When tech companies kick the can down the road sometimes it turns out to be more like a massive kick in the ass.
A relational database is one of the hardest components of a platform to upgrade. It’s old technology that is from a time when people were used to downtime. Sure if we could just turn off the server for 2 days no problem; but in today’s world our customers rely on a platform that just works.
Microsoft developed a modernized version of their database engine called SQL Azure Elastic Pools. As the name implies, these elastic pools can stretch to meet capacity as our needs grow.
Moving to SQL Azure was a must do move for us to solidify our business for another 10 years of expansion. This was also a must do move before January 2019, in fact it was a must do move during the summer of 2018 while our usage is at its lowest point.
The entire summer was spent planning for and testing our approach. We ended up executing the procedure over labor day weekend.
I called this project ETOPS because it reminded me of the extreme care that we needed to take at each step. I also knew if we failed, we would have a catastrophe. Some customers would lose data, others would face significant downtime. Besides our customers being frustrated, our support staff would feel disgruntled at exactly the time they need to be on their A game.
I would like to say, if we got this wrong we wouldn’t survive as a business. As such, I’ll defer to the other way ETOPS is described by pilots, which is:
Engines Turn or Passengers Swim.
