Home

In the movie Air Force One, you might remember the scene where Harrison forward is bobbing around in the sky. His plane lost control and crashed into the ocean as another air force plane, Liberty 24 pulls him in.

As he boards the plane, we hear the pilot radio in to eagerly awaiting Washington, “Standby….Liberty 2–4 is changing call signs, Liberty 2–4 is now Air Force One”

Of course we all know the majestic blue and white plane only takes on the name Air Force One when the president is on board, otherwise it is known as SAM 26000 (Special Air Mission).

I’ve mentioned earlier how important a name can be framing how one things about a problem. Today’s story talks about how we took this concept of a name changing to help us solve a serious engineering problem.

When you use a website and click a link, you expect that website to do something right away. However what if that link is something like Facebook’s “Download all of my Data.” User expectation for something like that is some process will run in the background, and after some time you’ll be emailed the results.

The technique of running a process in the background is certainly something you could take for granted, but it’s the kind of thing that keeps websites alive when done right.

When we first rolled out background tasks and reports in 2012, we used a very naive method. We would run reports in the same thread pool as the main web request, which meant as we hit high web traffic we’d end up with reports simply not running.

In fact by early 2014 we started getting support tickets claiming reports simply weren’t running. Turns out those customers were right, about 30% of the time when you’d ask OpenWater (then called awardsCMS) to run a report you’d come back with nothing. Sometimes you’d only find out after hours of waiting that there would be nothing coming.

Around that time we scrambled for something that would help and Zack came across a tool called Hangfire. Hangfire intelligently ran jobs in the background and would retry them in case something went wrong. After a few months we had Hangfire implemented and without much fanfare complaints about jobs failing dwindled down to about 0.

2014 was also noteworthy because it was the year we re-branded from awardsCMS to OpenWater. This name that I was originally against, ended up being the backbone of how we build systems at the company.

The first tool we built in the OpenWater era was SONAR. SONAR is undersea technology that sends audio pings to receive data about objects. We were using a variety of software systems to keep our business going. Besides the platform itself, we needed customer support software (Zendesk, then, Intercom now), sales tracking software (Salesforce), and billing software (Xero). SONAR was the tool we developed that constantly pinged all of these systems and allowed for data sharing between then. For example if a sales person wanted to know which support rep was working on the customer they signed, SONAR would come through and link that information for them.

If you have been to Seaworld or seen Blackfish you know all about killer whales. In sales terminology, whales are the “big accounts” so for us, we needed a cost effective way to find out who these people might be and how we could reach out to them. Our sales research tool was aptly named Orca.

In 1990 spy thriller, The Hunt for Red October, a Russian commander piloting a top secret high tech submarine defects and transfers this technology to the U.S. When we needed a tool for our competitor research tool, Red October quickly became our code name for this project.

There are only two hard things in Computer Science: cache invalidation and naming things.

— PHIL KARLTON

I’ll leave the article for cache invalidation for another day, but Phil Karlton, a master of design and computer architecture was spot on when he said that naming was an insane challenge. Besides computer science, the social scientists at Freakonomics taught us how a person’s name can change their whole outlook and lot in life.

After we came up with all of these fun names for our internal tools we set out a rule — we can’t start on a new project until we come up with an appropriate naval related name for it. We realized that if we had the right name for something, we would build it following the appropriate metaphor. 90% of computer science is thinking, so if we spend a lot of time thinking about a name it also means we spend a lot of time thinking about why something has a name.

Our software operates in the public cloud, we mostly rely on Microsoft Azure for our computational workloads. I mentioned earlier how the backend of our platform runs on Hangfire, the front-end is powered by web servers running on Microsoft Azure App Service. When our servers are under extremely high load, some of our web servers would crap out. Since we always run multiple copies of our software, if one server was acting funny we would get reports along the lines of “the page seems to take forever to load, then its fast, then its slow again, very bizarre.”

Under high load, if a server starts acting up, Azure App Service is supposed to take the server out of rotation, restart it and bring it back online. This was the earlier days of the cloud though, so in some cases a node would get stuck — it would be dead, but not yet be taken out of rotation. This created the phenomenon described above. The only solution at the time was to stop the entire service and start it, doing so would guarantee us a fresh batch of new servers. The problem is under high load if we stopped the service it would mean a full 10–15 minute outage when several thousand people are using the site, it was better for us to hang on with intermittent complaints than an avalanche of anger hitting our support lines.

In the military a Hull Swap is an exercise where the crew of one ship finds its way onto another ship. Often the vessel they are on is need of major servicing and needs a new crew to do that work. To stay ready for operations they need to get onto a new craft.

Hull Swap was the perfect name for our tool to help us fix the stuck node situation. We realized if we could transfer all traffic from the impacted environment, restarted that environment, and then sent traffic back once all was fixed we could get around the stuck node issue without any downtime.

We are now in 2019 and Azure App Service has come a long way. The idea of stuck node isn’t something we really have to worry about anymore. The technology we built for Hull Swap however lives on with a new purpose.

From 2012 to early 2018 just about every morning my routine was as follows:

• Wake up and check for new updates that are ready to be rolled out (e.g. bug fixes)
• Review the code and then copy it to App Service 1
• I’d then shave, roughly a 6 minute process, and then begin copying of the code to App Service 2
• I’d hop in the shower, roughly 10 minutes, just in time for the copying to be finished
• I’d then finish updating our platforms and doing some quick checks

Major releases were even more hectic, I would wake up around 2 to 4 AM and begin a checklist that could take anywhere from 2 to 10 hours. Often I would have to give up a Friday or Saturday night (or both) each time we rolled out an update. Not the best way to spend my late 20s, but I was determined not to let this process enter my 30s.

Using the technology of Hull Swap + a few modifications we implemented our latest tool. After my morning code review I now click a single button and Tugboat begins its journey taking our code from the staging environment to our live environment. Tugboat then performs a Hull Swap with our live environment. With no downtime we get updates sent to the platform every morning. Major updates can now be done during slow periods (e.g. Sunday morning), instead of the middle of the night.

With Tugboat in full operation and Azure App Service working really well, I could start to add back normalcy to my morning routine. One of the first changes I made was to make sure I hit the gym in the mornings. As I could clock in each month of stability and perfect up-time I started feeling comfortable working out without my phone.

Big mistake.

On a Thursday morning during February, I return to my locker. 2 missed calls from Zack. I load up slack, I see our support chat room blowing up. This can’t be good.

I get Zack on the phone:

• Zack: Hey man…none of our background jobs are processing
• Kunal: What do you see in Hangfire?
• Zack: Hold on let me look….
• Zack: Oh s*(@!. 4,000 jobs and its increasing as we speak

It looks like that stuck node problem had a cousin and that cousin didn’t like Hangfire very much. The way we fixed this was a similar process to the stuck node problem, restart the servers and wait it out.

This problem happened a few more times during our busy season and then hid away. The best we could do was detect when it was about to happen and then implement the manual process. February ended and we went back to our smoother months. March, April, May, June, all went by fine. By the time the summer hit, we forgot we even had this problem.

Then seemingly out of nowhere the detector lit up in early September. Fortunately we were at hand to quickly solve it. Then again it happened, this time at around 3 AM on a Wednesday. We were not so lucky, it wasn’t until around 7 AM until one of us noticed and fixed the problem. Our European customers were not too happy.

We had to do something about this. We learned that the people who maintain Hangfire rolled out an update for this situation. It would take us about 3 months to implement the fixes in OpenWater, so what would we do in the meantime?

We got to thinking — of course we started with the name. We started talking through our current solution. I can’t remember which one of us said what, but here are some of our notes

• “Hangfire really needs help when its under heavy load”
• “Yea, its like when it is about to die it needs to make a call for help”
• “Dude, Save Our Sever, F*&!@ SOS”
• “OMG, SOS! Totally perfect name”
• “But who would save it, we’ll be asleep”
• “Dude… Tugboat”
• “Dude…”

We realized our emergency procedures altered Tugboat to act like Hangfire. That would buy us a precious 5 minutes of processing time while the main Hangfire servers were restarting.

Like Liberty 24, we decided Tugboat could change call signs and take on a new mission.

A few weeks later SOS was implemented and both Zack and I were both travelling to various onsite meetings.

Then this happened:

Hangfire died and came back to life. No intervention needed.

Mission accomplished!

About 2 weeks before Kanye West rose to internationally acclaimed fame, he came to George Washington University’s “Spring Fling” (undergraduate spring concert) and performed several soon to be hits alongside John Legend.

Among his songs was “Kanye’s New Workout Plan.” After a few loops of “eat your salad no dessert, get that man you deserve” he stopped the music, grabbed the microphone and proclaimed, “who better to remix this song I made, than me!” He was right, spun up a fist pump evoking remix that made the students on the university quad go crazy (precursor to going nuts / ape-s****).

Kanye is an asshole for sure, but he is also a master at his craft. In his song Spaceship he goes on to say, “Lock yourself in a room doing 5 beats a day for 3 summers.” He knew hard work and consistent work is what leads to mastery.

I’m not a musician, but I have to imagine there must be that moment, that ah-ha, I got it, this is it that the Kanyes of the world must feel when they get the beat just right.

But is it an ah-hah? Or is it something deeper within us?

I started thinking about one of my favorite musicians and the journey he must have taken to realize this 3 second clip is the ah-hah for his puzzle. Let’s hear it together.

Play Sample

Do you recognize it? How about this clip:

Play Sample

It is the theme song to the original 90210 clip from the 1990s era TV show.

So what exactly were Thomas Bangalter and DJ Falcon thinking and how long were they working on their song Together?

The song itself starts with a question, “Are we in this thing alone or are we in it together?” Together debuted in August of 2000, a repetitive loop with the single word Together. How many times did this loop play, and how many different samples did they have to play until this 3 second clip hit them like, YES.

…here is the full track.

Sure, I get it, Together may not be your jam.

Thomas Bangalter produced music as part of a few other groups. “Ooh baby, I feel like the Music Sounds Better with You…” — that international 1998 hit, Thomas Bangalter (Stardust).

Music’s got you feeling so free, you have to celebrate? Celebrate and dance so free? One More Time, an instant classic from 2000 by Daft Punk (Thomas Bangalter and Guy-Manuel de Homem-Christo).

Who can create a better remix than the masters of the original art themselves? In 2007, Daft Punk toured the world, and put on this finale.

Human After All, Together, One More Time, Music Sounds Better Than You — riffs from each of these songs pulled together into 10 minutes of nostalgia and excitement for their fans.

About halfway through, 5 minutes of crescendo build up and the duo drops the 90210 beat as the opening to the second crescendo bringing back their mega-hits like One More Time and Music Sounds Better Than You.

The crowd goes totally nuts, one more time, as they close out their concert.

When Zack and I decided to go forward with Api2PDF, Tim opted out. This meant we could not use OpenWater resources to build out our idea.

In the old days, I would design the UX, then I would write a coding plan, I would code myself, then figure out how to launch the product. Tim figured out how to sell the product and after he sold it, he was also the first support rep we had.

Over the years we have been fortunate and seen our teams grow. We now have a full team dedicated to support, another team dedicated to sales, and yet another who who is in charge of building the product. It is to the point where I can discuss my vision with our software architect and UX designer and a few months later the features appear in our software.

We wondered, without company resources at our disposal, would we have what it takes to stay motivated? Would we be able to stay focused enough to build a minimum viable product and would we be energized enough to actually get it passed launch?

We went back to our roots. How can we build something from nothing? We thought through the components and our expertise:

• (ZACK) Billing System
• (ZACK) Customer Portal
• (ZACK) API Usage Meter
• (KUNAL) Website to Advertise Ourselves
• (KUNAL) The Actual Convert to PDF API
• (ZACK) Implement Kunal’s Crazy Idea

We got started and set aside a small budget to help us offload work to affordable overseas developers. After a few weeks of moonlighting, we were ready.

Our goal with Api2PDF was to make PDF generation so cheap that we would take the bottom out of the market. To make that idea work, we needed to get the best of the both worlds of serverless technology. Serverless, at least in theory, means you pay only for the compute power you use, by the second.

Amazon Web Services (AWS) provides a serverless technology called lambda. Microsoft Azure provides one called functions. The crazy idea was to split the workload across both providers to get the high power of AWS with the accessibility of Azure.

We were ready for our test day, Trinity.

Zack arrived a few minutes before me on a Saturday morning. He had all 3 of his monitors fired up with the various components that we needed to glue together.

On one monitor, Visual Studio, fired up in debugging mode, running the web portal; on another monitor Azure Functions console, where we could see PDF processing in real time; and in the third a Node.JS test app ready to battle test the whole apparatus.

He clicks GO .

Data starts flying across the screen:

• PDF Generated (0.000021 billed)
• PDF Generated (0.000011 billed)
• $1 Monthly Charge Billed
• Auto Renewal Hit
• Large PDF Generated (0.030011 billed)
• Large PDF Generated (0.500011 billed)

Thousands of messages like the one fired off, we stopped the experiment and shared high fives and a few expletives.

Trinity was a success!

I wondered out loud if how we felt watching numbers fly across the screen gave us the same feeling that Dr. Feynman felt during trinity. (Trinity was the first test detonation of the atomic bomb. While the bomb overall has been controversial, when trinity happened all of the people working on the project basically were like “holy shit, this thing actually works.”)

I believe this feeling of wanting to know, can it be done, more than anything else is the secret sauce behind invention and art. The ah-hah is just the first step, the first taste to wanting to create more.

*As a side note Kanye West / Daft Punk collaborated on Stronger. It was actually a collaboration, not just Kanye sampling. Daft Punk re-composed their loops of Harder*Better*Faster*Stronger.

The hidden gem in naming our company OpenWater was the instant name generator when it came to new projects.

When we built a system to help our sales team find whales, we quickly came up with the name Orca. The naming continued as we built SONAR, Tugboat, Islands, Hull Swap, and the Ship Yards. I plan to share the story behind each of these tools, along with the OpenWater name; but today is about ETOPS.

What the heck is ETOPS? ETOPS stands for Extended-range Twin-engine Operational Performance Standards. ETOPS is a certification used in the aviation world to help ensure planes don’t crash when they travel across the ocean, just because one engine went out.

Prior to ETOPS, the only planes that were allowed overseas were the gas guzzling 4 engine behemoths. The idea behind ETOPS was that if a plane has two very reliable engines, and each engine was maintained by distinct teams, the likelihood of a two engine failure was so low that the government agencies would deem such flights airworthy.

Each year, in January, the OpenWater platform takes a beating. Hundreds of awards programs around the world have their submission deadlines resulting in millions of entries flowing through our system in a short period of time. This January surge caught us by surprise the first year and managed to knock our systems totally offline for a few hours.

Our solution then was to get the biggest and best server we could afford. Our thought process was that if we could run one server at 10% capacity, we could kick the can down the road until we had a better idea.

By the time January 2014 rolled around, this one server was hitting 30% during peak usage. As each year passed, I breathed a sigh of relief that my solution was holding strong. January 2018, this server peaked at 60% and I began to worry.

Even if we made it through January 2019, I knew by the time 2020 rolled around, it might be too late. When tech companies kick the can down the road sometimes it turns out to be more like a massive kick in the ass.

A relational database is one of the hardest components of a platform to upgrade. It’s old technology that is from a time when people were used to downtime. Sure if we could just turn off the server for 2 days no problem; but in today’s world our customers rely on a platform that just works.

Microsoft developed a modernized version of their database engine called SQL Azure Elastic Pools. As the name implies, these elastic pools can stretch to meet capacity as our needs grow.

Moving to SQL Azure was a must do move for us to solidify our business for another 10 years of expansion. This was also a must do move before January 2019, in fact it was a must do move during the summer of 2018 while our usage is at its lowest point.

The entire summer was spent planning for and testing our approach. We ended up executing the procedure over labor day weekend.

I called this project ETOPS because it reminded me of the extreme care that we needed to take at each step. I also knew if we failed, we would have a catastrophe. Some customers would lose data, others would face significant downtime. Besides our customers being frustrated, our support staff would feel disgruntled at exactly the time they need to be on their A game.

I would like to say, if we got this wrong we wouldn’t survive as a business. As such, I’ll defer to the other way ETOPS is described by pilots, which is:
Engines Turn or Passengers Swim.

The year was 2006. I was 21 years old and fresh out of college with a newly minted degree in computer science. While I had acquired a stable job with the government, at night time I would scour the Gigs section on Craigslist to fuel my side hustle. I would reach out to anyone looking for help building software or databases. The plan was to acquire enough customers of my own to begin a full-time software consultancy.

The first major project I landed was PriorArt.com — a patent search and discovery firm. PriorArt was drowning in reams of paper that was the result of thousands of faxes, printed emails, and legal documents. I had to provide a mechanism to store all of this information into a centralized, web accessible, online system. I was naive and had no idea what I was getting myself into with this project or the challenges I would face.

The greatest obstacle to building this “paperless office” centered around PDFs. Every physical piece of paper PriorArt dealt with needed to be converted to a PDF in this online system. The PDFs were heavily used for “Docket Agreements” — mini-contracts or statements of work.

To make things more complicated, one requirement I had to meet was that the system needed to run in a shared hosting environment. The mid-2000s was a transformational time for web development. It was still very much the wild west. Amazon Web Services hit the market, but it would be several years before I would pay much attention to it. All I knew about shared hosting back then was that you would pay $10 per month and you would be allocated some space on that server to run some code. Security and server isolation were distant thoughts.

My first attempt was to use a component called ABCPDF, miraculously still for sale. As with most .NET components back in 2006, ABCPDF required some wrangling to get the DLLs running in a shared hosting environment. EasyHosting, the hosting provider at the time, rubber-stamped my custom DLLs and we were off to the races. PriorArt was generating PDFs.

Barely two months go by and providers started to wise up to the security holes posed by allowing custom DLLs in shared environments. Microsoft issued guidance on a concept called Medium Trust which ensured that neighboring customers on the same server could not interfere with each other. ABCPDF required Full Trust, so that was the end of that.

My backup plan was to run PDF generation on a shared host using PHP. I turned this into an API only accessible to the .NET platform. After a lot of trial and error, I came across https://tcpdf.org/, a library that persists to this very day. At the time, TCPDF, provided HTML to PDF conversion, but did not allow the use of external stylesheets. It was immensely painstaking, but I was able to convert all of my templates into a format suitable for TCPDF.

PriorArt was live and I was hopeful I would never have to go through PDF hell again.

That wish did not come true, but I did succeed in leaving the government and started my software consultant business, vOfficeware and then later nonprofitCMS. In 2009 I was approached by Cambridge Information Group (CIG), a private equity firm that owns several education companies including B2RMusic.com and Sothebysinstitute.com.

My PDF journey continues. I had to generate PDFs of statements and invoices for students and their parents for the online portal of this music school. CIG had their own servers though, on premises. I was back to using ABCPDF with Full Trust and I forgot I had ever faced any PDF problems at all.

Fast forward to 2012. By now my company had launched its own product — an application and review platform that helps organizations run their awards, grants, and scholarships. We rebranded as OpenWater and left consulting work behind.

Part of any applications process requires a form where an applicant submits their information, and will often have the ability to upload files, such as transcripts or letters of reference to produce a full application package.

Once again I was thrust into PDF purgatory. When OpenWater first launched, we were running on Mosso, a cloud-based platform that allowed for Full Trust packages. We decided to go with the open source iTextSharp library to handle our PDF needs. Almost immediately after we launched, we found out iTextSharp was pivoting their licensing model and would no longer be free for commercial use. We stayed on their old version as long as we could until a suitable replacement came out.

WKHTMLTOPDF fell into our laps like a gift from the heavens. It was the rendering engine of Google Chrome and Safari (WebKit) that created reliable PDFs time and time again. By this point too, the idea of renting a virtual machine from a cloud provider became affordable. We were able to swap out iTextSharp with an API call to a secret VM that would sit idle and generate PDFs as we needed it to.

As we expanded OpenWater, we continued to find new ways of linking up to this secret API. Having a VM dedicated to producing PDFs saved us many hours when we had to add PDF functionality on any new project. However, we were headed down a path we would later come to regret.

As the success of OpenWater continued, our PDF needs would increase as well. We went from generating dozens of PDFs per day, to hundreds, to thousands. In 2016, we rolled out a feature to let our users download their data to PDF in bulk. These were not your average PDFs. In many cases, the PDFs could be thousands of pages long. We saw our PDF usage explode. Tens of thousands of PDFs were pulled daily! Our single VM dedicated to producing PDFs was getting crushed, would shut down and require restarts. With so many projects depending on this single point of failure, everything was coming to a grinding halt. Customers were upset and could not understand why our system struggled to generate what was a simple PDF.

We were able to temporarily mitigate the issues with some combination of scaling the server up and out. We found we were stable again at around a burn rate of $400 per month to keep this now cluster of VMs running.

This reprieve was short-lived. We rolled out the ability to include images and merge multiple documents into a single PDF. With the increase in both PDF size and raw processing power, our PDF servers could not keep up with the load. We could not even get them to respond to a reboot request.

We scaled up to the max, and now our PDF generation costs surged to nearly $4,000 per month, but the inundation kept coming.

A few customers ended up being a chokehold for our entire platform’s PDF needs. We had no choice but to cut off these customers from the PDF engine. To help these customers out, we had to work through the night to generate their PDFs using our local computing resources so they could make their deadlines.

Our customers have come to expect reliable PDF generation and do not understand what goes on behind the scenes to produce massive PDF files at scale. Admittedly, when we told our friends about Api2Pdf.com, their first reactions were “I can just print to PDF, how hard can it be?” We knew that $4,000 per month was not sustainable for a steady state server that will be used sporadically for PDF generation.

Over coffee, my colleague and Api2Pdf co-founder, Zack Schwartz, proposed the idea of using Amazon Web Service’s Lambda service for our PDF generation needs. On paper it seemed like the perfect solution. AWS Lambda would give us computational power for each PDF request, independent from the next request. If we have 1,000 people needing PDFs all at once, AWS Lambda would give us the resources we need on the spot.

Over the 2017 winter holidays, we set out to rebuild our PDF virtual machine cluster into a set of AWS Lambda functions and roll them out before our January busy season kicks in. If we could build a drop-in-replacement for WKHTMLTOPDF, we could always just revert back to the $4,000 per month solution.

Throughout January 2018 — May 2018, we continued battle testing our lambda functions. The result worked out way better than we could have ever expected. Our PDF generation stayed up throughout the entire busy season and our costs dropped from $4,000 per month to a range between $31 and $61. We generated several hundred gigabytes worth of PDFs.

By March, we realized we were on to something. If we put a bit more effort, we could offer a tool to all developers at a nominal cost. PDF Generation really sucks — there are a lot of great resources out there and a lot of free ones for those developers who have the DIY mindset. We wanted to create a tool for those who just don’t have the time or are overworked and have too many priorities. In the end, customers just want things to work.

Api2Pdf.com launches today and is a wrapper around popular tools most developers already use today:

• WKHTMLTOPDF
• Headless Chrome
• Libre Office
• Merge / Concatenate PDFs

Our goal is to simply make it easier to use the wheels that already exist at the lowest cost possible. For about $1.50/month, we think that 90% of developers can have their PDF needs fully met. We also believe our service can scale for the remaining 10% that have heavy duty PDF generation needs.

Cheers!

Back in 2007, when I was still operating as ComputerHelpDC I created a quick database called ScheduleC to help keep track of Schedule C expenses that could be written off taxes.  With the start of vOfficeware we saw the need to bill clients in a recurring manner, so I decided to build a generic invoicing platform that others could use + allow us to eat our own dog food and use it to bill clients.

As we continued to grow, we had to start to track revenue in different codes. Instead of refactoring vOInvoice (which was used by perhaps 40-50 clients — but very few who would pay), I built another system we called “secure vo” — simply because it was hosted at secure.vofficeware.com.  The key difference in functionality was that while vO Invoice could do automatic billing, secure vo would store credit cards and charge them — not exactly PCI Compliant.

Unfortunately that server has been killed off, so I don’t have a screenshot to take.

Once we started selling Awards, we quickly realized it we needed a tool to help track that business.  Zack’s first project was to explore .NET 4.0  and he built Vendetta our first license restriction system.

What a joke that was — we stopped enforcing licensing restrictions almost out of the gate and this system became a glorified license key generator.  Zack continued to work on Vendetta, by this time he wanted to learn AngularJS and our sales were growing to the point where we needed a system that could help track commissions on complex deals (split payments, multi-year deals, multiple people getting commission).

We worked together to roll out this system.  It was clunky, but it supported everything Secure VO did + commission tracking on recurring and non-recurring deals.  It also contained a system for support people to log into clients with. (Prior to this we had a google doc that had hashed links).  Pretty crazy to think how ghetto things were.  This was a part time project for nearly 1.5 years.  It was slow, and buggy.

It had one key flaw, it didn’t produce invoices, instead it relied on statements.  Like many of my bad decisions, I based my solution based on accounting that an accountant would appreciate, not a common person.  Thus each month resulted in confusion and frustration for people who had to pay our bills.  Eventually we built in a hacked invoicing workaround but by this point Vendetta’s days were numbered.

As we built up Vendetta we also used it to synchronous information from multiple systems.  Today marks the last of those syncs being turned off (with Basecamp).

We learned a ton building Vendetta.  Namely AngularJS, using Azure Cloud Services, and the limitations of Azure Cloud Sites.

A few months ago we decided on the replacement for Vendetta.  Our whole strategy shifted from building these super custom tools to match our specific workflow, to finding the best of breed software to solve our business needs — then sacrificing the fringe cases that weren’t super important.

Part of making this decision was deciding that the NonprofitCMS business was no longer relevant.  The consulting business model was way different from a SaaS sales model, so that decision too allowed us to simplify things.

Keeping the Open Water theme in mind, we named this project SONAR.  We were able to stand this up in a few weeks of development by following two principles:

1. Find systems that solve core needs
2. Use APIs to make integration happen
3. Refactor but mostly copy previous integration code from Vendetta

Now we solve our most common needs as follows:

1. Commission Tracking
   1. Sales tracked in salesforce
   2. Billing automatically tracked in Xero
   3. Sonar combines both feeds of data to compute monthly commissions
2. Provisioning and Billing Clients
   1. Client Licensing Restrictions tracked in SONAR
   2. Billing tracked in Xero
   3. SONAR then combines data to make sure clients have paid up
   4. The OpenWater Platform queries SONAR to lock out users who haven’t paid
3. Ensuring Only Supported Admins Get Support
   1. Supported Admins Tracked in Xero
   2. All Help Tickets get pushed to SONAR before they are processed and are tagged with SUPPORTED vs NOT SUPPORTED attribute

We frowned upon this strategy earlier because we thought we could do it better and cheaper if we went custom.  In hindsight, we spent hundreds of hours building Vendetta and supporting it.  When needs changed we became a bottleneck to our own progress.  Perhaps when we set out on this course years ago, the APIs of these platforms were not robust or easy enough to work with, but more likely it was just overconfidence in our own skills and ability to code quickly and bug free that held us back.

Here is a look at SONAR.