How we use Lisp to be independent and resilient

A cautionary tale

I remember this very specific day… I went to work- at a company I used to work for- a large multi-national software development corporation. A regular day, except that when I fired up the compilation toolchain around 8 AM, I was displeased to see an unexpected error:

License for <proprietary product> expired.

The build chain was so monolithic that it was impossible to work around the product whose license had expired. In effect, no code could be compiled, thus leaving thousands of developers across the world stuck useless in front their screens. The whole company ground to a halt, until, some hours later, the license got renewed and redeployed across all the machines.

Back then, it really frustrated me. Today, thinking like a product manager, I'm left thinking of it as a weak business strategy.

A lot of risks surface when we examine this strategy:

• What if it had taken a week to renew the license?
• What if the license could not be renewed?
• What if we could not find a replacement?

However, were we even able to secure a replacement, it would have required work and even further delay to plug it back into the toolchain.

All of this seems like a big waste of time and money: literally thousands of man hours lost.

Looking back, we got off easy.

Sometimes, core components may not be possible: should the essence of your product rely on a feature that exists nowhere else- you are out of luck.

More generally, the reliability of dependencies matters: should one of them have a non-fixable bug or missing feature, it can and often will halt all product development and maintenance.

Non-dependability is important for the business reasons highlighted above, and more generally, for the perennity of your product. If you want your software product to last, you need to be confident about its dependencies.

The risk of SaaS

In modern business, it's common practice to rely on third-party services that are not self-hosted (frequently referred to as SaaS products).

There are many issues with this approach:

• As we saw earlier, it puts your business on a razor thin edge, subject to the mercy of the SaaS provider.
• For users and customers, external services are things we cannot control nor guarantee. Their behavior is a black-box, of which we know nothing.

As an example, you may want to give your customers a privacy guarantee, but by utilizing a third party service, you cannot do so.

This is why it's important that your whole product with all its dependencies can be self-deployed (which means you must have the right to do so legally).

The right tools for independence

It turns out that there are certain measures one can take to increase their independence.

The first that comes to mind is utilizing free, open source dependencies:

• No more licensing issues, ever.
• When a feature is missing, it's possible to add it yourself.
• When there is a bug, it can be fixed.
• …

But even in a free, open source environment, there is no guarantee of no friction:

• It may take ages to deploy a bug fix or feature.
• Not all dependencies can be vendored, which means you'll have to rely on the versions available on other systems. Which means you cannot benefit from local updates and fixes you make yourself.

This is where programming languages show their differences. Most programming languages won't allow you to override the limitations of their dependent libraries. Common Lisp, however, can: nothing is hard-wired, everything can be reshaped, redesigned.

From http://www.gigamonkeys.com/book/macros-standard-control-constructs.html about the extensibility of the language:

A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern. A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide "expert group" to hash everything out. That process–according to Sun–takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer's favorite compiler supports the new version of Java, they probably still can't use the new feature until they're allowed to break source compatibility with older versions of Java. So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years.

Tools matter^{1 2}.

What do I gain from independence?

Independent, self-deployable products are resilient: should one go down, another one can be brought back up instantly. Self-deployable products give you you new decision parameters: decentralization, privacy, and security, to name a few.

In short, if you want to be in control of your business insist on full source licenses for all of your dependencies. Insist that should your dependencies have bugs or missing features, that you can fix or add them. Stay in control by choosing the right tools and libraries that form the pillars that support your software.

References