Batteries Included

Sam Currie

2019-12-14T00:00:00.000Z

Developing features that solve your customer's problems should be the focus of development - at the same time there are so many little things required to get a modern web app to run; auth, proxies, blah blah

When I sat down this past summer to start developing Superplus, I wrote down all the little things that needed to be done before it would be "ready". Spoiler: there's a lot of "piping" and support work that needs to get done before any of the cool stuff in an app is at all usable.

Let's see, non-feature work might include:

  • Routing
  • Containerization
  • Authentication
  • Authorization
  • A database
  • Internationalization
  • A CDN
  • DNS entries for email
  • SSL certificates
  • A reverse proxy for handling requests
  • A production application server to run your code
  • Static hosting for the front end
  • A build server
  • A deployment pipeline
  • A load balancer
  • Logging
  • Security groups
  • ...

Anyways, you get the point, there's a lot that goes into making a modern application run let alone all that goes into actually building the damn thing.

So I made a list and I thought about all of the pain I'd gone through in the last several years dealing with some of this stuff and I came up with a few things that I absolutely had to do right so that I wouldn't be living through the same pain all over again.

Number 1: Authentication

I have migrated authentication twice in my career - it is not fun, not in the least. Once it was required because of a hellishly bad setup; instead of hashing passwords, they were being stored as Base64 encoded text and the login mechanism would perform a Base64 encoding of the string and check if that matched what was in the database, if that failed it would then check the raw string against the DB for a match.

Security aside, not thinking through authentication can become a royal pain in other ways. On one project the senior developer creating the API went with an HMAC strategy without much though. Now, HMAC is fine really - a high entropy hash function was used, the session handling would have largely prevented forgery attacks, and repeater attacks are basically impossible. There were a couple of unforeseen downsides however that hurt the project later in production. If you haven't used HMAC before, here's the basic workflow:

  1. The customer logs in with a username/email and password and the API sends back a secret key for the session (it's invalidated afterwards)
  2. Now, anytime the client application needs to communicate with a protected route on the API, it needs to use that key to create a string with some other details like: ohLe1gooyi**This*is*a*secret*key*e5de3be4Fae6hu + Date.now() + Nonce + Message - the nonce is just a random variable that gets stored in the session and has to be different each time, else the session invalidates (logs you out) and the message has some info about the request being made
  3. The string is then hashed using the secret key and sent to the server, which responds based on the message contents

Now, here's the rub - if you want to just use the API without the client app, you have to replicate the message structure and hashing to be able to authenticate which makes it a lot harder to use your API since now your customers have to write a script just to make a GET request for their data.

Another sneaky thing that went wrong had to do with database concurrency - how the database handles multiple requests for the same data. We were using MongoDB as the central datastore and one thing Mongo just isn't very good at is arrays. Those nonces that get stored with the session? They're in an array so that we can check and see if the same values are used twice in a session and therefore aren't coming from our client. At some point, the client app of ours was doing a lot of work and firing off multiple authenticated requests all at once. Sometimes a little bit of network latency would cause those requests to pile up and Mongo was asked to update the same session array multiple times at once - Mongo's array handling is such that it can't always guarantee the requests are atomic, so the end result was that the session would invalidate and our customer would be logged out - seemingly at random for all they knew.

What I'm getting at is that authentication should be thought of as a feature - it should be robust so that most of users never have to think about it, it should be secure so that your engineers never have to mess with it, and it should be easy to use so that outside developers and API customers don't get frustrated with it.

Number 2: Admin and Analytics

If you're trying to sell software to people, you have to know how your customers are using your stuff. You also have to have the ability to jump in and troubleshoot, fix problems, and verify data. Or at least someone in the company has to and if the salespeople, customer support or the CEO has to come bug an engineer every time a one off customer issue arises or to get data for marketing, you're seriously going to drag down productivity and really annoy everyone involved - the engineer who has to go query the database and the marketing folks who are waiting DAYS just to figure out who logged in this month.

Now, one problem is that building a bunch of internal admin panels doesn't sell software in and of itself. And most of time your team is of the "feature feature feature" mindset - there's such a huge list of stuff you want to build and have to build that an admin panel just doesn't seem all that important when you're writing code. The real value of internal tooling like this however, is avoiding a bunch of time when you aren't writing code because your non-technical staff can't independently do their jobs. So I see the admin panel as a requirement, not a nice to have.

Number 3: Nice to Haves (that you're probably only going to have if you do at the beginning)

There's a few things that I've always wanted in my apps but I didn't build from the get-go and just never went back to do later because they were too hard to implement at a late stage. They include things like two-factor authentication and social authentication that touch your auth API and they include internationalization which touches nearly all of your UI code.

All this for what?

All of this is to say that when I sat down to develop the bones of Superplus, I wanted to do it right. And I want to share that experience with my clients as well - that's why in the next couple of weeks I'm going to fork the project, make it generic and open source the repo. So what do you get when you clone it down?

  • A React client built with the latest create-react-app with hooks and React Router v5
  • A Django 3.0 REST API that comes with a built in admin interface
  • PostGreSQL with awesome migration support built into Django
  • A well thought out authentication system including Knox tokens, social authentication, and opt-in two factor
  • Private routing built into the client with an Auth provider that hooks into the API
  • Internationization built right in from day 1
  • Full containerization using Docker and Docker-Compose
  • WSGI and NGINX assets for serving the application and routing traffic in production
  • The Bulma CSS framework for super quick prototyping

Like the Django folks say, it's "batteries included" app development supercharged with everything you need to start writing features. If you're interested, get in touch!