Author: Kevin O'Donnell

Our Favorite Tech: NixOS, Django, and Docker

~ Kevin O'Donnell ~ Leave a comment

Two of those names you probably know, but one you may not, and you really should.  These are three of the key technologies we use in Chill Code, I thought it might be useful to describe what we like about these technologies, why we use them, why we love them, and maybe a couple of things we don’t like about them.

NixOS

NixOS is a Linux distribution with a unique approach to package and configuration management. Built on top of the Nix package manager, it is completely declarative, makes upgrading systems reliable, and has many other advantages.”

— from NixOS.org

We use NixOS for almost everything, including the deployed environments you get when you deploy an environment with Chill Code.  There’s a lot of really nice aspects of NixOS – and its expression language Nix – and, of course, a couple not so nice aspects but on balance we’re big fans.

I encourage everyone to read about Nix and a closely related project Guix (pronounced “geeks” apparently).  Personally, I am not a huge fan of Guile Scheme.  I prefer Nix, but that’s only a case of different strokes for different folks.  NixOS provides us with a lot of important features that ease not only our own infrastructure but the infrastructure of the infrastructure we deploy as well.  To highlight a few:

  • Declarative system configuration model – This is a big one.  It is very difficult to deal with a description of a system configuration that gets implemented in a procedural/sequential fashion.  Ordering of instructions becomes a huge issue without it, and you get into very serious problems when you try to deal with the complete software stack.  NixOS doesn’t ultimately solve this at every level since it doesn’t cover your own software for instance, unless you’ve been a hero and defined your software and configuration in Nix, but it goes a very long way.
  • Atomic Upgrade + Rollback – All or Nothing or nothing config, I really can’t tell you how much easier this makes life.  My earliest incarnation as a technologist was as a database administrator – I can’t even count the number of times I wanted the equivalent of “ROLLBACK;” in systems administration
  • Reproducibility- If you have the same source, and the same Nix definitions, you get the same output, almost no exceptions (and the Nix community is pretty constantly working on no exceptions).  This is more about software build tools and packaging systems than it is about system administration, although the two are clearly closely related.  How many folks reading this have had this situation: You have a pretty well crafted Maven build, but one day your well oiled and elaborate pom.xml starts producing software that is just plain broken.  After a painful amount of time you realize that so dependency deep down in the dependency tree changed in what seemed an innocuous way that broke the whole thing.

All that said there are a couple of things, which we are not fans of, but you take the good and the bad.  To highlight a couple of those:

  • A very unfamiliar file system layout and a rather intimidating looking set of paths.  It’s not that bad when you get used to it, but it is a reasonably sophisticated thing to get used to.  From the Chill Code perspective we recognize it as “not great,” but we also think – generally speaking – dealing intimately with an operating system is a distraction from delivering on your ideas.
  • Nix is essentially YACL (Yet Another Configuration Language) – and how many of those are there today?  It seems like over the past few years there have been a half dozen or more attempts to build the better mouse trap for configuration, complete with fanatical fan base extolling the virtues of their own dialect.  Any time a new expression language comes out, I shed a solitary tear, read up on it and then move on.  You never know when you’ll need it.  That said Nix’s declarative approach is powerful and succinct if unfamiliar.

As I’ve said though, we are big fans of Nix and NixOS.

Django

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source.

— from www.djangoproject.com

Django just works. That is about the highest compliment in my arsenal.  Not out of the box obviously, but surprisingly close to it.  It’s one of the few frameworks that I’ve come across that saves an enormous amount of effort without forcing elaborate mental contortions trying to sync up with a new conceptual model.

My only complaint about Django is really just grumpiness on my part, and really its more about Python than Django.   I’ve been using it to one degree or another since the time I was experimenting with Xlib programming – back before the dark ages – the dark ages being my fascination with the first couple of iterations of MFC and Hungarian Notation. Over the years, I’ve developed an appreciation for strong typing and compilation combined with a powerful IDE.  Python IDEs (Komodo, PyDEV, PyCharm)  simply don’t compare to the Java (Eclipse JDT, IntelliJ, etc.) or C++ IDEs (Visual Studio, which is all but incomparable and I don’t even like MSFT programming).  In fact, I find Python IDEs,  to be more of a hindrance than a help, so I am back to just using Emacs.  Given all the other goodness that is in Python and Django its far from a painful compromise.

 

Docker

Ah, Docker, or was it Moby?  Let’s forget about the identity crisis for the moment.  We use the most basic piece of the docker ecosystem the container and relatively little else.  We use docker for two very specific reasons

  1. We use it to simplify application configuration to the simplest most universal form available to day.  Simple here means that as much as possible applications are configured using the same mechanisms – at the moment we keep roughly to the 12 Factor way. Universal means that running and configuring the application should be roughly the same as way you would run an app on your desktop.  That may sound odd given that it’s still a bit unusual to run containers on your desktop, but again we are sticking to roughly the 12 Factor way.
  2. We use it to encapsulate application artifacts and their runtime.  This makes it easy for Chill Code to deploy and redeploy an application and its components in a fairly predictable way, without introducing any huge new concepts for the developer to digest.

It’s interesting to note that we could have used NixOS based systemd-nspawn containers which are pretty good, but which suffer from the same problems as NixOS itself, since you essentially configure the container the same way you configure a NixOS.  Essentially, it is esoteric, and at the moment a fair amount of inconsistency about how to configure things from one service to the next.  Docker containers give us essentially three axes of configuration 1) software content 2) environment variables 3) command line.  Add to that whatever particulars your application may have (probably driving a lot of configuration off of a database a la WordPress) and quite a lot of situations are covered.

On the bad side, DockerFiles themselves are yet another configuration.  That would be fine except it weakens the immutability too much in our opinion.  There are directives you can use in DockerFiles which are by nature procedural will produce a significantly different result on from one execution to the next. That’s not a good thing in our opinion.  We ended up solving this by taking advantage of a part of Nix (dockerTools) using it to generate our docker images, but using our own mechanisms for shipping around images and running up the containers themselves.  I’d be remiss if I didn’t credit the Nix community and in particular @lethalman for coming up with a positively elegant solution to integrate docker with NixOS.  Truly very good work folks.

We may expand to use more of the docker/moby ecosystem as we progress but for now we have a pretty straightforward approach, and we always consider very carefully before we introduce new complexity to Chill Code.   The docker/moby ecosystem is in a bit of a confused mess.  At the moment really I wouldn’t say we are particularly opinionated about what should happen in the container world.  We require the most basic features of containers, and docker had the most elegant solution within reach for us.  Right now, it wouldn’t be all that difficult to enhance NixOS containers to work the way we need, or use rkt containers and we may yet do so, but its a trade off between effort spent and value we get back out of that effort.

That’s not all we use…

That is not an exhaustive survey of the technology we use at Chill Code.  Like any modern software project there are dozens of other components we use, but those are the majors.  It’s worth noting that our technology choices are guided by the same principle with which we design our own service, i.e. as much as possible the supporting infrastructure and components should get out of the way.  This is definitely the case with NixOs, Docker, and Django.  Each piece helps us implement our ideas and so far hasn’t become a job unto itself.

A Little Bit of Background

~ Kevin O'Donnell ~ Leave a comment

There’s a lot of philosophy behind the way we built Chill Code.  Not so much in terms of development techniques or Agile methodology but rather in terms of what we are up to, who we are doing it for, and why we think they need it.  I thought I might take a little time to explain how we got to where we are.

We began with an observation.  Two actually.  The first observation was that the number of people out there with a great idea vastly exceeds the number of software architects, and infrastructure engineers (cloud or otherwise) available to help those people out.  If you are a small group of very creative engineers you are faced with a few options, none of which are particularly appealing.  You first option would be to just “wing it” and put together your infrastructure and to the best of your abilities.  If you do that you are making a couple of pretty bad compromises.  For one thing, you are guaranteed to get some serious things wrong (security, and scalability are the obvious ones). For another, you end up taking a significant amount of time learning to do something which is neither directly you “big idea” nor is it something which you are likely to add any value.  That’s the first observation – the one that produced one of our first tag lines “helping developers make good software, not bad infrastructure.”  Your other option would be to hire on a very expensive architect type early on in your project, an expensive proposition which – speaking as someone who spent a fair amount of time in enterprise architecture – I wouldn’t recommend.  Not that enterprise architects are bad folks, but good ones are rare and expensive and likely will continue to be so.  On top of that they aren’t really going to contribute too much to your “big idea” either.  Software architects, and infrastructure engineers have spent their careers learning and practicing precisely all that esoteric stuff that you don’t know, but they almost certainly have not spent their lives thinking about the thing that drives you to create your project.

The second observation was more geared toward how software and infrastructure architecture is practiced.  If you compare how your friendly neighborhood software architect works and develops their skills, to how a developer works you should be struck by how arcane and in some ways backward it is.  If you are a developer and you want to figure out how to do something, yes you probably read books, but chances are one of your first stops is Google where you will find an enormous set of references and examples of similar things to what you were after.  Go to some place like Stack Overflow and you have an incredible repository of practical advice and information.  A software architect on the other hand, must trawl through a mountain of detailed reference material, generally quite theoretical and then synthesize that with her own experience to come up with something which may or may not be right.  She’ll spend a frightening amount of time trying to keep up on the latest developments in a tremendous variety of technologies, with essentially no hope of keeping up.  Someone once asked me how much time I spent just doing reading and research, my answer – about 20 hours a week.  I can’t say whether that is typical or not, but I wouldn’t be surprised if it were.  Now I recommend spending as much time as possible reading and researching, but when you compare what an architect has to do to get a solution to a problem to what a developer needs to do – i.e. google, and failing that a post a question “Hey how do I do X?” it’s pretty backward.  For all intents and purposes there is no real online community equivalent of Stack Overflow, or GitHub for infrastructure and software architecture patterns.  The mechanism for that kind of doesn’t really exist – and although some developments have been made in the realm of “Infrastructure As Code” there’s no social exchange for the exchange of practical and actionable material.  We sometimes refer to this as the “Stack Overflow argument” or the “GitHub for infrastructure” argument, in the sense that what we want is the same sort of vibrant referenceable community resource as those two sites.

These observations led us to the conclusion that what we should be doing is removing the friction in this whole area of implementing an idea.  If you are “Bob with The Big Idea” then you shouldn’t be forced to make the compromises I made above, and if you are “Sue the Expert Architect” you should have an easier life.  So how do we help Bob and Sue – well it’s an old story by now – let’s build a useful online community for Bob and Sue to exchange information, i.e. the patterns, we call them Chills. Sue gets to test out and refine her ideas, and see how many folks are using her patterns and get direct feedback.  Bob, on the other hand gets what amounts to a fitness tested design pattern.  Throw in a about a year’s solid work and presto change-o you have Chill Code.

We have a quite a few other “philosophical” points of view, which I or Jasper will explain in other posts going forward.  These all have their own little sobriquets such as “Reward learning and effort, don’t punish ignorance,” and “Make doing the right thing to do the easiest thing to do.”