Community infrastructure patterns and preventing another Equifax

The Equifax hack is likely the greatest theft of personal data ever with 143 million American consumers impacted, 44 million in the UK and I’m sure many more around the world.

Equifax didn’t get hacked because they lacked engineers. Equifax didn’t get hacked because they were unaware security was important. Caring and having resources is not sufficient.

The relentless exposure of the insecurity of our government, security & corporate infrastructure, across the world, makes plain that an entirely new culture and approach is required to bring this under control.

Our early, target demographic at Chill Code is engineers & designers at startups and web development firms that want to outsource infrastructure designs to communities that have the ability to iterate those designs in such a way that they evolve and respond to shifting needs and threats. The collective wisdom of a community with respect to the immediacy of response, the perspectives that shape change, and the perspectives that validate or invalidate alternatives is capacity far beyond an individual developer.

This wisdom has layers.

  1. Design is transparent & iterative.
  2. Iteration is constant.
  3. Intractable disagreement is resolved through forks – code and community.
  4. Community and time tells us the winner.
  5. Deployment is constant, driven by automation.

This is no more than the established wisdom of the development and DevOps communities.

What should be plain is that large firms require the same capacity – and that capacity needs to be deeply incorporated into their philosophies. Corporations need to stop thinking about what they do as ‘special’ or ‘different’. Where once, organisations like banks, had ground-breaking technologies, today they sorely lag the startups defining the new approaches. The security issues facing everyone require collective responses in the form of community patterns for infrastructure and deployment and the capabilities of the communities patterns trumps by far the value of bespoke design and secrecy. All of this driven by automation that is baked into the solutions for threat detection, notification & patching.

The alternative is you keep inventing your architecture. You keep building partial solutions with limited review. You let egos and secrecy continue to dominate. And where you do achieve something positive and original – hell, why not keep it to yourself?

Of course, as a technology community, we’ve long known the right way. This is the very reason why open source has triumphed. It is the same reason that GitHub has triumphed. Community delivers better technology solutions.

 

Launch buttons for GitHub readme files

Have you ever looked at a GitHub project and thought, ‘I’d like to give this a go, but it will take me an hour or more to get it all up and running on a server because it has so many dependencies’?

I have and so I added a new feature to Chill Code that allows developers to embed a “Launch” button inside the readme file that takes users directly to the fork page for  a project. This means that developers on a GitHub project can build a Chill that models all the dependencies, the startup commands, the configuration parameters and which references the code on GitHub. New users can fork the Chill and have the project running on AWS in under 5 minutes.

Below is a screenshot taken from an example I created that allows users to install our demo projects quickly:

emebed.png

You can see this example here.

Here is the markdown:

<img src="http://tomcat.apache.org/images/tomcat.png" alt="Drawing" width="32" height="32" />    [![Deploy Button](https://chillcode.io/images/launch.png)%5D(https://www.chillcode.io/s/1O)

In this instance, I have provided an icon to represent the project (Tomcat), and a link to the fork page (https://www.chillcode.io/s/1O). Just replace these references with your own and you’re away with your own GitHub Launch button.

Of course, this is a markdown example, but you can use the same image and approach in any piece of HTML, such as the front page of a project.

New Release

We release continuously, but some releases have more visible changes than others and I thought it was worth highlighting some of these in the latest version as I think they have really made Chill Code easier to use and understand.

Firstly, we have introduced messages and alerts on the right hand side. More generally, we tidied the screen to make clear that you do your work on the left, and helpful meta stuff resides on the right. In the image below we have a welcome message and alerts highlighting that I haven’t added my AWS credentials (i.e. I can’t deploy) or any GitHub repos (I can’t add source).

alerts.png

You can use these alerts to set credentials outside of the credentials screen as they expand.

inline.png

The message/alert layout is carried through to the Chill screen. This view of the Chill itself is now much tighter and the tabs have been removed so that your environments are more visible. The reasoning here is that people were making great Chills but never deploying them – so we’ve tried to make the deployment functionality far more visible.

edit.png

Back on the front page, we’ve added a feature that allows you to create a new Chill from a software package search. In the instance below I’m looking for Node:

search.png

And I get a collection of packages back…

 

node.png

I can select one of these and then create a complete Chill from it.

new.png

Theoretically you could then deploy straight from here, though you probably want to add content and startup instructions. The goal with this mechanism is to encourage everyone to search for software that they can deploy earlier in the journey of learning about Chill Code. Indeed, we intend to add this to the front page so that anyone that visits get site is tempted to explore and get drawn in.

A brief word on the next release. The code to allow deployment to Google Compute Engine is in place (via the native API’s rather than lib cloud that we use for AWS). We need to bring these features to the front end – so hopefully that will be ready in the next fortnight.

 

 

Why we built Chill Code (aka, I hate deployment, I hate infrastructure, I just want to code).

I’m a good software engineer, but I’m not one of those engineers that delights in engineering, and is sought out for their engineering brilliance. I just want to build stuff and I want people that play with the stuff I build to be excited by their experience.

The implication is that any activity that is not focussed on creating and building great digital experiences is, for me, a distraction that grates. If there is something I’m building, then my tax return is gonna be late, and the office will be untidy. However, I reserve my maximum contempt for that time spent working out what software infrastructure to run, where to run it and how to deploy it correctly, consistently, securely, cheaply, flexibly – and quite a lot of other adverbs. Because that time isn’t occasional, it is constant, it is unavoidable, it is tedious and I am shit at it.

Designing and deploying infrastructure requires a lot of research, testing, experimentation, process and discipline. Many web agencies seek to mitigate this risk by employing the same pattern for years at a time. For example, they select Django or PHP and they get a contract with Rackspace and there are a bunch of other dependencies and that is the pattern. Node.js comes and goes. AWS and Digital Ocean arrive but the pattern is the pattern and all the great benefits of that new technology are ignored.

And I sympathise because I struggle to keep with up the latest technologies. I am frequently conservative in my choices because I simply don’t want to spend time learning what the recommended web server is today for a given class of app. What the recommended driver levels are or learning the subtle differences in how to deploy that latest version and testing all the combinations of packages.

Many engineers enjoy learning new technology – they consider it an opportunity to expand their understanding of the universe. But most don’t, and I have become one of those. We have jobs that require us to produce stuff and we are rarely judged on the currency of our technology choices.

The problem is getting worse as the pace of new new technology and resulting options increases. And whereas 15 years ago there were operators that did the installation today we have DevOps – and this is frequently just an excuse to get the developers to manage the environments without the tools, budgets and training to make it meaningful. I can barely keep up with the new languages and development methodologies – how does anyone expect me to keep up with DevOps – especially as I hate touching infrastructure and all it awkward knobbly bits?

So we built Chill Code. For me, it’s the product that I have always wanted, as a good but not fanatical software engineer. I want to know that I can go to a single location, search for patterns that broadly match my need and that have been implicitly embraced by the software community (as expressed through the fork count). I want to be able to deploy those patterns, with my source code or content from GitHub, in one click, and know that the deployment will be consistent regardless of which cloud I elect to use. That way I get to concentrate on what I care about – building things in code that give people great digital experiences.

 

 

Our Favorite Tech: NixOS, Django, and Docker

Two of those names you probably know, but one you may not, and you really should.  These are three of the key technologies we use in Chill Code, I thought it might be useful to describe what we like about these technologies, why we use them, why we love them, and maybe a couple of things we don’t like about them.

NixOS

NixOS is a Linux distribution with a unique approach to package and configuration management. Built on top of the Nix package manager, it is completely declarative, makes upgrading systems reliable, and has many other advantages.”

— from NixOS.org

We use NixOS for almost everything, including the deployed environments you get when you deploy an environment with Chill Code.  There’s a lot of really nice aspects of NixOS – and its expression language Nix – and, of course, a couple not so nice aspects but on balance we’re big fans.

I encourage everyone to read about Nix and a closely related project Guix (pronounced “geeks” apparently).  Personally, I am not a huge fan of Guile Scheme.  I prefer Nix, but that’s only a case of different strokes for different folks.  NixOS provides us with a lot of important features that ease not only our own infrastructure but the infrastructure of the infrastructure we deploy as well.  To highlight a few:

  • Declarative system configuration model – This is a big one.  It is very difficult to deal with a description of a system configuration that gets implemented in a procedural/sequential fashion.  Ordering of instructions becomes a huge issue without it, and you get into very serious problems when you try to deal with the complete software stack.  NixOS doesn’t ultimately solve this at every level since it doesn’t cover your own software for instance, unless you’ve been a hero and defined your software and configuration in Nix, but it goes a very long way.
  • Atomic Upgrade + Rollback – All or Nothing or nothing config, I really can’t tell you how much easier this makes life.  My earliest incarnation as a technologist was as a database administrator – I can’t even count the number of times I wanted the equivalent of “ROLLBACK;” in systems administration
  • Reproducibility- If you have the same source, and the same Nix definitions, you get the same output, almost no exceptions (and the Nix community is pretty constantly working on no exceptions).  This is more about software build tools and packaging systems than it is about system administration, although the two are clearly closely related.  How many folks reading this have had this situation: You have a pretty well crafted Maven build, but one day your well oiled and elaborate pom.xml starts producing software that is just plain broken.  After a painful amount of time you realize that so dependency deep down in the dependency tree changed in what seemed an innocuous way that broke the whole thing.

All that said there are a couple of things, which we are not fans of, but you take the good and the bad.  To highlight a couple of those:

  • A very unfamiliar file system layout and a rather intimidating looking set of paths.  It’s not that bad when you get used to it, but it is a reasonably sophisticated thing to get used to.  From the Chill Code perspective we recognize it as “not great,” but we also think – generally speaking – dealing intimately with an operating system is a distraction from delivering on your ideas.
  • Nix is essentially YACL (Yet Another Configuration Language) – and how many of those are there today?  It seems like over the past few years there have been a half dozen or more attempts to build the better mouse trap for configuration, complete with fanatical fan base extolling the virtues of their own dialect.  Any time a new expression language comes out, I shed a solitary tear, read up on it and then move on.  You never know when you’ll need it.  That said Nix’s declarative approach is powerful and succinct if unfamiliar.

As I’ve said though, we are big fans of Nix and NixOS.

Django

Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source.

— from www.djangoproject.com

Django just works. That is about the highest compliment in my arsenal.  Not out of the box obviously, but surprisingly close to it.  It’s one of the few frameworks that I’ve come across that saves an enormous amount of effort without forcing elaborate mental contortions trying to sync up with a new conceptual model.

My only complaint about Django is really just grumpiness on my part, and really its more about Python than Django.   I’ve been using it to one degree or another since the time I was experimenting with Xlib programming – back before the dark ages – the dark ages being my fascination with the first couple of iterations of MFC and Hungarian Notation. Over the years, I’ve developed an appreciation for strong typing and compilation combined with a powerful IDE.  Python IDEs (Komodo, PyDEV, PyCharm)  simply don’t compare to the Java (Eclipse JDT, IntelliJ, etc.) or C++ IDEs (Visual Studio, which is all but incomparable and I don’t even like MSFT programming).  In fact, I find Python IDEs,  to be more of a hindrance than a help, so I am back to just using Emacs.  Given all the other goodness that is in Python and Django its far from a painful compromise.

 

Docker

Ah, Docker, or was it Moby?  Let’s forget about the identity crisis for the moment.  We use the most basic piece of the docker ecosystem the container and relatively little else.  We use docker for two very specific reasons

  1. We use it to simplify application configuration to the simplest most universal form available to day.  Simple here means that as much as possible applications are configured using the same mechanisms – at the moment we keep roughly to the 12 Factor way. Universal means that running and configuring the application should be roughly the same as way you would run an app on your desktop.  That may sound odd given that it’s still a bit unusual to run containers on your desktop, but again we are sticking to roughly the 12 Factor way.
  2. We use it to encapsulate application artifacts and their runtime.  This makes it easy for Chill Code to deploy and redeploy an application and its components in a fairly predictable way, without introducing any huge new concepts for the developer to digest.

It’s interesting to note that we could have used NixOS based systemd-nspawn containers which are pretty good, but which suffer from the same problems as NixOS itself, since you essentially configure the container the same way you configure a NixOS.  Essentially, it is esoteric, and at the moment a fair amount of inconsistency about how to configure things from one service to the next.  Docker containers give us essentially three axes of configuration 1) software content 2) environment variables 3) command line.  Add to that whatever particulars your application may have (probably driving a lot of configuration off of a database a la WordPress) and quite a lot of situations are covered.

On the bad side, DockerFiles themselves are yet another configuration.  That would be fine except it weakens the immutability too much in our opinion.  There are directives you can use in DockerFiles which are by nature procedural will produce a significantly different result on from one execution to the next. That’s not a good thing in our opinion.  We ended up solving this by taking advantage of a part of Nix (dockerTools) using it to generate our docker images, but using our own mechanisms for shipping around images and running up the containers themselves.  I’d be remiss if I didn’t credit the Nix community and in particular @lethalman for coming up with a positively elegant solution to integrate docker with NixOS.  Truly very good work folks.

We may expand to use more of the docker/moby ecosystem as we progress but for now we have a pretty straightforward approach, and we always consider very carefully before we introduce new complexity to Chill Code.   The docker/moby ecosystem is in a bit of a confused mess.  At the moment really I wouldn’t say we are particularly opinionated about what should happen in the container world.  We require the most basic features of containers, and docker had the most elegant solution within reach for us.  Right now, it wouldn’t be all that difficult to enhance NixOS containers to work the way we need, or use rkt containers and we may yet do so, but its a trade off between effort spent and value we get back out of that effort.

That’s not all we use…

That is not an exhaustive survey of the technology we use at Chill Code.  Like any modern software project there are dozens of other components we use, but those are the majors.  It’s worth noting that our technology choices are guided by the same principle with which we design our own service, i.e. as much as possible the supporting infrastructure and components should get out of the way.  This is definitely the case with NixOs, Docker, and Django.  Each piece helps us implement our ideas and so far hasn’t become a job unto itself.

What does Chill Code do?

Chill Code is a public recipe book, and a robot, for deploying software infrastructure into the cloud.

The recipes (chills) are provided by all of you (our users) and defined within our user interface (the Chill Code website). Many of the best early chills are recipes for deploying websites. For example, the WordPress chill defines:

  • software (Apache HTTPD, PHP)
  • files from GitHub (config + demo content)
  • a slot for a database
  • a startup command and various configuration options (e.g. port 80 is public)

Here is a screenshot of the public page for that WordPress chill:

clone.png

I describe Chill Code as a public recipe book. By default, every chill is public, although private source code remains private. If you have taken the time to connect bits of software together and push them to the cloud, then you have a use case and some reasoning behind the implementation. The chances are that others have a similar use case and can learn or benefit from your design and reasoning – so why start research from scratch each time? Collective wisdom creates better software infrastructure in an age when there are fewer specialist infrastructure folk.

Therefore chills are optimised by us to be discovered by other users via search engines, social media etc…. tuned by the various indicators we have for how popular a given chill is. Every public chill page highlights the author, the number of forks and how to fork it.

The robot is our automation engine that takes the recipe defined by a given user (or a version forked by other users) and executes the steps in the recipe to deliver a piece of software infrastructure in the cloud.  In practice this means that the robot is doing some of the following:

  • Instantiating one or more server instances on AWS (other providers such as Azure, Rackspace, Cloud to come soon).
  • Installing the open-source software defined defined within the recipe onto the relevant server instances.
  • Instantiating other services provided by the cloud provider – today, this means databases, but other services such as load balancers are in the pipeline.
  • Installing software or content from GitHub (other repositories to be supported eventually).
  • Compiling source code if a supported build file (nix) has been provided
  • Configuring the server instances with environmental variables and port visibility
  • Starting the software defined in step 1.

Thereafter, Chill Code monitors your service at a high level (we are not a monitoring tool, but one could design chills with monitoring built in) and provides buttons to start and stop your environment.

The recipes are abstracted away from any definition of environment so that one may run multiple environments (Dev, QA, Prod) with different configurations (1x server vs 3x servers) in different clouds (AWS vs Azure). Eventually we shall expose an API that allows users to embed control over these environments in their workflows (e.g. spin up environments in unit tests, provide failover for production). That API is extant but unhardened.

envs.png

Indeed, it is not hard to see all the directions that we could take Chill Code. We could provide a lot of poorly implemented solutions. But we are disciplined and our focus is on providing the greatest solution for capturing and delivering the collective wisdom of developers and engineers pertaining to software infrastructure and automating that wisdom in a form that delivers immutable,  consistent and consistently great software infrastructure.

 

 

A Little Bit of Background

There’s a lot of philosophy behind the way we built Chill Code.  Not so much in terms of development techniques or Agile methodology but rather in terms of what we are up to, who we are doing it for, and why we think they need it.  I thought I might take a little time to explain how we got to where we are.

We began with an observation.  Two actually.  The first observation was that the number of people out there with a great idea vastly exceeds the number of software architects, and infrastructure engineers (cloud or otherwise) available to help those people out.  If you are a small group of very creative engineers you are faced with a few options, none of which are particularly appealing.  You first option would be to just “wing it” and put together your infrastructure and to the best of your abilities.  If you do that you are making a couple of pretty bad compromises.  For one thing, you are guaranteed to get some serious things wrong (security, and scalability are the obvious ones). For another, you end up taking a significant amount of time learning to do something which is neither directly you “big idea” nor is it something which you are likely to add any value.  That’s the first observation – the one that produced one of our first tag lines “helping developers make good software, not bad infrastructure.”  Your other option would be to hire on a very expensive architect type early on in your project, an expensive proposition which – speaking as someone who spent a fair amount of time in enterprise architecture – I wouldn’t recommend.  Not that enterprise architects are bad folks, but good ones are rare and expensive and likely will continue to be so.  On top of that they aren’t really going to contribute too much to your “big idea” either.  Software architects, and infrastructure engineers have spent their careers learning and practicing precisely all that esoteric stuff that you don’t know, but they almost certainly have not spent their lives thinking about the thing that drives you to create your project.

The second observation was more geared toward how software and infrastructure architecture is practiced.  If you compare how your friendly neighborhood software architect works and develops their skills, to how a developer works you should be struck by how arcane and in some ways backward it is.  If you are a developer and you want to figure out how to do something, yes you probably read books, but chances are one of your first stops is Google where you will find an enormous set of references and examples of similar things to what you were after.  Go to some place like Stack Overflow and you have an incredible repository of practical advice and information.  A software architect on the other hand, must trawl through a mountain of detailed reference material, generally quite theoretical and then synthesize that with her own experience to come up with something which may or may not be right.  She’ll spend a frightening amount of time trying to keep up on the latest developments in a tremendous variety of technologies, with essentially no hope of keeping up.  Someone once asked me how much time I spent just doing reading and research, my answer – about 20 hours a week.  I can’t say whether that is typical or not, but I wouldn’t be surprised if it were.  Now I recommend spending as much time as possible reading and researching, but when you compare what an architect has to do to get a solution to a problem to what a developer needs to do – i.e. google, and failing that a post a question “Hey how do I do X?” it’s pretty backward.  For all intents and purposes there is no real online community equivalent of Stack Overflow, or GitHub for infrastructure and software architecture patterns.  The mechanism for that kind of doesn’t really exist – and although some developments have been made in the realm of “Infrastructure As Code” there’s no social exchange for the exchange of practical and actionable material.  We sometimes refer to this as the “Stack Overflow argument” or the “GitHub for infrastructure” argument, in the sense that what we want is the same sort of vibrant referenceable community resource as those two sites.

These observations led us to the conclusion that what we should be doing is removing the friction in this whole area of implementing an idea.  If you are “Bob with The Big Idea” then you shouldn’t be forced to make the compromises I made above, and if you are “Sue the Expert Architect” you should have an easier life.  So how do we help Bob and Sue – well it’s an old story by now – let’s build a useful online community for Bob and Sue to exchange information, i.e. the patterns, we call them Chills. Sue gets to test out and refine her ideas, and see how many folks are using her patterns and get direct feedback.  Bob, on the other hand gets what amounts to a fitness tested design pattern.  Throw in a about a year’s solid work and presto change-o you have Chill Code.

We have a quite a few other “philosophical” points of view, which I or Jasper will explain in other posts going forward.  These all have their own little sobriquets such as “Reward learning and effort, don’t punish ignorance,” and “Make doing the right thing to do the easiest thing to do.”