Community infrastructure patterns and preventing another Equifax

The Equifax hack is likely the greatest theft of personal data ever with 143 million American consumers impacted, 44 million in the UK and I’m sure many more around the world.

Equifax didn’t get hacked because they lacked engineers. Equifax didn’t get hacked because they were unaware security was important. Caring and having resources is not sufficient.

The relentless exposure of the insecurity of our government, security & corporate infrastructure, across the world, makes plain that an entirely new culture and approach is required to bring this under control.

Our early, target demographic at Chill Code is engineers & designers at startups and web development firms that want to outsource infrastructure designs to communities that have the ability to iterate those designs in such a way that they evolve and respond to shifting needs and threats. The collective wisdom of a community with respect to the immediacy of response, the perspectives that shape change, and the perspectives that validate or invalidate alternatives is capacity far beyond an individual developer.

This wisdom has layers.

  1. Design is transparent & iterative.
  2. Iteration is constant.
  3. Intractable disagreement is resolved through forks – code and community.
  4. Community and time tells us the winner.
  5. Deployment is constant, driven by automation.

This is no more than the established wisdom of the development and DevOps communities.

What should be plain is that large firms require the same capacity – and that capacity needs to be deeply incorporated into their philosophies. Corporations need to stop thinking about what they do as ‘special’ or ‘different’. Where once, organisations like banks, had ground-breaking technologies, today they sorely lag the startups defining the new approaches. The security issues facing everyone require collective responses in the form of community patterns for infrastructure and deployment and the capabilities of the communities patterns trumps by far the value of bespoke design and secrecy. All of this driven by automation that is baked into the solutions for threat detection, notification & patching.

The alternative is you keep inventing your architecture. You keep building partial solutions with limited review. You let egos and secrecy continue to dominate. And where you do achieve something positive and original – hell, why not keep it to yourself?

Of course, as a technology community, we’ve long known the right way. This is the very reason why open source has triumphed. It is the same reason that GitHub has triumphed. Community delivers better technology solutions.

 

Launch buttons for GitHub readme files

Have you ever looked at a GitHub project and thought, ‘I’d like to give this a go, but it will take me an hour or more to get it all up and running on a server because it has so many dependencies’?

I have and so I added a new feature to Chill Code that allows developers to embed a “Launch” button inside the readme file that takes users directly to the fork page for  a project. This means that developers on a GitHub project can build a Chill that models all the dependencies, the startup commands, the configuration parameters and which references the code on GitHub. New users can fork the Chill and have the project running on AWS in under 5 minutes.

Below is a screenshot taken from an example I created that allows users to install our demo projects quickly:

emebed.png

You can see this example here.

Here is the markdown:

<img src="http://tomcat.apache.org/images/tomcat.png" alt="Drawing" width="32" height="32" />    [![Deploy Button](https://chillcode.io/images/launch.png)%5D(https://www.chillcode.io/s/1O)

In this instance, I have provided an icon to represent the project (Tomcat), and a link to the fork page (https://www.chillcode.io/s/1O). Just replace these references with your own and you’re away with your own GitHub Launch button.

Of course, this is a markdown example, but you can use the same image and approach in any piece of HTML, such as the front page of a project.

New Release

We release continuously, but some releases have more visible changes than others and I thought it was worth highlighting some of these in the latest version as I think they have really made Chill Code easier to use and understand.

Firstly, we have introduced messages and alerts on the right hand side. More generally, we tidied the screen to make clear that you do your work on the left, and helpful meta stuff resides on the right. In the image below we have a welcome message and alerts highlighting that I haven’t added my AWS credentials (i.e. I can’t deploy) or any GitHub repos (I can’t add source).

alerts.png

You can use these alerts to set credentials outside of the credentials screen as they expand.

inline.png

The message/alert layout is carried through to the Chill screen. This view of the Chill itself is now much tighter and the tabs have been removed so that your environments are more visible. The reasoning here is that people were making great Chills but never deploying them – so we’ve tried to make the deployment functionality far more visible.

edit.png

Back on the front page, we’ve added a feature that allows you to create a new Chill from a software package search. In the instance below I’m looking for Node:

search.png

And I get a collection of packages back…

 

node.png

I can select one of these and then create a complete Chill from it.

new.png

Theoretically you could then deploy straight from here, though you probably want to add content and startup instructions. The goal with this mechanism is to encourage everyone to search for software that they can deploy earlier in the journey of learning about Chill Code. Indeed, we intend to add this to the front page so that anyone that visits get site is tempted to explore and get drawn in.

A brief word on the next release. The code to allow deployment to Google Compute Engine is in place (via the native API’s rather than lib cloud that we use for AWS). We need to bring these features to the front end – so hopefully that will be ready in the next fortnight.

 

 

Why we built Chill Code (aka, I hate deployment, I hate infrastructure, I just want to code).

I’m a good software engineer, but I’m not one of those engineers that delights in engineering, and is sought out for their engineering brilliance. I just want to build stuff and I want people that play with the stuff I build to be excited by their experience.

The implication is that any activity that is not focussed on creating and building great digital experiences is, for me, a distraction that grates. If there is something I’m building, then my tax return is gonna be late, and the office will be untidy. However, I reserve my maximum contempt for that time spent working out what software infrastructure to run, where to run it and how to deploy it correctly, consistently, securely, cheaply, flexibly – and quite a lot of other adverbs. Because that time isn’t occasional, it is constant, it is unavoidable, it is tedious and I am shit at it.

Designing and deploying infrastructure requires a lot of research, testing, experimentation, process and discipline. Many web agencies seek to mitigate this risk by employing the same pattern for years at a time. For example, they select Django or PHP and they get a contract with Rackspace and there are a bunch of other dependencies and that is the pattern. Node.js comes and goes. AWS and Digital Ocean arrive but the pattern is the pattern and all the great benefits of that new technology are ignored.

And I sympathise because I struggle to keep with up the latest technologies. I am frequently conservative in my choices because I simply don’t want to spend time learning what the recommended web server is today for a given class of app. What the recommended driver levels are or learning the subtle differences in how to deploy that latest version and testing all the combinations of packages.

Many engineers enjoy learning new technology – they consider it an opportunity to expand their understanding of the universe. But most don’t, and I have become one of those. We have jobs that require us to produce stuff and we are rarely judged on the currency of our technology choices.

The problem is getting worse as the pace of new new technology and resulting options increases. And whereas 15 years ago there were operators that did the installation today we have DevOps – and this is frequently just an excuse to get the developers to manage the environments without the tools, budgets and training to make it meaningful. I can barely keep up with the new languages and development methodologies – how does anyone expect me to keep up with DevOps – especially as I hate touching infrastructure and all it awkward knobbly bits?

So we built Chill Code. For me, it’s the product that I have always wanted, as a good but not fanatical software engineer. I want to know that I can go to a single location, search for patterns that broadly match my need and that have been implicitly embraced by the software community (as expressed through the fork count). I want to be able to deploy those patterns, with my source code or content from GitHub, in one click, and know that the deployment will be consistent regardless of which cloud I elect to use. That way I get to concentrate on what I care about – building things in code that give people great digital experiences.

 

 

What does Chill Code do?

Chill Code is a public recipe book, and a robot, for deploying software infrastructure into the cloud.

The recipes (chills) are provided by all of you (our users) and defined within our user interface (the Chill Code website). Many of the best early chills are recipes for deploying websites. For example, the WordPress chill defines:

  • software (Apache HTTPD, PHP)
  • files from GitHub (config + demo content)
  • a slot for a database
  • a startup command and various configuration options (e.g. port 80 is public)

Here is a screenshot of the public page for that WordPress chill:

clone.png

I describe Chill Code as a public recipe book. By default, every chill is public, although private source code remains private. If you have taken the time to connect bits of software together and push them to the cloud, then you have a use case and some reasoning behind the implementation. The chances are that others have a similar use case and can learn or benefit from your design and reasoning – so why start research from scratch each time? Collective wisdom creates better software infrastructure in an age when there are fewer specialist infrastructure folk.

Therefore chills are optimised by us to be discovered by other users via search engines, social media etc…. tuned by the various indicators we have for how popular a given chill is. Every public chill page highlights the author, the number of forks and how to fork it.

The robot is our automation engine that takes the recipe defined by a given user (or a version forked by other users) and executes the steps in the recipe to deliver a piece of software infrastructure in the cloud.  In practice this means that the robot is doing some of the following:

  • Instantiating one or more server instances on AWS (other providers such as Azure, Rackspace, Cloud to come soon).
  • Installing the open-source software defined defined within the recipe onto the relevant server instances.
  • Instantiating other services provided by the cloud provider – today, this means databases, but other services such as load balancers are in the pipeline.
  • Installing software or content from GitHub (other repositories to be supported eventually).
  • Compiling source code if a supported build file (nix) has been provided
  • Configuring the server instances with environmental variables and port visibility
  • Starting the software defined in step 1.

Thereafter, Chill Code monitors your service at a high level (we are not a monitoring tool, but one could design chills with monitoring built in) and provides buttons to start and stop your environment.

The recipes are abstracted away from any definition of environment so that one may run multiple environments (Dev, QA, Prod) with different configurations (1x server vs 3x servers) in different clouds (AWS vs Azure). Eventually we shall expose an API that allows users to embed control over these environments in their workflows (e.g. spin up environments in unit tests, provide failover for production). That API is extant but unhardened.

envs.png

Indeed, it is not hard to see all the directions that we could take Chill Code. We could provide a lot of poorly implemented solutions. But we are disciplined and our focus is on providing the greatest solution for capturing and delivering the collective wisdom of developers and engineers pertaining to software infrastructure and automating that wisdom in a form that delivers immutable,  consistent and consistently great software infrastructure.