Notes on Full Stack Fest 2016

I've just been to Full Stack Fest, a diverse tech conference featuring great speakers such as Joe Armstrong and Juan Benet. As I did regarding other conferences I've been to, I'll share some thoughts about each of the talks and the things I've learned. However, before we jump to that -

TL;DR

Hardware is getting faster and faster and it makes developers less careful when it comes to software. We should up our game and ship less bloated software. Maybe unikernels will help with that.

Caching and throttling will only get you this far; at one point you need to rethink your architecture and scale out horizontally by sharding. When you do, make sure you read the academic literature in order to know why a high uptime number is useless if your yield is awful. It might also help to take a look at Erlang and Elixir. And while you're at it, make sure to secure your Docker containers.

Simple, decoupled, microservice architectures are key to building robust systems. Try using something like Lambda to have fewer things to worry about.

More and more people are discovering machine learning and thanks to the great resources online it has become a really accessible field. If you are curious, try making a Go bot or a recommendation system. However, if you have a real problem to solve, try doing it the simple way first - it might be enough.

GraphQL seems like a really nice way to build an API. And IPFS looks like a secure and efficient way to distribute information online.


You can find the full schedule here. The talks have also been recorded and you can find them here. I'll only list the talks' titles - you can find the names of the speakers in the schedule. It's worth mentioning that I only went to the backend track.

Computing: The first 100 years

This was the first talk of the day and everybody was obviously excited to see Joe. He went through the history of computing a bit, talking about forgotten dangers.

kilburn tube

He also compared computing power over the years. His reference was the Cray 1, the first supercomputer.

cray 1

In comparison, computers today are considerably faster. The Raspberry Pi 3 is 15 times faster, while an iPhone 6 is 256 times faster. Last but not least, he mentioned the human brain as being 2.85 * 10^8 faster.

brain

While hardware keeps getting faster and faster, software performance does not reflect the same trend. He argues that the reason for this is that applications nowadays are bloated, taking away from the hardware's increasing performance.

Finally, Joe went a bit meta and talked about the dangers of tomorrow. Relying so much on technology and entities such as cloud providers leads to risks that weren't a factor in the past. The example he gave was family photos (and History in general): will your family photos still be around in 100 years? Who will pay for them? Same with the digitally recorded facts of history. He introduced the idea of content addressable storage and distributed storage models, hinting at Juan Benet's IPFS.

I suggest you check out his talk at Strange Loop.

Shopify in Multiple Datacenters

Simon talked about Spotify's journey over the years from one shop to 300K+ that they have today.

shopify numbers

They started with a multi tenant system (i.e. all shops sharing the same database) implemented in Rails. As traffic increased, they performed all the usual optimizations, improving their reads with caching, throttling and so on. At one point, they started sharding their database across data centers.

Now, they are working on a hybrid system (between single and multi tenant) using pods - clusters of shops sharing resources. Doing this will bring some of the isolation benefits from single tenant system, as well as allowing them to scale more efficiently.

The talk also went into some technical details about how requests are served across data centers and how they deprecate APIs (keywords: shitlist driven development!), all cool stuff. You can find the slides here.

Architectural patterns of resilient distributed systems

This was an interesting talk that approached resiliency in distributed systems both from an academic perspective and from an industry-oriented one.

strategies to build resilience

It was a really dense talk that you should watch on your own (protip: look out for that 0.5x button on YouTube). If reading is more of your thing, you can find the bibliography for the talk here.

Unikernels and why they're useful (or not)

Interestingly enough, this talk was given by a Docker employee. It contained a lot of intro to unikernels type of information and it was capped off with a cool demo, showing how one can use Docker to target any platform while developing and then using unikernels for deployment on the Internet of Things (i.e. building in an ARM container, running the resulting image using Xen on a Cubieboard).

unikernels

What did AlphaGo do to beat the strongest human Go player?

An interesting intro to computer game playing, going through the differences between Chess and Go, as well as why Go is more difficult to play by the computer - the main reason being the lack of a good evaluation function.

Having the basics covered, the speaker explained how Google's AlphaGo made use of neural networks and random algorithms (Monte Carlo) to win at Go.

In addition to the new knowledge, this talk's takeaway was that even if you lack a machine learning/maths background, you can still try and hack on projects outside of your comfort zone - the speaker's day job is Web development, but in his spare time he implements Go bots.

How secure are Docker containers?

This talk went through a number of Linux exploits and how they can be avoided using Docker. Generally, Docker has good security defaults but one can turn on even more features to make sure their containers are safe.

The speaker hosts a website that exposes access to a container, so he has a big interest in Docker security. The main features that were mentioned are that you can use pids-limit to avoid fork bombs, no-new-privileges to avoid root escalation and you can even choose which syscalls programs within the container can run.

Another cool thing is that you can use Shodan.io to find public Docker daemons around the Internet - make sure you won't find yours!

Finally, you can use dockerbench to check for common misconfigs.

High availability with Elixir and Erlang

This talk was pretty high level and generic, defining availability and how the Erlang VM helps in keeping a high uptime. If you've read about Erlang or Elixir before, you probably already know all that was covered.

How to write a search engine in 15 lines of code

Another high level talk, this one introducing Unison, a next generation programming language. I didn't get too much out of it, the only thing that I got was that the author (the speaker is the inventor of Unison) somehow wants to abstract infrastructure and tooling (e.g. Kubernetes, Docker, Consul) into the language. No idea how.

Whirlwind tour through the HTTP2 spec

This was a refreshing, fast-paced intro to HTTP2 and its advantages. Check the talk on Youtube on 2x.

Microservice Pipeline Architecture

The speaker works at a consulting company that had to build BBC newsbeat, and this talk presents the final architecture that they went for, as well as a set of iterations that one could go through in achieving that.

architecture

The flow that they went for (seen above) is really clean and easy to keep available. Every time an article is published, a set of Lambda functions are called, writing the formatted information in S3. Then, a simple server finds the corresponding information in S3 and passes it on to the Web or mobile client for rendering.

The State Of Serverless

I found this talk inspiring both from a technical perspective, as well as from an entrepreneurial one. The speaker is the founder of Serverless, a company/framework that makes interacting with solutions such as AWS Lambda a viable and easy option.

The tech demo was nice, showing how you can quickly get up and running with an online API without having to deal with renting/deploying servers and cumbersome AWS configurations.

However, what was more interesting to me was the story of his company. Basically, Austen was working on his startup at the time Amazon announced Lambda. The idea seemed really cool to him and he decided to rewrite everything using it. Along the process, he discovered that things were missing.

framework time

Framework time! he said, so he pivoted and started working on it in a "place in Oakland that serves cheap coffee". Turns out that this was very much a right idea, right time thing and Amazon got interested in the project, using it to promote their Lambda service and things really took off.

Make sure to check out the talk to hear more about it, and consider a serverless approach in your next project.

Low-res NLP for your everyday life

Enjoyable intro to processing natural language, using Slackbot's help answers as an example.

The talk goes through the process of going from how do I delete a channel to delete,channel and then using that key to lookup the answer in a database of predefined (question, answer) pairs.

This was a nice demo of how you don't need crazy advanced machine learning/NLP flows to solve a meaningful problem.

Distributed Apps with IPFS

This talk was supposed to be an intro to IPFS, but Juan changed his plan and spent the better part of it explaining how software developers are cyberwizards and they should use their powers for good. Check out the recorded talk, and make sure to try out IPFS.

From REST to GraphQL

The second Shopify talk at the conference, this time about GraphQL. The speaker highlighted some of the pain points of using REST when developing UIs (including some fields, expanding other, adding specialized business logic that does not fit REST 100%) and how GraphQL can help. All the points resonated with me, so this got me interested even more in GraphQL.

Apparently Shopify is looking into using this for a mobile API. However, "looking into" means that they open sourced a Ruby implementation of the parser, as well as a batcher for solving N+1 type problems when fetching data. Honestly, I was impressed by how deep such research efforts go.

You should watch the recording, it might convince you to take a[nother] look at GraphQL.

Building a Recommendation Engine with Machine Learning Techniques

The last talk of the track and the speaker had slides to fill up 3 slots. A dumbed-down introduction to machine learning using RapidMiner, a IDE-like tool for doing ML.

Summing up

Was it the best conference ever? Definitely not. Are there better conferences in Europe? Certainly - Craft comes to mind. However, I really enjoyed Full Stack Fest. It was a great opportunity to learn more about what people in our industry are interested in at the moment, from small consulting shops to big companies. It also made me want to go to a hackathon as soon as possible.