Notes on Craft Conf 2015

This past weekend I've been to the second edition of the awesome, up-and-coming Craft Conf. It is a diverse technology conference that does not have focus on any specific technology - rather, as the name implies, it is all about the craft of programming - from bleeding edge data structures and testing to team organization and organizational process. I discovered Craft last year thanks to a Google student scholarship and I was left with such a good impression that I decided to come back this year - especially since they had a great deal for student tickets.

What follows are a few notes on the presentations that I've seen. Then, I'll try to synthesize some thoughts on the stuff that I observed and learned. Warning, what follows is mostly a brain dump.

Keynote: Complexity is Outside the Code

by Jessica Kerr and Dan North

This was the main presentation of the conference. It was a funny duo between the two and the slides were pretty funny. There was a bit of acting involved. The main point was showing that with all the developments in technology and programming, complexity is moved to communication.

Basically, there was a rundown of architectures through time, from coupled monoliths, to SOA and then microservices. In the microservices case, the speakers showed that the code and its data store (if any) are nicely isolated but that the trouble comes up in making sure that services correctly discover and communicate one with another, and this is the new complexity that we have to deal with.

Another important point started from the fact that we are used to having a team for development, a team for operations, a team for metrics and so on. In the microservices / pizza-sized teams case, the speakers pointed out that it makes more sense for the team responsible for the microservice to do all of development, testing, ops, measuring etc. That way, the main communication loop is closed within the team and the overhead of cross-team communication (ex dev with ops) disappears.

The final idea was that coding should not be the main item for a programming team. Building on the lean startup loop, the speakers emphasized that the team should first define what they want to learn. Then, based on that they should define how to measure their progress. From there, a way to test should be put together. And finally, knowing what they want to learn, how to measure and test it, the team should get to coding.
Overall, the talk was a bit idealistic and full of unicorns and funny little GIFs, but I tried to filter out those parts in my notes in order to stay close to the main points which, I find, are extremely relevant.

Writing testable code - A functional programming point of view

by Alvaro Videla

I went to this talk hoping that I'll maybe see some cool tech like Quick Check in Haskell applied to more common programming languages and whatnot. However, from the start of the talk the speaker mentioned that the functional programming part was pretty much "click bait".

Most of the things presented were common sense - don't have long functions, try to have as many pure functions as possible (this was the only FP-related part of the talk), use parameters as much as possible in order to easily mock things in tests etc.

The most important point of the talk in my opinion was arguing that one shouldn't be fanatical about TDD or testing practices in general. Paraphrasing Kent Beck, the speaker mentioned that companies sell working code, not tests. The same about demos - functionality is demoed, not tests. This was in the context of people who missed deadlines because they invested too much time in testing and they neglected the actual functionality.

The Perfect Storm Intensifies - The Convergence of BigData, Cloud and the IoT is Now at Full Strength

by Mac Devine

I wasn't sure what to expect from this buzzword-plenty talk, but I was curious on what someone from IBM has to say nowadays so I went along.

There were two parts - first, the speaker made the point that data is becoming more valuable than functionality; he actually thought as data as of a currency. Also, he argued that in today's world there are no strong barriers for new competition, since any small team can rent all the infrastructure they need from cloud providers. Then, in the 2nd part, the talk focused on interconnected sensors, data acquisition and processing - all from a very high level. I found out about how The Weather Company pivoted towards being a data/API company, and how they place sensors on planes for all sorts of measurements.

Overall, very high level talk, a few nice insights but nothing more. The Internet of Things is supposed to be a big big thing and it is starting to take shape. No news there.

Architecture Without an End State

by Michael Nygard

This presentation covered some issues with enterprise architecture in general, as well as a few best practices. My first impression was twofold: 1. enterprise programming isn't necessarily this Java EE, boring monster. They deal with real-life, hard problems - no cat pictures there! 2. Few things seem as complicated as enterprise architecture (the bad ones) - I learned about SOR and how things break when the business expands and it seemed a pain.

The speaker argued that generally architectures are planned for 3 years, with the 3rd year being the one in which everything is refactored and cleaned up. However, for various business reasons, the 3rd year is also usually when something changes and a new architecture comes up, picking up where the last one left off - and this forms a cycle of increasing clusterfuck. He also mentioned that many problems arise from the fact that people try to find an all-encompassing model for their business needs and such a model never exists (more than a while).

A few pointers for winning at enterprise architecture:

  • be aware of the required tradeoffs and accept them
  • distribute responsibility in isolated components that support local optimization
  • define failure domains (related to the distributed aspect above)
  • modularity
  • decentralized responsibility - he gave an example of how when designing the Boeing 777, engineers were allowed to add 300$ to the budget for every pound of weight that they saved; no approval required
  • simple interfaces, leading to easy replacements
Concurrency: It's Harder (and Easier) Than You Think

by Paul Butcher

I was really curious about this talk, since I've had my eyes on the 7 concurrency models in 7 weeks book for a while.

The talk started off with a few examples of code going horribly wrong in multithreaded situations. There was a short section on instruction level parallelism and the JVM memory model.

The speaker argued that locks and threads concurrency is hard, not concurrency in general. To prove this, he introduced the actor model and showed some demos using Akka. Disclaimer: the speaker did research on actors and he mentioned that he's been waiting for 25 years for actors to become popular again.

There was a question on what his thoughts are on using the compiler and type systems to handle concurrency issues such as data races, like in Rust and the reply was that it is interesting but at the moment it is just research.

The Final Causal Frontier

by Sean Cribbs

This talk was about CRDTs, a bleeding edge kind of concurrent data structures that achieve (eventual) consistency without locking. This talk had lots of PhD in it and I definitely need to do more reading, but the concepts were interesting and I found out that Riak is using such data structures internally. Neat!

Distributed: Virtuality and Reality

by Viktor Klang

This talk was all over the place. I was really confused almost until the end about what exactly the speaker wanted to communicate. Eventually, everything made sense - he started with some examples from the real world in order to get to the principles of the Reactive Manifesto, basically.

distributed

Closing Keynote: How To Save Innovation From Itself

by Alf Rehn

This was a great (de)motivational talk. The idea was that innovation isn't really happening that much nowadays, and that there are many (most, the speaker argued) technology startups that do not solve any real problem.

To emphasize the lack of value in the tech startup world, the speaker gave two examples: edytt and the smart jar. I won't comment on these, other than the fact that I can see what he meant :).

Another example was the fact that California, the center of innovation (supposedly) is going through a massive drought and that there are almost no tech startups trying to solve this (real) problem.

The speaker also argued that he feels that innovation (the word) has lost its meaning, and that it is now a superficial concept, one that aspiring entrepreneurs use to invent problems and established corporations use to tick off a yearly objective.

This talk reminded me of Steve Yegge's talk a few years ago.

Keynote: The New New Software Development Game: Containers, Micro Services and Contract Tests

by Mary Poppendieck

The speaker had a great perspective on technology, especially since she started her career working on mainframes (the kind that fill a big room). She argued that while hardware scales through miniaturization and multiplication, software is different. According to her, the main two ways software has scaled are the Internet and the Cloud. The Internet was a great means of distribution, and the cloud services were great for enabling people to develop software using more resources than they would normally have access to.

The main point of software scaling was the idea of decoupling, isolation and federation. She gave Project Ara, microservices and containers as examples for this.

Suggested reading: The Box - a book on containers (the cargo ship kind).

She mentioned that the main problem with microservices is individual deployment and making sure that no deploy breaks anybody. To help with this, contract testing (and mocks) were mentioned.

In the context of microservices, the speaker suggested that ops and developers should be on the same team (similar to the ideas in the first keynote). However, she pointed out that dev and ops are different, in that ops are focused on preventing failure and devs are focused on creating gains through addition.

dev-ops

Related to federation and decoupling, she described the notion of command intent. Adaptation is key, since failure is going to happen.

Using Logs To Build a Solid Data Infrastructure

by Martin Kleppmann

This talk was similar to the speaker's Strangeloop talk this year. The main idea was that there are multiple datastores needed for different operations (e.g. sql, elasticsearch and redis) and that the application logic handles all the necessary writes, adding complexity to the code, as well as creating longer response times.

The speaker suggested an approach where the application writes data to a queue (Kafka was suggested), and then all datastores are subscribed to that queue, and data ends up "where it needs to" automatically. The approach is really similar to oplogs. I suggest you watch the Strangeloop talk to get all the details.

Interaction Driven Design

by Sandro Mancuso

This talk was mostly about CQRS and how to structure MVC code. Focus on behavior and not on data. Use libraries not frameworks. That kind of stuff.

The Front End Architecture Revolution

by David Nolen

I was really excited about David Nolen's talk, mainly because I follow all his Clojure[script] stuff. This one was pretty high level, he introduced a few concepts and technologies:

  • immutability
  • React
  • Relay
  • Datomic

He also talked about working with systems that support global optimization (with reference to the Google Closure Compiler that ClojureScript uses). Also, regarding Relay and Datomic, he advocated the idea of a simple backend that does not focus on aggregating and formatting data, leaving that part to the query protocol (I suggest you watch the Relay talk).

I also found out that Netflix is working to something similar to Relay, called JSON Graph.

Changing Software Physics to Make Problems Disappear

by Sadache Aldrobi

I don't have many things to say about this talk. The title won my attention but the content was not on the same level. Basically, the speaker explained how he designed his new product (I felt there was a lot of advertising for his startup in this talk). The main points:

  • CQRS
  • modularity
  • separate concenrs - this way, you might end up having to solve smaller problems with less constraints
Simple and Extensible

by Jeff Lindsay

Jeff wrote dokku, worked on Docker and did a lot of other great things, so I was curious to hear what he had to say.

He believes that all code is a liability, so codebases should be as small as possible. The UNIX philosophy (do one thing well and easily interact with others) was brought up as well.

Another suggestion was adding hooks to designs in order to support future integrations. Hooks are preferred to plugins since they allow more decoupling.

Contracts and Clojure: The Best-Yet Compromise Between Types and Tests

by Jessica Kerr

This talk was mainly about the speaker's frustrations with the Scala type system, as well as the lack of a type system in Clojure. Then, as a middle ground, she introduced Prismatic's schema and gave a thorough example on how it can be used to achieve "the best of both worlds" (the freedom to explore and iterate that Clojure provides and some of the readability and safety of Scala).

The idea was to use Schema to provide annotations to functions, and then use generative testing (similar to quick check) in order to ensure that the annotations match the functions' inputs and outputs.

Closing Keynote: Great Engineering, Failed Product

by Marty Cagan

The closing keynote was really interesting. The speaker has a deep background in the tech scene and he seems the "seen it all" kind of person. The premise of the talk was that engineering and the craft of programming have considerably improved through the years, but that the success rate of startups has pretty much stagnated. The speaker argued that the problem relies within the process that startups trying to reach product-market fit are using and he has outlined a few examples of what not to do.

Here are the main points:

  • ideas tend to come from clients or highly ranked employees, but these are not the best sources; The best sources are engineers (because they know what is possible) and the customer data (seeing what customers are doing / trying to do)
  • business cases are not of much value because in this phase it is hard to predict both the cost and the time necessary to build the product
  • roadmaps are a pain because estimations are given without full thought and when the time comes teams realize they need more time - stakeholders view it as a promise but it is actually just a rough estimation
  • while the development happens in sprints, the rest of the processes are actually resembling the waterfall model

waterfall

He suggests that the cycle should look more like this:

agile

The speaker mentioned that good teams iterate extremely quickly (along the lines of 10 iterations per week). Also, a great nuance was the suggestion to focus on outcome, not on output. This means: increase order completion rate to 50% versus integrate PayPal into the billing module.

He seemed very informed and I will definitely check out some items on the suggested reading list: John Doerr - missionaries vs mercenaries; "the power of customer misbehavior" (article) and Inspired - a book written by the speaker.

Bonus: Lean Poker workshop

by Rafael DeVill

The day after the conference ended I went with some friends to a Lean Poker workshop, organized by the creator of the concept. It was pretty cool - you get to experience quick iteration and continuous delivery firsthand.

The idea is to write a bot that plays poker and play against other teams in the tournament. The organizers already had everything in place - the server, the API, docs, a way to spin up Heroku dynos for the bots - everything. And they even support most programming languages out of the box.

I felt that the event was a bit too long for my taste but overall it was a fun experience and I'd definitely recommend trying it out. It might be a good opportunity to try out a new language (that is, if you aren't too competitive in which case you'll go with what you know best).

Summing up

Microservices are the new black. I wonder if (more like when) a new thing will come up and microservices will be viewed like SOA is viewed now.

Containers are becoming really popular - especially Docker, especially in microservices setups.

Concurrency is becoming more and more necessary. Programming models from decades ago are becoming popular again. Check out Bret Victor's awesome talk for more on that.

We need to focus more on bringing real value to people, rather than inventing the next GIF indexing app.

Immutability and pure functions are becoming widely accepted best practices, in big part thanks to frameworks such as React.

Great conferences with great speakers are becoming a thing in Central/Eastern Europe!


That is all, sorry for the MVP-ish structure of the text. If you are curious to get more details about any of the above, feel free to ping me on Twitter.