From 1 to 100 developers:
Scaling for developer productivity at Spotify

@dawhiting
HUG UK Strata 11/11/2013

How do I scale?

How do I scale my development?

A brief history of Hadoop development at Spotify

What could possibly go wrong?

Contention for resources

Priority and isolation

Hadoop scheduling

YARN

Repetition repetition repetition

Refactor data, not just code

Work at a higher level

Code Quality & Technical Debt

Stable platform

Abolish custom infrastructure

Testing

HDFS

Data retention

Establish convention

Enforce convention

Data Catalogue

Core datasets

Data library as code library

You can have it easier than us

Act now

Plan to decentralise

Make it simpler to do things the right way

Want to join the band?

 

We're hiring for Stockholm and NYC

www.spotify.com/jobs

@dawhiting