Erik Bernhardsson

APRIL 1, 2021

I joined Better in early 2015 because I thought the team was crazy enough to actually change one of the largest industries in the US. For six years, I ran the tech team, hiring 300+ people, probably doing 2,000+ interviews, and according to GitHub I added 646,941 lines of code and removed 339,164. But I also got married, had two kids, bought an apartment and renovated it!

Data Engineering 151

Data Engineering Engineering Blockchain Software Engineering 151

Erik Bernhardsson

DECEMBER 15, 2020

It's a popular attitude among developers to rant about our tools and how broken things are. Maybe I'm an optimistic person, because my viewpoint is the complete opposite! I had my first job as a software engineer in 1999, and in the last two decades I've seen software engineering changing in ways that have made us orders of magnitude more productive.

Software Engineering 304

Software Engineering Engineering Tools Software 304

Erik Bernhardsson

OCTOBER 5, 2020

Development 150

Development 150

Erik Bernhardsson

SEPTEMBER 22, 2020

We live in a year of about 350,000 amateur epidemiologists and I have no desire to join that “club” But I read something about COVID-19 deaths that I thought was interesting and wanted to see if I could replicated it through data. Basically the claim is that Sweden had an exceptionally “good” year in 2019 in terms of influenza deaths causing there to be more deaths “overdue” in 2020.

Exercises 185

Exercises Trends Data Research 185

Erik Bernhardsson

MARCH 9, 2020

Hanlon's razor is a classic aphorism I'm sure you have heard before: Never attribute to malice that which can be adequately explained by stupidity. I've found that neither malice nor stupidity is the most common reason when you don't understand why something is in a certain way. Instead, the root cause is probably just that they didn't have time yet.

Course 288

Course Advertising Windows Metrics 288

Erik Bernhardsson

JANUARY 12, 2020

Let’s consider a toy model where you’re hiring for two things and that those are equally valuable. It’s not very important what those are, so let’s just call them “thing A” and “thing B” for now. For one set of abilities, the scatter plot looks like this: The assumption here is that A and B are drawn from a 2D-Gaussian with a mild positive correlation.

Marketing 287

Marketing How To Recruiting Performance 287

Erik Bernhardsson

DECEMBER 30, 2019

The title of this blog post makes a claim that seems pretty obvious, but there’s some real “statistical” reasons why it’s true. Let’s consider a toy model where you’re hiring for two things and that those are equally valuable. It’s not very important what those are, so let’s just call them “thing A” and “thing B” Candidates have abilities with those things that are drawn randomly and independently from a distribution, and to

Marketing 26

Marketing Recruiting Programming Industry 26

Erik Bernhardsson

DECEMBER 18, 2019

I recently finished the excellent book Kochland. This isn’t my first interest in Koch—I read The Science of Success by Charles Koch himself a couple of years ago. Charles Koch inherited a tiny company in 1967 and turned it into one of the world’s largest ones. That’s impressive! Just a quick disclaimer just to get it out of the way.

Industry 26

Industry Culture Compliance Metrics 26

Erik Bernhardsson

DECEMBER 8, 2019

Just a quick note that my team is always hiring at Better. A lot of new people have been joining the team here in NYC lately—the tech team has actually grown from 35 to 60 in just ~3 months. We’re looking for mostly senior software engineers (5+ years work experience, possibly having managed in the past), although we would love to talk if you have less experience too!

Machine Learning 26

Machine Learning Artificial Inteligence Software Engineering Industry 26

Erik Bernhardsson

SEPTEMBER 25, 2019

No one asked for this, but I’m something like ~12 years into my career and have had my fair share of mistakes and luck so I thought I’d share some. Honestly, I feel like I’ve mostly benefitted from luck. Some of the things I did on a whim turned out to be excellent choices many years later. Some of the things were clear blind spots in hindsight.

Weak Development Team 108

Weak Development Team Machine Learning Artificial Inteligence Industry 108

Erik Bernhardsson

OCTOBER 20, 2012

I’m shamelessly promoting my first major open source project. Luigi is a Python module that helps you build complex pipelines of batch jobs, handle dependency resolution, and create visualizations to help manage multiple workflows. It also comes with Hadoop support built in (because that’s where really where its strength becomes clear). We use Luigi internally at Spotify to run thousands of tasks every day, organized in complex dependency graphs.

Open Source 100

Open Source Analysis Infrastructure Testing 100

Erik Bernhardsson

OCTOBER 28, 2012

Something that pops up pretty frequently is to implement time decay, especially where you have recursive chains of jobs. For instance, say you want to keep track of a popularity score. You calculate today’s output by reading yesterday’s output, discounting it by and then adding some hit count for today. Typically you choose so that for a day or something like that.

Lambda 100

Lambda 100

Erik Bernhardsson

NOVEMBER 17, 2012

Not sure how I managed to miss this, but I’m watching this Tumblr presentation and they talk about their projects named after Arrested Development topics: Gob, Parmesan, Buster, Jetpants , Oscar, George and Motherboy. Still, the best software project name is probably still Apple’s BHA.

Software 100

Software Development 100

Erik Bernhardsson

DECEMBER 4, 2012

This was posted on the Twitter Engineering blog a few days ago: Dimension Independent Similarity Computation (DISCO). I just glanced at the paper, and there’s some cool stuff going on from a theoretical perspective. What I’m curious about is why they didn’t decide to use dimensionality reduction to solve such a big problem. The benefit of this approach is that it scales much better (linear in input data size) and produces much better results.

Scalability 100

Scalability Engineering Data Tools 100

Erik Bernhardsson

JANUARY 12, 2013

The Economist just published an article called The best, the worst and the ugly. By looking at historical performance for mutual funds, they find strong support for momentum and mean reversion. Picking the best or the worst fund over the previous five years gives great returns over the next five years. I think this is just confusion around what risk reward is.

Performance 100

Performance 100

Erik Bernhardsson

JANUARY 21, 2013

From the NYC Machine Learning talk I had last week: Haven’t looked at it yet except briefly. Unfortunately the quality isn’t the best.

Machine Learning 100

Machine Learning Artificial Inteligence 100

Erik Bernhardsson

JANUARY 26, 2013

Slides from the talk. Slightly edited because (a) some of the slides make little sense taken out of context (b) Slideshare seem to have problem converting some of the stuff. Collaborative filtering at Spotify from Erik Bernhardsson.

Machine Learning 100

Machine Learning Artificial Inteligence 100

Erik Bernhardsson

FEBRUARY 5, 2013

This article from today in Mashable describes some of the fun stuff I get to work with: Erik Bernhardsson is technical lead at Spotify, where he helped to build a music recommendation system based on large-scale machine learning algorithms, mainly matrix factorization of big matrices using Hadoop. He moved into this role after heading the Business Intelligence team, where he collected, aggregated and made sense of all the data at Spotify, whether that’s ad-hoc insights, A/B testing, visualizatio

Machine Learning 100

Machine Learning Artificial Inteligence Business Intelligence Testing 100

Erik Bernhardsson

FEBRUARY 26, 2013

I recently came across this paper describing how they do ML at Twitter. TL;DR Their approach is pretty interesting. Everything is a Pig workflow and then they do everything as UDF’s. This approach seems pretty interesting. As long as your data can be expressed as small atomic machine learning functions, I’m sure it works great. But there’s so much more than that.

Machine Learning 100

Machine Learning Artificial Inteligence Programming Engineering 100

Erik Bernhardsson

MARCH 21, 2013

Elias Freider just talked about Luigi at PyData 2013: The presentation above is much better than one I put together a few weeks ago.

100

100

Erik Bernhardsson

APRIL 11, 2013

Annoy is a simple package to find approximate nearest neighbors (ANN) that I just put on Github. I’m not trying to compete with existing packages, but Annoy has a couple of features that makes it pretty useful. Most importantly, it uses very little memory and can put everything in a contiguous blob that you can mmap from disk. This way multiple processes can share the same index.

Data 100

Data 100

Erik Bernhardsson

APRIL 12, 2013

I picked up an issue of Foreign Affairs while flying back to NYC from SFO. It features this long interview with U.S. General Stanley McChrystal and I thought it was pretty interesting how striking some of the similarities are between fighting in a war and developing software. On cycle time and how it’s important to learn and integrate quickly: In 2003, in many cases we’d go after someone, we might locate them and capture or kill them, and it would be weeks until we took the intelligence we learn

Data 100

Data Software Development 100

Erik Bernhardsson

APRIL 25, 2013

I like the editing!

100

100

Erik Bernhardsson

APRIL 26, 2013

The simple way to get featured on big data blog these days seem to be. Build something that does 1 thing super well but nothing else. Benchmark it against Hadoop. Publish stats showing that it’s 100x faster than Hadoop. $$$. Spark claims their 100x faster than Hadoop and there’s a lot of stats showing Redshift is 10x faster than Hadoop. There’s a bunch of papers with similar claims.

Hardware 100

Hardware Big Data Scalability Tools 100

Erik Bernhardsson

MAY 6, 2013

Just promoting Spotify stuff here: check out the Snakebite repo on Github, written by Wouter de Bie. It’s a super fast tool to access HDFS over CLI/Python, by accessing the namenode directly over sockets/protobuf. Spotify’s developer blog features a nice blog post outlining what it’s useful for. I think this kicks ass and there will definitely be some kind of Luigi integration coming up at some point. .

Tools 100

Tools Development 100

Erik Bernhardsson

MAY 20, 2013

I was browsing around on the Internet and the physics geek in me started reading about Fermat’s principle. And suddenly something came back to me that I’ve been trying to suppress for many years – how I never understood why there’s anything fundamental about the principal of least time. The principle of least time states that the light will travel from A to B in such a way that the time is minimized.

Travel 100

Travel Construction Internet Course 100

Erik Bernhardsson

MAY 30, 2013

The Discovery page, the new start page in Spotify, is finally out to a fairly significant percentage of all users. Really happy since we have worked on it for the past six months. Here’s a screen shot: Some cool features. Artist/album/track recommendations based on stuff you’ve listened to before. New releases recommendations. Concert recommendations.

100

100

Erik Bernhardsson

JUNE 1, 2013

I’ve been obsessed with Wikipedia for the past ten years. Occasionally I find some good articles worth sharing and that’s why I created the wikiphilia Twitter handle. Just a long stream of stuff that for one reason or another may be interesting. It’s also a bunch of friends posting links. Anyway, the tragedy is that there’s 800 tweets but only 70 followers, so you should follow it now.

100

100

Erik Bernhardsson

JUNE 15, 2013

We had an unconference at Spotify last Thursday and I added a semi-trolling semi-serious topic about abolishing documentation. Or NoDoc , as I’m going to call this movement. This was meant to be mostly a thought experiment, but I don’t see it as complete madness. To be clear, I’m not talking about comments in the code here. I think those are great, and you should probably do more than you are already doing.

Guidelines 100

Guidelines .Net Architecture Examples 100

Erik Bernhardsson

JUNE 18, 2013

Just open sourced hdfs2cass which is a Hadoop job (written in Java) to do efficient Cassandra bulkloading. The nice thing is that it queries Cassandra for its topology and uses that to partition the data so that each reducer can upload data directly to a Cassandra node. It also builds SSTables locally etc. Not an expert at Cassandra so I’ll stop describing those parts before I embarrass myself.

Open Source 100

Open Source Data Tools 100

Erik Bernhardsson

JUNE 25, 2013

Continuing in the same spirit of shameless self-promotion, here’s some recent Luigi press: Reddit thread. A Guide to Python Frameworks for Hadoop (slides from the NYC Hadoop User Group ). This presentation from the Open Analytics NYC meetup about how Foursquare uses Luigi. . Luigi is in the middle of a pretty massive refactoring of the visualizer. David Whiting at Spotify just ripped out the old visualizer (based on Graphviz) and replaced it with one based on D3.

Analytics 100

Analytics Groups 100

Erik Bernhardsson

JULY 23, 2013

Sometimes you have to maximize some function where and. Usually, is concave and differentiable, so there’s one unique global maximum and you can solve it by applying gradient ascent. The presence of the constraint makes it a little tricky, but we can solve it using the method of Lagrange multipliers. In particular, since the surface has the normal , the following optimization procedure works: Go one step in the direction of the gradient.

100

100

Erik Bernhardsson

JULY 26, 2013

I was in Portland, OR for a few days hanging out at OSCON. Was fun. I also talked a bit about Luigi : Next week I’m presenting at the NYC Predictive Analytics meetup together with Blake Shaw from Foursquare. The topic is ML + Hadoop. Will be fun!

Analytics 100

Analytics 100

Erik Bernhardsson

JULY 27, 2013

I thought this article about the company culture at HubSpot is kind of funny. “HubSpot’s Awesome Presentation Shows how to Create a 21st Century Culture”. Just FYI: You’re not different. You’re a bunch of white hipsters aged 25-30 dressed up in the same theme. That’s not being different. On a more serious note, this represents one of the most challenging aspects of scaling a company culture.

How To 100

How To Culture Company 100

Erik Bernhardsson

AUGUST 2, 2013

I was just at the NYC Predictive Analytics meetup talking about how we build machine learning algorithms using Hadoop to power music recommendations. Great meetup, where we had two speakers, me and Blake Shaw from Foursquare. Blake talked about how they use machine learning at Foursquare, using Hadoop (and Luigi), and he uploaded his slides here ! Here’s the full video for the talk (both mine and Blake’s).

Analytics 100

Analytics Machine Learning Artificial Inteligence Video 100

Erik Bernhardsson

AUGUST 8, 2013

I’ve turned into a lazy bastard and I’m just posting presentations on this blog, but here’s one from Rohan Singh at Spotify talking about the backend infrastructure of the Discover page.

Infrastructure 100

Infrastructure 100

Erik Bernhardsson

AUGUST 10, 2013

I’m at KDD in Chicago for a few days. We have a Spotify booth tomorrow, and I wanted to put together some cool graphics to show. I’ve been thinking about doing a 2D embedding of the top artists forever since I read about t-SNE and other papers so this was a perfect opportunity to spend some time on it. So – I spent a couple of hours taking the lower dimensionality representation of all artists, plugging it into the C++ implementation they provide, then using matplotlib to render something cool.

100

100