Deep learning for… Go

Erik Bernhardsson

This is the last post about deep learning for chess/go/whatever. But this really cool paper by Christopher Clark and Amos Storkey was forwarded to me by Michael Eickenberg. It’s about using convolutional neural networks to play Go. The authors of the paper do a much better job than I would ever have done of modeling move prediction in Go and show that their model beat certain Go engines.

Cyber Strategy and the Dunning-Kruger Effect: Could some policymakers be too unskilled to recognize their ineptitude?


By Bob Gourley. The Dunning–Kruger effect is a cognitive bias manifesting in unskilled individuals suffering from illusory superiority , mistakenly rating their ability much higher than is accurate.

Flafka: Apache Flume Meets Apache Kafka for Event Processing

Cloudera Engineering

The new integration between Flume and Kafka offers sub-second-latency event processing without the need for dedicated infrastructure. In this previous post you learned some Apache Kafka basics and explored a scenario for using Kafka in an online application.

Working with Developers


There was a lot of passion in the room last week when I presented Working with Developers at the Stubbs Precellerator. I guess it should not be a surprise that Founders have lots of challenges working with developers. So I promised that I would provide a follow-up after the session.

The 5 Levels of Analytics Maturity

Basic dashboards used to be enough to thrill end users. But over time, modern capabilities have emerged—and bare-minimum features are no longer cutting it. How have analytics changed? And where do your BI offerings fall? Find out sophisticated ways to future-proof your application. Brought to you by Logi Analytics.

Let The Network Do The Work

Dion Hinchcliffe's Web 2.0 Blog

One of the most striking things I see when watching organizations make the transition from legacy industrial models of working to new network-based models , is that we keep trying to employ the new tools and ideas in the same old ways.

More Trending

CIOs Need To Plan For The End Of A Cloud Relationship

The Accidental Successful CIO

Even cloud relationships have to come to an end sometime – are you ready? Image Credit. Since the cloud is a new and shiny thing in the world of IT departments, we are all excited about it because it now has a lot to do with the importance of information technology.

Cloud 96

Al-Qaeda Innovations Correlated To Unauthorized Disclosures By Snowden: Independent Verification Through Open Source Intelligence


By Bob Gourley. Students of history know what happens when adversaries learn how the good guys collect intelligence on them. They change their tactics, techniques or procedures and it gets much harder to know their intentions.

The Visual Microphone: Passive Recovery of Sound from Video


By Bob Gourley. This post deals with exciting science that is fun to watch. But it is also a topic which will have significant implications for security professionals and perhaps also for law enforcement and other mission domains.

Video 113

How to build up a data team (everything I ever learned about recruiting)

Erik Bernhardsson

During my time at Spotify, I’ve reviewed thousands of resumes and interviewed hundreds of people. Lots of them were rejected but lots of them also got offers. Finally, I’ve also had my share of offers rejected by the candidate. Recruiting is one of those things where the Dunning-Kruger effect is the most pronounced: the more you do it, the more you realize how bad you are at it. Every time I look back a year, I realize 10 things I did wrong.

Monetizing Analytics Features: Why Data Visualizations Will Never Be Enough

Think your customers will pay more for data visualizations in your application? Five years ago they may have. But today, dashboards and visualizations have become table stakes. Discover which features will differentiate your application and maximize the ROI of your embedded analytics. Brought to you by Logi Analytics.

Watch My Friend Become My Hero: Cyber Security Champion and Lawyer Jody Westby Does Right Thing at Right Time


If you have worked at the nexus of cyber security and policy you no doubt know Jody Westby.

Tech Titans To Visit At Geoint: Here is my take on the must-see firms on the expo floor


By Bob Gourley. This is a follow on to our post on planning your time at the Geoint Symposium. With this post I want to make some technology recommendations that may be helpful to you as you plan out your time at the Geoint Symposium 14-17 Apr.

Marc Andreessen on the Future of Technology and Implications for Government Service to Citizens


By Bob Gourley. If you are a professional in technology, government and/or business you have no doubt already been exposed to the very interesting Marc Andreessen.

Apache Kafka for Beginners

Cloudera Engineering

When used in the right way and for the right use case, Kafka has unique attributes that make it a highly attractive option for data integration. Apache Kafka is creating a lot of buzz these days.

5 Early Indicators Your Embedded Analytics Will Fail

Many application teams leave embedded analytics to languish until something—an unhappy customer, plummeting revenue, a spike in customer churn—demands change. But by then, it may be too late. In this White Paper, Logi Analytics has identified 5 tell-tale signs your project is moving from “nice to have” to “needed yesterday.". Brought to you by Logi Analytics.

Everything I learned about technical debt

Erik Bernhardsson

I just made it to Sweden suffering from jet lag induced insomnia, but this blog post will not cover that. Instead, I will talk a little bit about technical debt. The concept of technical debt always resonated with me, partly because I always like the analogy with “real” debt. If you take the analogy really far, there are some curious implications. I always like to think of the “interest rate” of software development. Debt is really just borrowing from the future, with some interest rate.

Going Beyond ‘Bolt-On’ Digital Transformation

Dion Hinchcliffe's Web 2.0 Blog

Much has been made recently of the imperative to fully transition our businesses into the modern digital world. It now hardly needs to be said at this point.

Deep learning for… chess (addendum)

Erik Bernhardsson

My previous blog post about deep learning for chess blew up and made it to Hacker News and a couple of other places. One pretty amazing thing was that the Github repo got 150 stars overnight. There was also lots of comments on the Hacker News post that I thought were really interesting. See this skeptical comment for instance). A couple of things came up in several places. I actually fully agree with a lot of the skepticism my blog post got. Here’s a bit of clarification + other stuff.

CIOs Need To Start To Create The New Face Of IT

The Accidental Successful CIO

CIOs need to prepare for the day that traditional IT comes to the end of the line Image Credit. The era of IT as we’ve come to know it is coming to an end. The IT department functions that have gotten us this far are now coming to an end.

3D 95

Why “Build or Buy?” Is the Wrong Question for Analytics

Every time an application team gets caught up in the “build vs buy” debate, it stalls projects and delays time to revenue. There is a third option. Partnering with an analytics development platform gives you the freedom to customize a solution without the risks and long-term costs of building your own. Learn when and why it makes sense to build, buy, or take a combined approach to embedded BI. Brought to you by Logi Analytics.

Optimizing things: everything is a proxy for a proxy for a proxy

Erik Bernhardsson

Say you build a machine learning model, like a movie recommender system. You need to optimize for something. You have 1-5 stars as ratings so let’s optimize for mean squared error. Great. Then let’s say you build a new model. It has even lower mean squared error. You deploy it. This model turns out to give a lower mean squared error. You roll it out to users and the metrics are tanking. Ok so maybe mean squared error isn’t the right thing to optimize for.

Luigi conquering the world

Erik Bernhardsson

I keep forgetting to buy a costume for Halloween every year, so this year I prepared and got myself a Luigi costume a month in advance. Only to realize I was going to be out of town the whole weekend. If anyone wants a Luigi costume, let me know! (I’m I’m not as big as the guy in the picture). Anyway, that’s not the Luigi this blog post is about. This is about the Python workflow manager that I’ve open sourced. If you’re putting together batch jobs into complex pipelines, you should check it out.

The Filter Bubble is Silly and you Can’t Guess What Happened Next

Erik Bernhardsson

I’m at RecSys 2014 , meeting a lot of people and hanging out at talks. Some of the discussions here was about the filter bubble which prompted me to formalize my own thoughts. I firmly believe that it’s the role of a system to respect the user’s intent. Any sensible system will optimize for user’s long-term happiness by providing info back to the user that s/he finds useful.

Running Theano on EC2

Erik Bernhardsson

Inspired by Sander Dieleman’s internship at Spotify, I’ve been playing around with deep learning using Theano. Theano is this Python package that lets you define symbolic expressions (cool), does automatic differentiation (really cool), and compiles it down into bytecode to run on a CPU/GPU (super cool). It’s built by Yoshua Bengio’s deep learning team up in Montreal.

AWS 108

New Study: 2018 State of Embedded Analytics Report

Why do some embedded analytics projects succeed while others fail? We surveyed 500+ application teams embedding analytics to find out which analytics features actually move the needle. Read the 6th annual State of Embedded Analytics Report to discover new best practices. Brought to you by Logi Analytics.

NGA joins GitHub, offers code to help disaster response


By Michael Johnson. We read this press release with pleasure and have deep respect for the incredibly hard work required to achieve the kinds of success demonstrated here. It is not easy to get things like this done in government. From: the NGA pless release at [link].

Ensure Your Command Center/Ops Center Stays Relevant By Leveraging Wayin: See video of this broadcast media use case


By Bob Gourley. We have previously written about Wayin (I am proud to be one of their advisors). Their capabilities continue to evolve to enable organizations to continually improve how social media is used to both understand and engage.

Media 108

Getting Started with Big Data Architecture

Cloudera Engineering

What does a “Big Data engineer” do, and what does “Big Data architecture” look like? In this post, you’ll get answers to both questions. Apache Hadoop has come a long way in its relatively short lifespan.

Luigi success

Erik Bernhardsson

So Luigi , our open sourced workflow engine in Python, just recently passed 1,000 stars on Github, then shortly after passed mrjob as (I think) the most popular Python package to do Hadoop stuff. This is exciting! A fun anecdote from last week: we accidentally deleted roughly 10TB of data on HDFS, and the output of 1,000s of jobs.

The Complete Predictive Analytics Lifecycle for Application Teams

Speaker: Sriram Parthasarathy, Senior Director of Predictive Analytics, Logi Analytics

Applications with predictive analytics are able to deliver massive value to end users. But what steps should product managers take to add predictive analytics to their applications? In this webinar, we’ll walk through an end-to-end lifecycle of embedding predictive analytics inside an application. Find out how a real-world application decided what predictive questions to ask, sourced the right data, organized resources, built models, deployed predictive analytics in production, and monitored model performance over time.

What is the Future of Work?

Dion Hinchcliffe's Web 2.0 Blog

Much has been made recently about one of the stand out trends of the times we live in: Everything is becoming infused with technology. Software is eating the world it is said. Some have claimed that next it might even eat the jobs, which to some degree is almost certainly the case.

3D 114

Technology Context From the Verizon Data Breach Investigations Report (DBIR)


By Bob Gourley. For a decade now the cyber security community has been treated to important strategic context coordinated by Verizon in their Data Breach Investigations Report (DBIR).

Report 108

Big Data Means 5 Big Problems For CIOs

The Accidental Successful CIO

With more data comes more problems for CIOs… Image Credit. If you pick up just about any IT magazine these days, you’ll see that everyone seems to have decided that we are now living in the age of “big data” No data set is too large to be processed by your IT department.