Interactive Data Visualization for the Web

Preface
Introduction
- Why Interactive?
Introducing D3
Technology Fundamentals
- A Note on Compatibility
Setup
- Setting Up a Web Server
Data
- Binding Data
Drawing with Data
- Multivalue Maps
Scales
Axes
Updates, Transitions, and Motion
Interactivity
- Introducing Behaviors
Layouts
Geomapping
More reference

nm#+SETUPFILE: ~/.emacs.d/src/org-templates/level-2.org

Preface

On design process:

Designing Data Visualizations: Intentional Communication from Data to Display by Noah Iliinsky and Julie Steele. O’Reilly Media, 2011.
Data Visualization: A Successful Design Process by Andy Kirk. Packt Publishing, 2012.
The Functional Art: An Introduction to Information Graphics and Visualization by Alberto Cairo. New Riders, 2012
Information Dashboard Design: The Effective Visual Communication of Data by Stephen Few. O’Reilly Media, 2006.

On the practicalities of working with data:

Bad Data Handbook: Mapping the World of Data Problems by Q. Ethan McCallum. O’Reilly Media, 2012
Data Analysis with Open Source Tools: A Hands-On Guide for Programmers and Data Scientists by Philipp K. Janert. O’Reilly Media, 2010.
Python for Data Analysis: Agile Tools for Real World Data by Wes McKinney.O’Reilly Media, 2012.

Introduction

Why Interactive?

The basic functions of most interactive visualization tools have changed little since 1996, when Ben Shneiderman of the University of Maryland first proposed a “Visual Information-Seeking Mantra”: overview first, zoom and filter, then details-ondemand.

Introducing D3

D3’s official home on the Web is d3js.org.

What It Doesn’t Do

D3 is intended primarily for explanatoryvisualization work, as opposed to exploratory visualizations.
D3 doesn’t even try to support older browsers.
D3’s core functionality doesn’t handle bitmap map tiles, such as those provided by Google Maps or Cloudmade. D3 is great with anything vector—SVG images or GeoJSON data—but wasn’t originally intended to work with traditional map tiles.
D3 doesn’t hide your original data.

Origins and Context

Protovis made generating visualizations simple, even for users without prior programming experience. Yet to achieve this, it created an abstract representation layer. The designer could address this layer using Protovis syntax, but it wasn’t accessible through standard methods, so debugging was difficult.

In 2011, Mike Bostock, Vadim Ogievetsky, and Jeff Heer officially announced D3, the next evolution in web visualization tools. Unlike Protovis, D3 operates directly on the web document itself. This means easier debugging, easier experimentation, and more visual possibilities. The only downside to this approach is a potentially steeper learning curve.

Alternatives

Easy Charts
- DataWrapper: A beautiful web service that lets you upload your data and quickly generate a chart that you can republish elsewhere or embed on your site. This service was originally intended for journalists, but it is helpful for everyone.
- Flot; A plotting library for jQuery that uses the HTML canvas element and supports older browsers, even all the way back to Internet Explorer 6. It supports limited visual forms (lines, points, bars, areas), but it is easy to use.
- Google Chart Tools: Having evolved from their earlier Image Charts API, Google’s Chart Tools can be used to generate several standard chart types, with support for old versions of IE.
- gRaphaël: It has more visual flexibility than Flot, and—some might say—it is prettier.
- Highcharts JS: A JavaScript-based charting library with several predesigned themes and chart types. It uses SVG for modern browsers and falls back on VMLfor old versions of IE, including IE6 and later.
- JavaScript InfoVis Toolkit: The JIT provides several preset visualization styles for your data. It includes lots of examples, but the documentation is pretty technical. The toolkit is great if you like one of the preset styles, but browser support is unclear.
- jqPlot: A plug-in for charting with jQuery. This supports very simple charts and is great if you are okay with the predefined styles.
- jQuery Sparklines: A jQuery plug-in for generating sparklines, typically small bar, line, or area charts used inline with text.
- Peity: A jQuery plug-in for very simple and very tinybar, line, and pie charts that supports only recent browsers.
- Timeline.js: A library specifically for generating interactive timelines. No coding is required; just use the code generator. There is not much room for customization, but hey, timelines are really hard to do well.
- YUI Charts: The Charts module for the Y ahoo! User Interface Library enables creation of simple charts with a goal of wide browser support.
Graph Visualizations
- Arbor.js: A library for graph visualization using jQuery. Even if you never use this, you should check out how the documentation is presented as a graph, using the tool itself.
- Sigma.js: A very lightweight library for graph visualization. Sigma.js is beautiful and fast, and it also uses canvas.
Geomapping
- Kartograph: A JavaScript-and-Python combo for gorgeous, entirely vector-based mapping by Gregor Aisch with must-see demos. Kartograph works with IE7 and newer.
- Leaflet: A library for tiled maps, designed for smooth interaction on both desktop and mobile devices.
- Modest Maps: The granddaddy of tiled map libraries, Modest Maps has been succeeded by Polymaps, but lots of people still love it, as it is lightweight and works with old versions of IE and other browsers. Modest Maps has been adapted for ActionScript, Processing, Python, PHP, Cinder, openFrameworks…yeah, basically everything.
- Polymaps: A library for displaying tiled maps, with layers of data on top of the tiles.
Almost from Scratch
- Processing.js: A native JavaScript implementation of Processing, the fantastic programming language for artists and designers new to programming. Processing is written in Java, so exporting Processing sketches to the Web traditionally involved clunky Java applets. Thanks to Processing.js, regular Processing code can run natively, in the browser.
- Paper.js: A framework for rendering vector graphics to canvas.
- Raphaël: Another library for drawing vector graphics, popular due to its friendly syntax and support for older browsers.
Three-Dimensional
- PhiloGL: A WebGL framework specifically for 3D visualization.
- Three.js: A library for generating any sort of 3D scene you could imagine, produced by Google’s Data Arts team.
Tools Built with D3
- Crossfilter: A library for working with large, multivariate datasets, written primarily by Mike Bostock. This is useful for trying to squeeze your “big data” into a relatively small web browser.
- Cubism: A D3 plug-in for visualizing time series data.
- Dashku: An online tool for data dashboards and widgets updated in real time
- dc.js: The “dc” is short for dimensional charting, as this library is optimized for exploring large, multidimensional datasets.
- NVD3: Reusable charts with D3. NVD3 offers lots of beautiful examples, w visual customizations without requiring as much code as D3 alone.
- Polychart.js: More reusable charts, with a range of chart types available. Polychart.js is free only for noncommercial use.
- Rickshaw: A toolkit for displaying time series data that is also very customizable.
- Tributary: A great tool for experimenting with live coding using D3.

Technology Fundamentals

A Note on Compatibility

Older browsers don’t support SVG. So, generally speaking, Internet Explorer version 8 and older will not display SVG images at all.

That said, it’s polite to notify users of older browsers why the piece isn’t working. I recommend using Modernizr or a similar JavaScript tool to detect whether or not the browser supports SVG. If it does, then you can load your D3 code and proceed as normal. If SVG is notsupported, then you can display a static, noninteractive version of your visualization alongside a message explaining that a current browser is needed. (Be nice and provide links to the Chrome and Firefox download pages.)

I’d typically have something like this in the <head>of my document:

<script src="js/modernizr.js"></script>
<script type="text/javascript">
    Modernizr.load({
        test: Modernizr.svg && Modernizr.inlinesvg,
        yep : [ 'js/d3.v3.min.js',
                'js/script.js' ]
    });
</script>

caniuse.comis a fantastic resource for supported browser features. See their list of browsers with SVG support.

Setup

Setting Up a Web Server

python -m SimpleHTTPServer 8888 &.

use http://localhost:8888/.

Data

Binding Data

Loading CSV data

var dataset;
d3.csv("food.csv", function(error, data) {
    if (error) {  //If error is not null, something went wrong.
        console.log(error);  //Log the error.
    } else {  //If no error, the file loaded correctly. Yay!
        console.log(data);  //Log the data.
        //Include other code to execute after successful file load here
        dataset = data;
        generateVis();
        hideLoadingMsg();
    }
});

One more tip: if you have tab-separated data in a TSV file, try the d3.tsv() method.

Loading JSON data

d3.json("waterfallVelocities.json", function(json) {
    console.log(json);  //Log output to console
});

Drawing with Data

attr() sets DOM attribute values, whereas style() applies CSS styles directly to an element.

Multivalue Maps

  svg.select("circle")
      .attr("cx", 0)
      .attr("cy", 0)
      .attr("fill", "red");

  svg.select("circle")
      .attr({
          cx: 0,
          cy: 0,
          fill: red
      });

svg.selectAll("rect")
    .data(dataset)
    .enter()
    .append("rect")
    .attr({
        x: function(d, i) { return i * (w / dataset.length); },
        y: function(d) { return h - (d * 4); },
        width: w / dataset.length - barPadding,
        height: function(d) { return d * 4; },
        fill: function(d) { return "rgb(0, 0, " + (d * 10) + ")"; }
    });

Scales

var scale = d3.scale.linear()
    .domain([100, 500])
    .range([10, 350]);

d3.min() and d3.max()

d3.max(dataset, function(d) {
    return d[0];
});

Setting Up Dynamic Scales

var xScale = d3.scale.linear()
    .domain([0, d3.max(dataset, function(d) { return d[0]; })])
    .range([0, w]);

Incorporating Scaled Values

.attr("x", function(d) {
return xScale(d[0]);
})
.attr("y", function(d) {
    return yScale(d[1]);
})

Other Methods

nice()
rangeRound()
clamp(): Calling clamp(true)on a scale, however, forces all output values to be within the specified range. This means excessive values will be rounded to the range’s low or high value (whichever is nearest).

Other Scales

sqrt
pow
log
quantize
quantile
ordinal
d3.scale.category10(), d3.scale.category20(), d3.scale.category20b(), and d3.scale.category20c()
d3.time.scale()

Axes

Setting Up an Axis

  var xScale = d3.scale.linear()
      .domain([0, d3.max(dataset, function(d) { return d[0]; })])
      .range([padding, w - padding * 2]);
  //Create SVG element
var svg = d3.select("body")
    .append("svg")
    .attr("width", w)
    .attr("height", h);

svg.append("g")
    .call(xAxis);

Cleaning It Up

  <style type="text/css">
      .axis path,
      .axis line {
          fill: none;
          stroke: black;
          shape-rendering: crispEdges;
      }

      .axis text {
          font-family: sans-serif;
          font-size: 11px;
      }
  </style>
<script type="text/javascript">
//Create scale functions
    var xScale = d3.scale.linear()
    .domain([0, d3.max(dataset, function(d) { return d[0]; })])
    .range([padding, w - padding * 2]);
//Create SVG element
var svg = d3.select("body")
    .append("svg")
    .attr("width", w)
    .attr("height", h);
//Create X axis
svg.append("g")
    .attr("class", "axis")
    .attr("transform", "translate(0," + (h - padding) + ")")
    .call(xAxis);
</script>

review your property namesvery closely to ensure you’re using SVG names, not CSS ones. (You can reference the complete SVG attribute list on the MDN site.)

Check for Ticks

var xAxis = d3.svg.axis()
    .scale(xScale)
    .orient("bottom")
    .ticks(5);  //Set rough # of tick

D3 inteprets the ticks() value as merely a suggestion and will override your suggestion with what it determines to be the most clean and human-readable values

Formatting Tick Labels

var formatAsPercentage = d3.format(".1%");
xAxis.tickFormat(formatAsPercentage);

Updates, Transitions, and Motion

Modernizing the Bar Chart

  var xScale = d3.scale.ordinal()
      .domain(d3.range(dataset.length))
      .rangeRoundBands([0, w], 0.05);
//Create bars
svg.selectAll("rect")
    .data(dataset)
    .enter()
    .append("rect")
    .attr("x", function(d, i) {
        return xScale(i);
    })
    .attr("width", xScale.rangeBand())

Transitions

Making a nice, super smooth, animated transition is as simple as adding one line of code:

.transition()

ease() must also be specified after transition(), but before the attr() statements to which the transition applies.

…  //Selection statement(s)
.transition()
.duration(2000)
.ease("linear")
…  //attr() statements

cubic-in-outis the default.

each() Transition Starts and Ends

//Update all circles
svg.selectAll("circle")
    .data(dataset)
    .transition()
    .duration(1000)
    .each("start", function() {  // <-- Executes at start of transition
        d3.select(this)
            .attr("fill", "magenta")
            .attr("r", 3);
    })
        .attr("cx", function(d) {
            return xScale(d[0]);
        })
    .attr("cy", function(d) {
        return yScale(d[1]);
    })
    .each("end", function() {  // <-- Executes at end of transition
        d3.select(this)
            .attr("fill", "black")
            .attr("r", 2);
    });

Other Kinds of Data Updates

Adding Values

we can use enter() to address the one new corresponding DOM element, without touching all the existing rects.

dataset.push(newNumber);
var bars = svg.selectAll("rect")                        //Select all bars
    .data(dataset); //Re-bind data to existing bars, return the 'update' selection
bars.enter()                                                //References the enter selection (a subset of the update selection)
    .append("rect")         //Creates a new rect
    .attr("x", w)                       //Sets the initial x position of the rect beyond the far right edge of the SVG
    .attr("y", function(d) {               //Sets the y value, based on the updated yScale
        return h - yScale(d);
    })
    .attr("width", xScale.rangeBand())    //Sets the width value, based on the updated xScale
    .attr("height", function(d) {          //Sets the height value, based on the updated yScale
        return yScale(d);
    })
    .attr("fill", function(d) {        //Sets the fill value
        return "rgb(0, 0, " + (d * 10) + ")";
    });

Removing Values

//Remove one value from dataset
dataset.shift();

bars.exit()                             //References the exit selection (a subset of the update selection)
    .transition()           //Initiates a transition on the one element we're deleting
    .duration(500)
    .attr("x", w)           //Move past the right edge of the SVG
    .remove();              //Deletes this element from the DOM once transition is complete

Add and Remove

//See which p was clicked
var paragraphID = d3.select(this).attr("id");
//Decide what to do next
if (paragraphID == "add") {
    //Add a data value
    var maxValue = 25;
    var newNumber = Math.floor(Math.random() * maxValue);
    var lastKeyValue = dataset[dataset.length - 1].key;
    console.log(lastKeyValue);
    dataset.push({
        key: lastKeyValue + 1,
        value: newNumber
    });
} else {
    //Remove a value
    dataset.shift();
}

Interactivity

Making your visualization interactive is a simple, two-step process that includes:

Binding event listeners
Defining the behavior

Introducing Behaviors

Hover to Highlight
- mouseover
```
.on("mouseover", function() {
    d3.select(this)
        .attr("fill", "orange");
});
```
- mouseout
```
.on("mouseout", function(d) {
    d3.select(this)
        .attr("fill", "rgb(0, 0, " + (d * 10) + ")");
});
```
Mouse events are triggered only on elements with pixels that can be “touched” by the mouse. If two elements overlap, and the mouse moves over the element that is “on top” (in other words, closer to the front), then the mouseover event will be triggered on the frontmost element, and noton the element behind it.

Layouts

Pie Layout

var w = 300;
var h = 300;

var dataset = [ 5, 10, 20, 45, 6, 25 ];

var outerRadius = w / 2;
var innerRadius = 0;
var arc = d3.svg.arc()
    .innerRadius(innerRadius)
    .outerRadius(outerRadius);
var pie = d3.layout.pie();

//Easy colors accessible via a 10-step ordinal scale
var color = d3.scale.category10();

//Create SVG element
var svg = d3.select("body")
    .append("svg")
    .attr("width", w)
    .attr("height", h);

//Set up groups
var arcs = svg.selectAll("g.arc")
    .data(pie(dataset))
    .enter()
    .append("g")
    .attr("class", "arc")
    .attr("transform", "translate(" + outerRadius + "," + outerRadius + ")");

//Draw arc paths
arcs.append("path")
    .attr("fill", function(d, i) {
        return color(i);
    })
    .attr("d", arc);

Stack Layout

//Set up stack method
                        var stack = d3.layout.stack();

                        //Data, stacked
                        stack(dataset);

                        //Set up scales
                        var xScale = d3.scale.ordinal()
                                .domain(d3.range(dataset[0].length))
                                .rangeRoundBands([0, w], 0.05);

                        var yScale = d3.scale.linear()
                                .domain([0,                             
                                        d3.max(dataset, function(d) {
                                                return d3.max(d, function(d) {
                                                        return d.y0 + d.y;
                                                });
                                        })
                                ])
                                .range([0, h]);

                        //Easy colors accessible via a 10-step ordinal scale
                        var colors = d3.scale.category10();

                        //Create SVG element
                        var svg = d3.select("body")
                                                .append("svg")
                                                .attr("width", w)
                                                .attr("height", h);

                        // Add a group for each row of data
                        var groups = svg.selectAll("g")
                                .data(dataset)
                                .enter()
                                .append("g")
                                .style("fill", function(d, i) {
                                        return colors(i);
                                });

                        // Add a rect for each data value
                        var rects = groups.selectAll("rect")
                                .data(function(d) { return d; })
                                .enter()
                                .append("rect")
                                .attr("x", function(d, i) {
                                        return xScale(i);
                                })
                                .attr("y", function(d) {
                                        return yScale(d.y0);
                                })
                                .attr("height", function(d) {
                                        return yScale(d.y);
                                })
                                .attr("width", xScale.rangeBand());

Force Layout

//Initialize a default force layout, using the nodes and edges in dataset
                        var force = d3.layout.force()
                                                                 .nodes(dataset.nodes)
                                                                 .links(dataset.edges)
                                                                 .size([w, h])
                                                                 .linkDistance([50])
                                                                 .charge([-100])
                                                                 .start();

                        var colors = d3.scale.category10();

                        //Create SVG element
                        var svg = d3.select("body")
                                                .append("svg")
                                                .attr("width", w)
                                                .attr("height", h);

                        //Create edges as lines
                        var edges = svg.selectAll("line")
                                .data(dataset.edges)
                                .enter()
                                .append("line")
                                .style("stroke", "#ccc")
                                .style("stroke-width", 1);

                        //Create nodes as circles
                        var nodes = svg.selectAll("circle")
                                .data(dataset.nodes)
                                .enter()
                                .append("circle")
                                .attr("r", 10)
                                .style("fill", function(d, i) {
                                        return colors(i);
                                })
                                .call(force.drag);

                        //Every time the simulation "ticks", this will be called
                        force.on("tick", function() {

                                edges.attr("x1", function(d) { return d.source.x; })
                                         .attr("y1", function(d) { return d.source.y; })
                                         .attr("x2", function(d) { return d.target.x; })
                                         .attr("y2", function(d) { return d.target.y; });

                                nodes.attr("cx", function(d) { return d.x; })
                                         .attr("cy", function(d) { return d.y; });

                        });

Geomapping

JSON, Meet GeoJSON

Get Lat+Lon is a great resource by Michal Migurski for doublechecking coordinate values.

Its core features from Squares. Squares is a small, extensible, free and open-source library for in-browser maps, written in Typescript and using D3 v2 under the hood..

Paths

var path = d3.geo.path();
var svg = d3.select("body")
    .append("svg")
    .attr("width", w)
    .attr("height", h);

//Load in GeoJSON data
d3.json("us-states.json", function(json) {
    //Bind data and create one path per GeoJSON feature
    svg.selectAll("path")
        .data(json.features)
        .enter()
        .append("path")
        .attr("d", path);
});

Projections

//Define map projection
var projection = d3.geo.albersUsa()
//Define path generator
var path = d3.geo.path()
                                 .projection(projection);
//Create SVG element
var svg = d3.select("body")
                        .append("svg")
                        .attr("width", w)
                        .attr("height", h);
//Load in GeoJSON data
d3.json("us-states.json", function(json) {
        //Bind data and create one path per GeoJSON feature
        svg.selectAll("path")
           .data(json.features)
           .enter()
           .append("path")
           .attr("d", path)
           .style("fill", "steelblue");

});

Choropleth

//Define map projection
var projection = d3.geo.albersUsa()
    .translate([w/2, h/2])
    .scale([500]);
//Define path generator
var path = d3.geo.path()
    .projection(projection);
//Define quantize scale to sort data values into buckets of color
var color = d3.scale.quantize()
    .range(["rgb(237,248,233)","rgb(186,228,179)","rgb(116,196,118)","rgb(49,163,84)","rgb(0,109,44)"]);
//Colors taken from colorbrewer.js, included in the D3 download
//Create SVG element
var svg = d3.select("body")
    .append("svg")
    .attr("width", w)
    .attr("height", h);
//Load in agriculture data
d3.csv("us-ag-productivity-2004.csv", function(data) {
    //Set input domain for color scale
    color.domain([
        d3.min(data, function(d) { return d.value; }), 
        d3.max(data, function(d) { return d.value; })
    ]);
    //Load in GeoJSON data
    d3.json("us-states.json", function(json) {
        //Merge the ag. data and GeoJSON
        //Loop through once for each ag. data value
        for (var i = 0; i < data.length; i++) {
            //Grab state name
            var dataState = data[i].state;
            //Grab data value, and convert from string to float
            var dataValue = parseFloat(data[i].value);
            //Find the corresponding state inside the GeoJSON
            for (var j = 0; j < json.features.length; j++) {
                var jsonState = json.features[j].properties.name;
                if (dataState == jsonState) {
                    //Copy the data value into the JSON
                    json.features[j].properties.value = dataValue;
                    //Stop looking through the JSON
                    break;
                }
            }               
        }
        //Bind data and create one path per GeoJSON feature
        svg.selectAll("path")
            .data(json.features)
            .enter()
            .append("path")
            .attr("d", path)
            .style("fill", function(d) {
                //Get data value
                var value = d.properties.value;
                if (value) {
                    //If value exists…
                    return color(value);
                } else {
                    //If value is undefined…
                    return "#ccc";
                }
            });

    });
});

More reference

Peter-Paul Koch’s event compatibility tables
Getting Starting with D3by Mike Dewar . O’Reilly, 2012
github.com/mbostock/d3/wiki/Gallery The D3 gallery contains hundredsof examples.
bl.ocks.org/mbostock Even more examples, in this case all by Mike Bostock, each one typically highlighting just one of D3’s features.
github.com/mbostock/d3/wiki/API-Reference The D3 API reference, an essential reference for every method and its parameters.
groups.google.com/forum/?fromgroups#!forum/d3-js Everyone who’s anyone is on the D3 Google Group.
bl.ocks.org A service for posting code hosted on GitHub’s Gist, by Mike Bostock.
blog.visual.ly/creating-animations-and-transitions-with-d3-js/ An excellent tutorial on Creating Animations and Transitions With D3 with lots of inline, interactive examples by Jérôme Cukier.
d3noob.org A new, promising resource for D3 tips and tricks.
tributary.io A live-coding environment for experimenting with D3 code, by Ian Johnson.
D3 Plug-ins A listing of all the official plug-ins that extend D3’s functionality, in case it doesn’t do enough for you already.