machine learning + visualization + connectomics

Rainbow Brain Chip Brain

In connectomics we create a map of the mammalian brain at nano-scale. For this one must acquire image stacks of the brain of a rodent using electron microscopy. These images are of such high resolution that individual neurons (nerve cells) and their connections (synapses) are then visible. Once these images are acquired, machine learning algorithms then classify cell structures and connections within these massive (terabytes or petabytes) images as manual processing is, thus far not possible.

Neurons in Rat Brain
A slice of rat brain

Understanding the wiring of the brain, curing mental and neurological diseases, and deriving new artificial intelligence methods are ultimate goals. While these goals are far from achieved, the next developmental milestones are a fully processed 100 micron cube followed by its’ extension to a 1 millimeter cube of brain tissue. The video on the right shows selected neurons in a 100 micron cube.

[1] A. Suissa-Peleg, D. Haehn, S. Knowles-Barley, V. Kaynig, T.R. Jones, A. Wilson, R. Schalek, J.W. Lichtman, H. Pfister: Automatic Neural Reconstruction from Petavoxel of Electron Microscopy Data, Microscopy and Microanalysis, 2016.
[2] R. Schalek, D. Lee, N. Kasthuri, A. Suissa-Peleg, T.R. Jones, V. Kaynig, D. Haehn, H. Pfister, D. Cox, J.W. Lichtman: Imaging a 1 mm 3 Volume of Rat Cortex Using a MultiBeam SEM, Microscopy and Microanalysis, 2016.


Currently, the automatic classification of cells and connections is far from absolute. Humans are still needed to double-check the results and this task is called proofreading. In 2014 we published Dojo, which is an interactive proofreading software that enables proofreading for untrained individuals recruited from all walks of life.

Interactive Proofreading using Dojo
The Dojo proofreading software to fix automatic labelings

All of the results and data that have been gathered from the published user study can be found in The Proofreading Benchmark.

We discovered that the majority of time spent during interactive proofreading is looking for errors. To reduce this, we developed the Guided Proofreading system. The artificial intelligence within our system suggests potential errors and corrections to a user thus speeding up the proofreading task. Our results have demonstrated our trained classifier is also able to perform proofreading automatically up to a certain threshold. For increased performance, the Guided Proofreading user interface simplifies error correction by humans to a yes/no decision (see video on the left).

[3] D. Haehn, S. Knowles-Barley, M. Roberts, J. Beyer, N. Kasthuri, J.W. Lichtman, H. Pfister: Design and evaluation of interactive proofreading tools for connectomics, IEEE transactions on visualization and computer graphics, 2014.
[4] D. Haehn, V. Kaynig, J. Tompkin, J.W. Lichtman, H. Pfister: Guided Proofreading of Automatic Segmentations for Connectomics, CVPR, 2018.


Brain data is beautiful and not only at a nano-scale resolution. In 2012 we developed XTK, the first web-based visualization framework for medical imaging data such as MRI scans. This framework quickly became an established collaborative effort with over twenty international contributors which powers many projects world-wide.

My Brain visualized using XTK
A glimpse into my brain rendered from an MRI scan using XTK

One example of these projects is our award-winning software, Slice:Drop, a web-based viewer that supports many different medical imaging formats. Not only clinicians but researchers, as well as patients, now use this viewer daily. The software visualizes data without requiring any server uploads and leverages the local graphics card for accelerated rendering.

The Slice:Drop viewer
The Slice:Drop viewer supports different visualizations

In turn, Connectomics data is much larger in size thus making visualization more difficult. We have successfully applied new processing strategies to overcome these challenges and have developed the MBeam viewer for the ZEISS MultiSEM 505 microscope. This software allows neuroscientists to view the high resolution images immediately after acquisition. This process has been so successful that Zeiss, the manufacturer of the microscope, installed the MBeam viewer internally.

The ZEISS MultiSEM 505 microscope with Jeff Lichtman in front of it The MBeam viewer
The ZEISS MultiSEM 505 microscope (with Jeff Lichtman in front of it) and the MBeam viewer

From an engineering standpoint, the MBeam viewer along with Dojo, our proofreading software, provide overlapping functionality such as the processing of data for transfer and visualization.


To avoid duplication, we have now developed Butterfly, a system for scalable data management which unifies logic and visualization. As a result we can now create visualizations of large scientific datasets for every step of the connectomics processing pipeline.

The Connectomics workflow
Our Butterfly application suite includes specifically targeted visualizations for every step of the connectomics workflow and is fully extendible for future visualizations

In addition, we have recently added the Neural Data Queries interface to the Butterfly platform, allowing semantic queries of neurons and their synaptic connections. This functionality is a core component in evaluating the performance of our connectomics pipeline for the Machine Intelligence from Cortical Networks (MICroNS) effort, a $100 million research program backed by the Intelligence Advanced Research Projects Activity (IARPA).

[5] D. Haehn, N. Rannou, B. Ahtam, E. Grant, R. Pienaar: Neuroimaging in the browser using the X Toolkit, Frontiers in Neuroinformatics, 2014.
[6] D. Haehn, N. Rannou, E. Grant, R. Pienaar: Slice: drop: Collaborative medical imaging in the browser, ACM SIGGRAPH Computer Animation Festival, 2013.
[7] D. Haehn, J. Hoffer, B. Matejek, A. Suissa-Peleg, A.K. Al-Awami, L. Kamentsky, F. Gonda, E. Meng, W. Zhang, R. Schalek, A. Wilson, T. Parag, J. Beyer, V. Kaynig, T.R. Jones, J. Tompkin, M. Hadwiger, J.W. Lichtman, H. Pfister: Scalable Interactive Visualization for Connectomics, Informatics, 2017.

further reading

Machine Intelligence from Cortical Networks (MICrONS)

Terascale Neuroscience

AMI.js: Medical Imaging JavaScript ToolKit

A Transfer Function Editor for Browsers

Neuroglancer: a WebGL-based viewer for volumetric data

Machine Learning for humans

The Visual Computing Group @ Harvard

Written on March 2, 2017