Content-aware image restoration: pushing the limits of fluorescence microscopy
Nature Methods, (2018)DOI: 10.1038/s41592-018-0216-7
Abstract
F luorescence microscopy is an indispensable tool in the life sciences for investigating the spatio-temporal dynamics of cells, tissues, and developing organisms. Recent advances such as light-sheet microscopy 1-3 , structured illumination microscopy 4,5 , and super-resolution microscopy 6-8 enable time-resolved volumetric imaging of biological processes within cells at high resolution. The quality at which these processes can be faithfully recorded, however, is determined not only by the spatial resolution of the optical device used, but also by the desired temporal resolution, the total duration of an experiment, the required imaging depth, the achievable fluo-rophore density, bleaching, and photo-toxicity 9,10. These aspects cannot all be optimized at the same time-trade-offs must be made, for example, by sacrificing signal-to-noise ratio (SNR) by reducing exposure time to gain imaging speed. Such trade-offs are often depicted by a design space that has resolution, speed, light exposure, and imaging depth as its dimensions (Fig. 1a), with the volume being limited by the maximal photon budget compatible with sample health 11,12. These trade-offs can be addressed through optimization of the microscopy hardware, yet there are physical limits that cannot easily be overcome. Therefore, computational procedures to improve the quality of acquired microscopy images are becoming increasingly important. Super-resolution microscopy 4,13-16 , deconvolution 17-19 , surface projection algorithms 20,21 , and denoising methods 22-24 are examples of sophisticated image restoration algorithms that can push the limit of the design space, and thus allow the recovery of important biological information that would be inaccessible by imaging alone. However, most common image restoration problems have multiple possible solutions, and require additional assumptions to select one solution as the final restoration. These assumptions are typically general, for example, requiring a certain level of smoothness of the restored image, and therefore are not dependent on the specific content of the images to be restored. Intuitively, a method that leverages available knowledge about the data at hand ought to yield superior restoration results. Deep learning is such a method, because it can learn to perform complex tasks on specific data by employing multilayered artificial neural networks trained on a large body of adequately annotated example data 25,26. In biology, deep learning methods have, for instance, been applied to the automatic extraction of connectomes from large electron microscopy data 27 , for classification of image-based high-content screens 28 , fluorescence signal prediction from label-free images 29,30 , resolution enhancement in histopathology 31 , or for single-molecule localization in super-resolution micros-copy 32,33. However, the direct application of deep learning methods to image restoration tasks in fluorescence microscopy is complicated by the absence of adequate training data and the fact that it is impossible to generate them manually. We present a solution to the problem of missing training data for deep learning in fluorescence microscopy by developing strategies to generate such data. This enables us to apply common convolutional neural network architectures (U-Nets 34) to image restoration tasks, such as image denoising, surface projection, recovery of isotropic resolution, and the restoration of sub-diffraction structures. We show, in a variety of imaging scenarios, that trained Fluorescence microscopy is a key driver of discoveries in the life sciences, with observable phenomena being limited by the optics of the microscope, the chemistry of the fluorophores, and the maximum photon exposure tolerated by the sample. These limits necessitate trade-offs between imaging speed, spatial resolution, light exposure, and imaging depth. In this work we show how content-aware image restoration based on deep learning extends the range of biological phenomena observable by microscopy. We demonstrate on eight concrete examples how microscopy images can be restored even if 60-fold fewer photons are used during acquisition, how near isotropic resolution can be achieved with up to tenfold under-sampling along the axial direction, and how tubular and granular structures smaller than the diffraction limit can be resolved at 20-times-higher frame rates compared to state-of-the-art methods. All developed image restoration methods are freely available as open source software in Python, FIJI, and KNIME. Articles NATurE METhods content-aware image restoration (CARE) networks produce results that were previously unobtainable. This means that the application of CARE to biological images transcends the limitations of the design space (Fig. 1a), pushing the limits of the possible in fluo-rescence microscopy through machine-learned image computation.