Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D

doi:10.1073/pnas.1804420115

. 2018 Sep 4;115(36):9026-9031.

doi: 10.1073/pnas.1804420115. Epub 2018 Aug 22.

Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D

Jay M Newby ^{1

2}, Alison M Schaefer ^{3

4}, Phoebe T Lee ⁴, M Gregory Forest ^{5

4

6}, Samuel K Lai ^{7

4

8}

Affiliations

¹ Department of Mathematics, University of Alberta, Edmonton, AB, Canada T6G 2R3.
² Department of Mathematics, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.
³ Division of Pharmacoengineering and Molecular Pharmaceutics, Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.
⁴ Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.
⁵ Department of Mathematics, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599; lai@unc.edu forest@unc.edu.
⁶ Department of Mathematics and Applied Physical Sciences, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.
⁷ Division of Pharmacoengineering and Molecular Pharmaceutics, Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599; lai@unc.edu forest@unc.edu.
⁸ Department of Microbiology and Immunology, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.

PMID: 30135100
PMCID: PMC6130393
DOI: 10.1073/pnas.1804420115

Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D

Jay M Newby et al. Proc Natl Acad Sci U S A. 2018.

. 2018 Sep 4;115(36):9026-9031.

doi: 10.1073/pnas.1804420115. Epub 2018 Aug 22.

Authors

Jay M Newby ^{1

2}, Alison M Schaefer ^{3

4}, Phoebe T Lee ⁴, M Gregory Forest ^{5

4

6}, Samuel K Lai ^{7

4

8}

Affiliations

¹ Department of Mathematics, University of Alberta, Edmonton, AB, Canada T6G 2R3.
² Department of Mathematics, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.
³ Division of Pharmacoengineering and Molecular Pharmaceutics, Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.
⁴ Joint Department of Biomedical Engineering, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.
⁵ Department of Mathematics, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599; lai@unc.edu forest@unc.edu.
⁶ Department of Mathematics and Applied Physical Sciences, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.
⁷ Division of Pharmacoengineering and Molecular Pharmaceutics, Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599; lai@unc.edu forest@unc.edu.
⁸ Department of Microbiology and Immunology, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599.

PMID: 30135100
PMCID: PMC6130393
DOI: 10.1073/pnas.1804420115

Abstract

Particle tracking is a powerful biophysical tool that requires conversion of large video files into position time series, i.e., traces of the species of interest for data analysis. Current tracking methods, based on a limited set of input parameters to identify bright objects, are ill-equipped to handle the spectrum of spatiotemporal heterogeneity and poor signal-to-noise ratios typically presented by submicron species in complex biological environments. Extensive user involvement is frequently necessary to optimize and execute tracking methods, which is not only inefficient but introduces user bias. To develop a fully automated tracking method, we developed a convolutional neural network for particle localization from image data, comprising over 6,000 parameters, and used machine learning techniques to train the network on a diverse portfolio of video conditions. The neural network tracker provides unprecedented automation and accuracy, with exceptionally low false positive and false negative rates on both 2D and 3D simulated videos and 2D experimental videos of difficult-to-track species.

Keywords: artificial neural network; bioimaging; machine learning; particle tracking; quantitative biology.

PubMed Disclaimer

Conflict of interest statement

Conflict of interest statement: J.M.N, M.G.F., and S.K.L. are the founders of and maintain a financial interest in AI Tracking Solutions, which is actively seeking to commercialize the neural network tracker technology. The terms of these arrangements are being managed by The University of North Carolina in accordance with its conflict of interest policies. The remaining authors declare no competing financial interests.

Figures

Fig. 1.

Fig. 1.

Sample frames from experimental videos, highlighting some of the challenging conditions for particle tracking. (Left to Right) Fifty-nanometer particles captured at low SNR and 200-nm particles with diffraction disc patterns, variable background intensity, and ellipsoid PSF shapes from

1 - 2 μ

m Salmonella.

Fig. 2.

Fig. 2.

The CNN. Diagram of the layered connectivity of the artificial neural network.

Fig. 3.

Fig. 3.

Performance analysis for randomized 2D and 3D synthetic test videos. (A–C) Two-dimensional test results showing the (A) percentage of false positives, (B) percentage of false negatives, and (C) predictions per frame vs. SNR. Mosaic shows a sharp rise in false positives for

S N R < 2

(in A), due to substantially more predictions than actual particles (in C). Conversely, the neural net (NN) and Icy showed no increase in false positives at low SNR. (E–G) Results showing the (E) percentage of false positives, (F) percentage of false negatives, and (G) localization error vs. the PSF radius. (D and H) Results showing the (D) percentage of false positives and (H) measured diffusivity vs. the ground truth particle diffusivity. (I–L) Violin plots showing the performance on 2D and 3D test videos for each of the four methods: the NN, Mos, Icy, and VST. The solid black lines show the mean, and the thickness of the filled regions shows the shape of the histogram obtained from 500 (50) randomized 2D (3D) test videos. Note that the VST results only included 100 test videos.

See this image and copyright information in PMC

References

1. Wang Y-Y, et al. IgG in cervicovaginal mucus traps HSV and prevents vaginal Herpes infections. Mucosal Immunol. 2014;7:1036–1044. - PMC - PubMed
1. Wang Y-Y, et al. Influenza-binding antibodies immobilize influenza viruses in fresh human airway mucus. Eur Respir J. 2016;49:1601709. - PMC - PubMed
1. Lai SK, et al. Rapid transport of large polymeric nanoparticles in fresh undiluted human mucus. Proc Natl Acad Sci USA. 2007;104:1482–1487. - PMC - PubMed
1. Yang M, et al. Biodegradable nanoparticles composed entirely of safe materials that rapidly penetrate human mucus. Angew Chem Int Ed. 2011;50:2597–2600. - PMC - PubMed
1. Vasquez PA, et al. Entropy gives rise to topologically associating domains. Nucleic Acids Res. 2016;44:5540–5549. - PMC - PubMed

Publication types

Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

R41 GM123897/GM/NIGMS NIH HHS/United States

LinkOut - more resources

Full Text Sources
Other Literature Sources
- The Lens - Patent Citations Database
- scite Smart Citations

[1] Wang Y-Y, et al. IgG in cervicovaginal mucus traps HSV and prevents vaginal Herpes infections. Mucosal Immunol. 2014;7:1036–1044. - PMC - PubMed

[2] Wang Y-Y, et al. IgG in cervicovaginal mucus traps HSV and prevents vaginal Herpes infections. Mucosal Immunol. 2014;7:1036–1044. - PMC - PubMed

[3] Wang Y-Y, et al. Influenza-binding antibodies immobilize influenza viruses in fresh human airway mucus. Eur Respir J. 2016;49:1601709. - PMC - PubMed

[4] Wang Y-Y, et al. Influenza-binding antibodies immobilize influenza viruses in fresh human airway mucus. Eur Respir J. 2016;49:1601709. - PMC - PubMed

[5] Lai SK, et al. Rapid transport of large polymeric nanoparticles in fresh undiluted human mucus. Proc Natl Acad Sci USA. 2007;104:1482–1487. - PMC - PubMed

[6] Lai SK, et al. Rapid transport of large polymeric nanoparticles in fresh undiluted human mucus. Proc Natl Acad Sci USA. 2007;104:1482–1487. - PMC - PubMed

[7] Yang M, et al. Biodegradable nanoparticles composed entirely of safe materials that rapidly penetrate human mucus. Angew Chem Int Ed. 2011;50:2597–2600. - PMC - PubMed

[8] Yang M, et al. Biodegradable nanoparticles composed entirely of safe materials that rapidly penetrate human mucus. Angew Chem Int Ed. 2011;50:2597–2600. - PMC - PubMed

[9] Vasquez PA, et al. Entropy gives rise to topologically associating domains. Nucleic Acids Res. 2016;44:5540–5549. - PMC - PubMed

[10] Vasquez PA, et al. Entropy gives rise to topologically associating domains. Nucleic Acids Res. 2016;44:5540–5549. - PMC - PubMed

Account

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D

Affiliations

Convolutional neural networks automate detection for tracking of submicron-scale particles in 2D and 3D

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources