Image-processing pipelines require the design of complex workflows combining many different steps that bring the raw acquired data to a final result with biological meaning. In the image-processing domain of cryo-electron microscopy single-particle analysis (cryo-EM SPA), hundreds of steps must be performed to obtain the three-dimensional structure of a biological macromolecule by integrating data spread over thousands of micrographs containing millions of copies of allegedly the same macromolecule. The execution of such complicated workflows demands a specific tool to keep track of all these steps performed. Additionally, due to the extremely low signal-to-noise ratio (SNR), the estimation of any image parameter is heavily affected by noise resulting in a significant fraction of incorrect estimates. Although low SNR and processing millions of images by hundreds of sequential steps requiring substantial computational resources are specific to cryo-EM, these characteristics may be shared by other biological imaging domains. Here, we present Scipion, a Python generic open-source workflow engine specifically adapted for image processing. Its main characteristics are: (a) interoperability, (b) smart object model, (c) gluing operations, (d) comparison operations, (e) wide set of domain-specific operations, (f) execution in streaming, (g) smooth integration in high-performance computing environments, (h) execution with and without graphical capabilities, (i) flexible visualization, (j) user authentication and private access to private data, (k) scripting capabilities, (l) high performance, (m) traceability, (n) reproducibility, (o) self-reporting, (p) reusability, (q) extensibility, (r) software updates, and (s) non-restrictive software licensing.

Masaryk University Brno Czech Republic

National Center of Biotechnology Madrid Spain

St Jude Children's Research Hospital Memphis TN USA

Structural Studies Division MRC Laboratory of Molecular Biology Cambridge United Kingdom

Superior Polytechnic School Autonomous University of Madrid Madrid Spain

Zobrazit více v PubMed

Namba K & Makino F (2022) Recent progress and future perspective of electron cryomicroscopy for structural life sciences. Microscopy 71, i3–i14. PubMed

D’Imprima E & Kühlbrandt W (2021) Current limitations to high-resolution structure determination by single-particle cryoEM. Q Rev Biophys 54, e4. PubMed

Seffernick JT & Lindert S (2020) Hybrid methods for combined experimental and computational determination of protein structure. J Chem Phys 153, 240901. PubMed PMC

Sorzano COS & Carazo JM (2022) Cryo-electron microscopy: The field of 1,000+ methods. J Struct Biol 214, 107861. PubMed

Sorzano COS, Jiménez-Moreno A, Maluenda D, et al. (2022) On bias, variance, overfitting, gold standard and consensus in single-particle analysis by cryo-electron microscopy. Acta Crystallogr D Struct Biol 78, 410–423. PubMed PMC

Scipion protocols ranking. https://scipion.i2pc.es/report_protocols/protocolTable/.

de la Rosa-Trevín JM, Quintana A, Del Cano L, et al. (2016) Scipion: A software framework toward integration, reproducibility and validation in 3D electron microscopy. J Struct Biol 195, 93–99. PubMed

Martínez M, Jiménez-Moreno A, Maluenda D, et al. (2020) Integration of cryo-EM model building software in Scipion. J Chem Inf Model 60, 2533–2540. PubMed

Jiménez de la Morena J, Conesa P, Fonseca YC, et al. (2022) ScipionTomo: Towards cryo-electron tomography software integration, reproducibility, and validation. J Struct Biol 214, 107872. PubMed PMC

Bengtsson VEG, Pacoste L, de La Rosa-Trevin JM, et al. (2022) Scipion-ED: A graphical user interface for batch processing and analysis of 3D ED/MicroED data. J Appl Crystallogr 55, 638–646. PubMed PMC

Sorzano COS, Jiménez-Moreno A, Maluenda D, et al. (2021) Image processing in cryo-electron microscopy of single particles: The power of combining methods. Methods Mol Biol 2305, 257–289. PubMed

Střelák D, Filipovič J, Jiménez-Moreno A, Carazo JM & Sánchez Sorzano CÓ (2020) FlexAlign: An accurate and fast algorithm for movie alignment in cryo-electron microscopy. Electronics 9, 1040.

Sorzano COS, Jonic S, Núñez-Ramírez R, Boisset N & Carazo JM (2007) Fast, robust, and accurate determination of transmission electron microscopy contrast transfer function. J Struct Biol 160, 249–262. PubMed

Rohou A & Grigorieff N (2015) CTFFIND4: Fast and accurate defocus estimation from electron micrographs. J Struct Biol 192, 216–221. PubMed PMC

Zhang K (2016) Gctf: Real-time CTF determination and correction. J Struct Biol 193, 1–12. PubMed PMC

Vargas J, Álvarez-Cabrera A-L, Marabini R, Carazo JM & Sorzano COS (2014) Efficient initial volume determination from electron microscopy images of single particles. Bioinformatics 30, 2891–2898. PubMed

Kimanius D, Dong L, Sharov G, Nakane T & Scheres SHW (2021) New tools for automated cryo-EM single-particle analysis in RELION-4.0. Biochem J 478, 4169–4185. PubMed PMC

Punjani A, Rubinstein JL, Fleet DJ & Brubaker MA (2017) cryoSPARC: Algorithms for rapid unsupervised cryo-EM structure determination. Nat Methods 14, 290–296. PubMed

Tang G, Peng L, Baldwin PR, et al.. (2007) EMAN2: An extensible image processing suite for electron microscopy. J Struct Biol 157, 38–46. PubMed

Strelak D, Jiménez-Moreno A, Vilas JL, et al. (2021) Advances in Xmipp for cryo–electron microscopy: From Xmipp to Scipion. Molecules 26, 6224. PubMed PMC

Grant T, Rohou A & Grigorieff N (2018) cisTEM, user-friendly software for single-particle image processing. Elife 7, e35383. PubMed PMC

Zheng SQ, Palovcak E, Armache JP, Verba KA, Cheng Y & Agard DA (2017) MotionCor2: Anisotropic correction of beam-induced motion for improved cryo-electron microscopy. Nat Methods 14, 331–332. PubMed PMC

Hall SR (1991) The STAR file: A new format for electronic data transfer and archiving. J Chem Inf Comput Sci 31, 326–333.

Harris CR, Millman KJ, Van Der Walt SJ, et al. (2020) Array programming with NumPy. Nature 585, 357–362. PubMed PMC

Sorzano COS, Marabini R, Vargas J, et al. (2014) Interchanging geometry conventions in 3DEM: Mathematical context for the development of standards. In Computational Methods for Three-Dimensional Microscopy Reconstruction, pp. 7–42 [Herman GT & Frank J, editors]. New York: Springer.

Bepler T, Morin A, Rapp M, et al. (2019) Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153–1160. PubMed PMC

Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. (2016) The FAIR guiding principles for scientific data management and stewardship. Sci Data 3, 160018. PubMed PMC

Vargas J, Abrishami V, Marabini R, et al. (2013) Particle quality assessment and sorting for automatic and semiautomatic particle-picking techniques. J Struct Biol 183, 342–353. PubMed

Sorzano COS, Fernández-Giménez E, Peredo-Robinson V, et al. (2018) Blind estimation of DED camera gain in electron microscopy. J Struct Biol 203, 90–93. PubMed

Jalili V, Afgan E, Gu Q, et al. (2020) The galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2020 update. Nucleic Acids Res 48, W395–W402. PubMed PMC

Indiana U cryo-EM pricing. https://medicine.iu.edu/service-cores/facilities/electron-microscopy/pricing.

Cianfrocco MA & Leschziner AE (2015) Low cost, high performance processing of single particle cryo-electron microscopy data in the cloud. Elife 4, e06664. PubMed PMC

Wikipedia Contributors (2022) Job scheduler. Wikipedia. https://en.wikipedia.org/w/index.php?title=Job_scheduler&oldid=1066771074.

Slurm workload manager – documentation. https://slurm.schedmd.com/.

Torque Resource Manager (2019) Adaptive Computing. https://adaptivecomputing.com/cherry-services/torque-resource-manager/.

Host configuration—Scipion 3.0.0 documentation. https://scipion-em.github.io/docs/release-3.0.0/docs/scipion-modes/host-configuration.html.

Scipion in the cloud—Scipion 3.0.0 documentation. https://scipion-em.github.io/docs/release-3.0.0/docs/developer/scipion-on-the-cloud.html.

Scipion-docker. (Github). https://github.com/i2pc/scipion-docker

Wagner T, Merino F, Stabrin M, et al. (2019) SPHIRE-crYOLO is a fast and accurate fully automated particle picker for cryo-EM. Commun Biol 2, 218. PubMed PMC

Pettersen EF, Goddard TD, Huang CC, et al. (2021) UCSF ChimeraX: Structure visualization for researchers, educators, and developers. Protein Sci 30, 70–82. PubMed PMC

Heymann JB & Belnap DM (2007) Bsoft: Image processing and molecular modeling for electron microscopy. J Struct Biol 157, 3–18. PubMed

Vilas JL, Gómez-Blanco J, Conesa P, et al. (2018) MonoRes: Automatic and accurate estimation of local resolution for electron microscopy maps. Structure 26, 337–344.e4. PubMed

Martinez-Sanchez A (2021) PySeg in Scipion: Making easier template-free detection and classification of membrane-bound complexes in cryo-electron tomograms. Acta Crystallogr A Found Adv 77, a231–a231.

Iudin A, Korir PK, Salavert-Torres J, Kleywegt GJ & Patwardhan A (2016) EMPIAR: A public archive for raw electron microscopy image data. Nat Methods 13, 387–388. PubMed

Web-workflow-viewer: HTML workflow viewer based on workflow json files exported from Scipion. (Github).

CryoEM workflow viewer. https://scipion.cnb.csic.es/cryoemworkflowviewer.

Streaming workflows—Scipion 3.0.0 documentation. https://scipion-em.github.io/docs/release-3.0.0/docs/facilities/facilities-workflows.html.

Scipion-chem: Base Scipion plugin defining objects and protocols for CHEMoinformatics. (Github).

Performance and Quality Comparison of Movie Alignment Software for Cryogenic Electron Microscopy