CryoSPARC v4.6 Now AvailableCryoSPARC v4.6 Now Available: Featuring 2x speedups across refinement and classification jobs, new tree and card views with collapsible job groups, automatic Inspect Picks, and more!
Learn more →
We’re excited to announce another release of CryoSPARC! Version 4.6 features 2x speedups across refinement and classification jobs, new tree and card views with collapsible job groups, automatic Inspect Picks, and more!
Note
Support for CentOS Linux 7 operating system is deprecated. CryoSPARC v4.6 will continue to function on CentOS 7, but future versions will no longer support this operating system. CentOS Linux 7 reached end of life on June 30, 2024.
New
Up to 2.1x speedup across refinement and classification jobs due to a redesign of CryoSPARC’s I/O system that provides significant optimizations to particle read times.
In v4.6.0, a homogeneous refinement of 2.3M particles with box size 300 can now be completed in just 27 minutes.
New
Rebuilt tree, card, and table views for jobs, with significant performance optimizations to support large workspaces and projects. Up to 10x faster initial load time for the tree view, and subsequent updates when connecting and creating jobs are nearly instantaneous. In the card and table views, all jobs are now instantly accessible with detailed information loading progressively as you scroll.
New
Job Groups: collapse multiple jobs into a single card to maintain legibility and easy navigation within a workspace. Job Groups work across the tree, card and table views, and jobs can be easily added and removed from groups using multi-selection and the right-click menu. Learn more
New
Improved navigation: the job search menu and targeting system allow you to quickly find and automatically navigate to any particular job, or quickly jump to the latest created, completed, or running job.
New
Inspect Particle Picks now has a parameter called “Auto cluster”, which clusters particles based on their NCC and power scores, and can automatically select a cluster that will ideally contain true particle picks based on those scores. This allows Inspect Particle Picks to be run without user intervention. Auto cluster is designed to be used with denoised micrographs from the Micrograph Denoiser job.
Labels designate type of output when downloading results from the job preview dialog output groups panel (e.g., map, mask, etc.).
Update
Added the ability to mark or unmark a job as final from the job sidebar actions panel. Read more about data cleanup in this guide.
Update
Added ability to sort projects, workspaces, and sessions by their titles.
Update
Added filter bar toggles to the projects view to quickly show and hide archived and/or detached projects.
Update
Archived and detached project filters are now additive, and can be toggled together to show both archived and detached projects, instead of just archived or detached projects.
Update
Introduced a quick action to create Homogeneous Reconstruction Only from an Import Particle Stack job.
Update
Unused particles will be connected when building a Homogeneous Refinement, Non-uniform Refinement or Heterogeneous Refinement from a quick action on Ab-inito.
Update
Introduced a quick action to build a Blob Picker Tuner job from Inspect Particle Picks.
Update
Added completed, failed or killed timing to the tooltip that displays when viewing the elapsed time of a job on the card.
Update
Downloading a CSV of projects, workspaces, sessions and jobs will include relevant project or workspace ID and a timestamp representing when the list was generated.
Update
Cluster configuration variables are now listed in ascending alphabetical order.
Update
Various improvements to job informational tags: Support for Homogeneous and Heterogeneous Reconstruct Only jobs, Streaming Homogeneous Refinement, and symmetry relaxation parameters.
Update
The Live Session data management tab is now deprecated and as such, a notice displays within the tab. It is recommended to perform Live Session compaction or restoration as a more robust way to compress, archive and restore sessions.
Fixed
Resolved an issue where exposure pixel size shown in the Live session sidebar would not account for EER upsampling.
Fixed
Reduced memory usage and lower likelihood of out-of-memory errors during data clean-up operations, job input loading and job output result saving.
Fixed
Deep Picker Train and Deep Picker Inference jobs are now restricted in workflows, and are only allowed if they do not include any child jobs inside of the workflow.
Fixed
Clicking the job’s workspace in the details panel will no longer reset the current view to cards view.
Fixed
Fixed an issue where the custom parameters tooltip in the job card’s footer would continue to be hidden even after queuing a job.
Fixed
Fixed issue where some workspaces were not being sorted into the correct order when sorting by job count.
Fixed
Fixed issue where job CSV downloads were using the job’s created at timestamp to populate the “Status Updated” field.
Fixed
Resolved a case where filesystem errors could case the web application server to crash when downloading job results.
Fixed
Resolved an issue where refinements with downstream Local Resolution Estimation jobs did not have an option to load a colour map within the volume viewer tab.
Fixed
Resolved an issue in the job report PDF where parameters set to false were reported as ‘Not Set’. Instead, boolean parameters are now listed as ‘True’ or ‘False’.
Fixed
Denoised micrographs will now be correctly scaled to the canvas and line up with pick locations when downloaded from the exposure viewer.
Fixed
Resolved an issue where jobs queued from the sidebar would sometimes show a perpetual loading indicator until deselected.
The volume viewer now has improved reliability, with reduced likelihood of issues when multiple large volumes are loaded.
Update
Streaming refinement in CryoSPARC Live now re-scales a provided input volume and mask to match the extent and pixel size of the extracted particles.
Fixed
Reference Based Motion Correction no longer fails with an error indicating that there are micrographs with fewer than 2 particles, when particle re-centering is turned on.
Fixed
Import Result Group job no longer fails when importing a result containing a Topaz model.
Fixed
Fixed a bug that caused the 3D Flex Generator job to run out of GPU memory for large datasets.
Fixed
Ab-initio Reconstruction no longer fails when upstream particles have alignments3D metadata attached, and the number of particles parameter is set.
Fixed
Fixed a bug that would cause the 3D Classification job to fail when class re-ordering was on, and the number of classes exceeded 12.
Fixed
Orientation Diagnostics no longer fails when using imported half-maps.
Fixed
Removed unnecessary SSD requirement in the Regroup 3D job.
Fixed
Removed unused parameter “Recenter templates” from the Blob Picker job.
CryoSPARC v4.5.3
June, 4, 2024
This release contains various stability improvements:
New
Check for Corrupt Micrographs job tests for and separates micrograph files that cannot be successfully read from disk.
Fixed
Refinement jobs default parameters have been changed so that the jobs use the half-set split from the input particles by default. This ensures that repeated application of Reference Based Motion Correction and refinement does not cause mixing of the half-sets. The Particle Sets Tool job also now has an option to balance unequal half-set splits.
Fixed
Orientation Diagnostics no longer fails on CPU-only worker nodes, and requests the correct number of CPUs when launching on a cluster.
Fixed
Relative signal plots in Orientation Diagnostics jobs are now less crowded.
Fixed
The Topaz Extract job now allows input in units of estimated particle diameter for the radius of extracted region parameter.
Fixed
Import Result Group no longer fails to import exposures when some paths are missing.
Fixed
Regroup 3D no longer outputs incorrect particles when classes are manually excluded.
Fixed
Fixed an issue where for some large CryoSPARC Live sessions, scrolling through the exposure feed to earlier exposures could sometimes cause the application server to crash with a memory leak.
Fixed
Excess UserWarnings are no longer printed in job log output.
Fixed
The default value for CRYOSPARC_JOB_LAUNCH_TIMEOUT is now extended to 60 seconds.
CryoSPARC v4.5.1
May 8, 2024
This release contains various stability improvements:
Fixed
Micrograph Denoiser no longer fails on launch due to PicklingError
Fixed
Scale bars on the Import Movies job card are now the correct size
Fixed
Gracefully handle missing chunks when attempting to save a file to the uploads directory via the local upload tool
CryoSPARC v4.5
May 7, 2024
We’re excited to announce another release of CryoSPARC! Version 4.5 includes a new micrograph denoiser, automatic 2D/3D class selection using a reference, upload of files directly through the UI, improvements to Workflows, orientation diagnostics, 3D classification, and motion correction, new tools for handling and aligning multiple 3D volumes, and many feature updates, fixes, and optimizations!
New
Micrograph Denoiser Beta job produces enhanced contrast in micrograph images to aid particle picking and visual inspection.
Micrograph Denoiser uses a neural network model that can be trained on as few as 100 exposures from a new dataset in just a few minutes on a single GPU. The trained denoiser can then be used on new micrographs from the same or a different dataset. A pre-trained general denoiser model is also available to use without any training required.
The denoiser model is designed to work across a wide range of defocus and contrast levels.
Denoised micrographs can be used seamlessly in blob and template picking jobs, and can substantially improve picking results. Denoised micrographs also show by default in all interactive jobs, making visual inspection of micrographs easy.
These jobs align and compare classes to the reference volume and select classes either based on score thresholds or clustering.
The jobs can be set up once on a dataset, and can subsequently allow one-click automated processing on new similar datasets using Workflows.
The jobs also make it easy to sort and select from hundreds of 2D classes or dozens 3D classes in datasets with substantial heterogeneity.
New
Upload files from your local computer to the CryoSPARC system directly in the browser. This new ability streamlines workflows involving local visualization and editing of files such as 3D maps or masks. Uploaded files are saved to the selected project directory and can be imported and used in jobs. Read more in the guide.
New
Orientation Diagnostics job now reports Relative Signal — a new diagnostic that can help directly identify underrepresented views that affect map quality and the cFAR score. Relative signal is computed by considering FSCs within toroidal sections of Fourier half-maps to relate 3D signal quality to viewing direction. Two new plots are produced that show where on the viewing sphere relative signal is attenuated, with sample projections and an embedded structure to simplify interpretation.
New
New Rebalance Orientations job removes particles from overpopulated viewing direction regions. Rebalancing viewing directions can improve downstream refinements when particles have a highly non-uniform orientation distribution.
New
Jobs that produce multiple 3D volumes (such as 3D classification, Heterogeneous Refinement, 3D Variability Display, etc.) now output a “Volumes group” output that contains multiple volumes, and can be connected to a downstream job with a single input connection. This enables seamless handling of arbitrary numbers of 3D volumes and automation using Workflows.
New
Align 3D Maps has been reimplemented with a new 3D alignment method to increase the robustness and precision of volume-to-volume alignment. In addition, it can now:
update particle alignments while searching over handedness,
transform all maps and masks associated with an input volume slot,
and accept inputs of different box and pixel sizes.
New
The file browser now features an expandable sidebar to search and navigate into project directories. Bookmark commonly-accessed directories, making them available for access at a later time through the sidebar. Read more in the guide.
New
Improved ‘Inputs and Parameters’ tab for jobs now displays additional input group information in a summarized view while retaining an option to view low-level connection details. By default only custom parameters are shown, making it easier to identify which parameters were set for a particular job.
New
Additional info tags now display on the job card with useful stats and summary data, for more jobs including Import Volumes, Exposure Curation, Exposure Sets Tool, and Helical Refinement. New quick actions allow quick creation and connection of many more job types in typical workflows.
New
The details section of the job sidebar now displays a more comprehensive summary of parents, children, and cloned jobs when hovered, including job type, status, summary stats, and custom parameters.
Update
Support has been added for Falcon C camera .eer movies (2k x 2k physical pixels).
Input particles are now re-centred based on their aligned shifts, at the start of the job. This feature can be switched off. The upsampling factor for .eer movies can now be overridden.
The job has been improved to be significantly more stable. An option has been added to prevent crashing if the input movies contain different frame counts (the option is off by default). This option relies on the assumption that the most common frame count is “correct”, and will discard movies that don’t match that frame count. The job also no longer terminates if the distances between particles results in a singular covariance matrix, as a pseudo-inversion is now used. The chance of getting NaN values in the highest-frequency column of the FCC plot (due to low signal near the FSC cut-off frequency) has been greatly reduced. The job also no longer crashes if the upstream motion correction skipped the movie’s first frame.
The job now outputs movies or particles that were rejected, as separate output groups. Trajectory plots now correctly reflect the micrograph’s aspect ratio. The mask_refine low-level input is now optional (a mask must be supplied, but if the separate static mask connection is used, mask_refine need not be present).
It is now possible to prevent the GPU responsible for generating 2D reference projections from also computing trajectory estimates. Using this option reduces VRAM consumption but requires the use of at least 2 GPUs.
Update
Refinement jobs now produce a cFSC summary plot at every iteration, enabling anisotropy tracking throughout optimization (this includes homogeneous, non-uniform, helical, and local refinement). A new fast GPU-based implementation of the core cFSC computation adds minimal overhead, requiring less than a second of additional compute time for typical box sizes.
In line with 3D Variability Analysis, the ‘Target Resolution’ parameter has been renamed to ‘Filter Resolution’ and must be explicitly set for the job to run. Also, the secondary RMS-based convergence criterion is now turned on by default. This secondary criterion can significantly reduce run times without substantially affecting the final class volumes.
The job now includes particle pre-processing and symmetry enforcement, and class re-ordering is now turned off by default to avoid potential confusion.
The job tile figure is now revamped to show class ids, relative populations, and a more distinctive difference-from-consensus projection for each class.
Update
In Downsample Particles, users can now specify the desired approximate pixel size after Fourier-cropping, instead of specifying the Fourier-crop box size directly, if desired.
Update
The 3D Flex Generator Beta latent space plot is now coloured according to the normalized difference in 3D deformation relative to the origin. Larger values indicate regions of the latent space that encode more substantial flexing of the particle.
Update
Regroup 3D now allows for pairwise alignment of volumes prior to regrouping, as well as alignment of volumes within a superclass after regrouping. It also now allows regrouping of all volumes into one single superclass.
Update
The Manual Picker interactive job now writes manually picked particle locations to disk every 10 seconds, to minimize data loss in case the job crashes.
Update
The new and more reliable SSD cache system introduced in v4.4 is now enabled by default. It also now features a configurable distributed locking strategy to use when multiple running jobs access the SSD cache simultaneously; the default strategy is best in most cases, but use the master strategy for caches on distributed file systems such as GPFS and BeeGFS where POSIX locks are disabled or unavailable. See guide for details.
Update
Deleting a project now completely removes the project folder on disk instead of leaving an empty folder.
Update
Archived projects can now be deleted. Deleting an archived project will clear any remaining database references to the project and should clear up more space on disk. A deleted archived project cannot be un-archived. For more details about archiving, see the Data Management Guide.
Update
Projects that are archived, detached, or deleted can no longer be shared with other users.
Update
Reduced default RAM requirements for some utility jobs.
Update
Jobs will provide a warning before starting if the amount of RAM available is less than the amount of RAM requested by the job.
Update
Jobs will no longer fail on completion if they cannot generate their passthrough outputs properly. They will instead raise a warning and continue to completion with potentially missing passthroughs.
Update
Running Export Exposures from CryoSPARC Live sessions now provides additional information about curation thresholds that were set in the session.
Update
When maintenance mode is active, a banner will display in the header when viewing CryoSPARC Live sessions communicating to users that additional jobs or sessions will not be queued.
Update
Ability to specify an arbitrary number of pre-processing GPU workers for a CryoSPARC Live session.
Update
The Curate Exposures interactive job can now load micrograph thumbnails that were generated in CryoSPARC Live.
Volume Tools now reports the centre of mass of the input volume, if it is a mask.
Update
Volume Alignment Tools now has the option to automatically re-centre the input volume and particles based on the centre of mass of the supplied mask, especially useful for local refinements.
Rebalance 2D Classes now always produces a “Particles Selected” output, containing selected particles from all of the superclasses.
Update
Added parameters Use spherical mask, Spherical mask diameter (A), and Spherical mask diameter outer (A) to allow users to customize the spherical mask used in Heterogenous Refinement. The default behaviour is still the same.
Update
Orientation Diagnostics will now use the mask_fsc_auto mask present within the input volume connection, if no separate mask input has been connected.
Update
3D Flex Generator now outputs a JXX_particle_flows.npy file that saves the trained mesh deformation (flow) of each particle.
Update
New Freeze latents during training parameter in the 3D Flex Training job allows 3D Flex to better preserve the 3DVA motions when trained from 3DVA latents.
Update
Inspect Particle Picks now shows the picking statistics histogram in the job’s event log when the job completes, including a border showing the selected thresholds.
Update
The exposure viewer in interactive jobs and CryoSPARC Live always pans when holding down the middle-mouse button, enabling navigation around the exposure while performing an action such as manual picking or measurement.
Update
Refinements with global pose search now throw warnings if symmetry expanded particles are connected.
Correctly set CPU resource requirements for Topaz jobs based on number of CPUs and number of workers specified in parameters.
Update
Added a Downsampling mode parameter to Topaz jobs with options of “manual” or “auto”. If set to manual, the user enters Downsampling factor as before. The default is now auto, where the user enters Estimated particle diameter (A) and the downsampling factor will be automatically calculated based on the type of model used.
Update
Exposure Group Utilities in “cluster&split” mode now respects the “Starting Exposure Group ID” parameter.
Update
New cluster_vars argument for Job.queue method for queueing to clusters.
Update
Allow copy or symbolic link operations in project directories for files outside the project.
Update
Replaced the “Output F-crop factor” parameter in Reference Based Motion Correction with the ability to directly specify the desired output box size.
Fixed
Previously, refinement jobs recorded internal runtime summary statistics in the database at every iteration. Stats from intermediate iterations are unnecessary and are no longer generated, and existing intermediate stats in the database will be removed in a migration upon updating to v4.5. We expect this change to noticeably improve performance in the web app when viewing or interacting with jobs due to the removed data’s size.
Fixed
Repetitive warning messages from libtiff during normal reading of .eer movies have been suppressed, reducing the size of the job log.
Fixed
Fixed a bug where ab-initio volumes could not be deselected after being selected in CryoSPARC Live sessions.
Fixed
Fixed an issue where it was possible to detach projects that still had jobs queued in the scheduler.
Fixed
Fixed a bug that could cause motion correction and CTF estimation jobs to freeze forever if a child process terminates abnormally.
Fixed
Prevent applying previously-downloaded patches that don’t apply to the current version of CryoSPARC.
Fixed
Fixed issue where thumbnails for Import Movies might not zoom in properly
Fixed
Fixed a bug introduced in v4.4 that would cause Patch Motion Correction jobs to fail if cleared and re-run, due to the missing input for empirical dose weights.
Fixed
Patch Motion Correction now correctly updates the frame count in exposure metadata, in case it was recorded incorrectly by the import job (e.g. if “skip header check” was used). This will help Reference Based Motion Correction to pre-screen its input movies for those with mismatched frame counts, potentially averting a crash.
Fixed
Allow using Job Cart with Reference Based Motion Correction.
Fixed
Fixed issue which could cause Local Motion Correction to crash at the end of the job while computing summary statistics, on an otherwise successful run.
Fixed
Fixed issue where particle diameter was not being calculated correctly in Manually Curate Exposures jobs with connected particle inputs.
Fixed
Resolved a bug in Inspect Particle Picks where particle picks from denoised micrographs would report NCC scores outside of [0,1], causing them to be inaccessible to filter.
Fixed
Fixed a bug in 2D Classification where if hard classification is enabled, particles would be assigned posterior scores of 0, causing particle counts to be incorrect in Select 2D and subsequent jobs.
Fixed
Added override parameter for micrograph pixel size in 2D Classification to fix an issue where duplicate particle removal used the wrong micrograph pixel size on particle datasets from before version 4.5.
Fixed
Fixed 2D Classification plots to be consistent with Select 2D Classes job.
Fixed
Fixed an issue in Helical Refinement where input scales were previously reset to 0 if greyscale estimation is disabled.
Fixed
Fixed plotting bug in 3D Classification which resulted in the focus mask being shifted from its true location by one pixel along each axis.
Fixed
Global CTF Refinement now throws an explicit error when the maximum and minimum fit resolutions are incompatible.
Fixed
If inconsistent aberration values within a single exposure group are found, refinements with global CTF refinement enabled now fail at the start of the job, rather than midway through.
Fixed
Fixed bug that would prevent 3D Variability Display from skipping reconstruction in cluster mode.
Fixed
Removed the download PDF option for cluster plots in 3D Variability; the PDFs take a long time to open due to the large number of points being plotted.
Fixed
Fixed an issue in which .seg-format 3DFlex mesh segmentation files with certain topologies would fail.
Fixed
Fixed an issue in which .mrc-format 3DFlex mesh segmentation files skipped the first segment.
Fixed
Particle Sets Tool now correctly uses both a particle’s file path and its index within the file to identify matches when intersecting by path.
Fixed
Fixed an issue where Symmetry Expansion and Volume Alignment Tools would ignore any additional particle blob slots (including those produced by Flex Data Prep, such as blob_train).
Fixed
In Simulate Data, fix to computation of the additive noise variance. The added noise no longer erroneously depends on the chosen CTF parameters.
Fixed
SSD cache system no longer fails when some input particle files are symbolic links that resolve to the same file.
Fixed
Prevent invalid multi-instance access when the environment of one instance is active while running cryosparcm commands for another.
Fixed
Fixed a bug which caused the Gctf wrapper job to fail if LD_LIBRARY_PATH was unset.
Fixed
Normalize project directory names when creating or attaching to avoid duplicate entries in the database.
Fixed
Correct Benchmark job directory access when there is a trailing slash in the specified benchmark data directory.
Fixed
Restrict accidental cloning of certain Live-managed jobs.
Fixed
Fixed an issue where event log PDF downloads for certain jobs (eg. Manually Curate Exposures) would fail due to control characters being interpreted incorrectly.
Fixed
Fixed an issue where the interactive scatter plot for 3D Variability Display would not update its layout when the window resized.
Fixed
Job PDF report now states the version of CryoSPARC the job was run with, not the version of CryoSPARC running when downloading the report.
Fixed
Resolved an issue where job output image previews did not display on the ‘Outputs’ tab when previewing a job.
cryosparcm commands configuredb and patch no longer show a segmentation fault after successful completion.
Fixed
Log messages for project and job size updates no longer contain inconsistent counts.
Fixed
Fixed issue introduced in v4.4.1 where benchmark job would not complete GPU Engine benchmarks.
Fixed
Reference-based motion correction no longer fails if GPUs are configured in EXCLUSIVE_PROCESS mode.
CryoSPARC v4.4.1
December 5, 2023
This release contains various stability improvements:
Update
The volume viewer now supports many more types of volume outputs: map, map_sharp, map_aligned, map_half_A, map_half_B, map_sym, map_sym_sharp, map_filtered, map_3DFSC
Update
Attempting to unarchive a project will now fail if the directory specified does not contain a the same project UID that is being unarchived. The target project directory’s project.json file is checked to see if it matches the UID of the project selected in the UI.
Update
Exposure Group Utilities now creates the exposure and particle output groups at build time rather than run time, enabling the job to be used in Workflows.
Update
Align 3D’s volume output group now outputs a map output, identical to the map_aligned output, for enabling drag-and-drop connection to refinement job input groups.
Update
The Benchmark job now includes a new class3D-small benchmark to test minimum supported hardware specs (32GB DRAM, 11GB VRAM). GPU benchmarks also now allow any specific benchmark to fail without halting the overall benchmark job.
Fixed
Fixed an issue where the default blueprint ‘2D Classification (Small Particle)’ would fail due to a missing custom parameter.
Fixed
Fixed an issue which caused the 3DVA Display job to sometimes duplicate upstream 3DVA component slots at build time.
Fixed
Fixed an issue causing Remove Duplicate Particles and 2D Classification to occasionally fail with an AssertionError.
Fixed
Fixed an issue where the ability to toggle display of denoised micrograph within Exposure Curation was not showing.
Fixed
Fixed an issue which caused I/O error messages to be displayed without the specific reason for the failure.
Fixed
Fixed an issue which could cause reference based motion correction to crash when a defect file was in use.
Fixed
Fixed an issue where reference based motion correction could crash with a message about the variable ‘uid’ being referenced before assignment.
Fixed
Increased memory limit per movie for reference based motion correction from 100 GB to the machine’s physical RAM.
Fixed
Fixed an issue where thumbnails were not being updated when manually rejecting exposures in the exposure curation cards view with a filter set.
Fixed
Project level parameter defaults are longer reset when creating a job from a blueprint (all parameters are still reset when applying a blueprint to an existing job).
Fixed
Template picking in Live no longer fails when only one template is selected.
Fixed
External dependencies that were removed in older versions of CryoSPARC are no longer installed.
Fixed
Fixed an issue where all required directories were not being checked for existence when restoring from backup.
Fixed
MONGO_EXTRA_FLAGS is no longer included in the default configuration file after installation.
Improved error message when driver version is incorrect
Fixed
CryoSPARC Live session worker jobs no longer fail during particle picking on sessions created prior to v4.4
Fixed
CryoSPARC Live sessions created prior to v4.4 no longer fail during pausing and exporting particles
Fixed
Class Probability Filter job no longer fails when particle blobs are not available
Fixed
Benchmark job correctly runs with more than one GPU
Fixed
Benchmark job no longer fails if benchmark data is outside the project directory
Fixed
Small Particle parameter definition in default blueprint is now correct
CryoSPARC v4.4
November 8, 2023
We’re excited to announce another release of CryoSPARC! Version 4.4 includes Workflows and Blueprints, Reference Based Motion Correction, new tools for orientation diagnostics, many feature updates, performance optimizations and various stability enhancements.
New
CryoSPARC now bundles CUDA Toolkit 11.8; a separate CUDA Toolkit installation no is longer required by the CryoSPARC worker application. BREAKING: CryoSPARC v4.4 requires NVIDIA driver version 520.61.05 or newer.
Workflows: Workflows make it easy to replicate processing that you’ve done before. Select a chain of jobs to create a templated processing Workflow, and re-create it in any project or workspace with a single click. Annotate the Workflow with a description and tags, set up default or locked parameter choices, and easily share Workflows from one CryoSPARC instance to another.
Workflows are flexible. You can create a linear Workflow that for example, starts with an import job and proceeds all the way to a refinement, to make it easier to process a new dataset using an existing set of steps. Alternatively, you can select a portion of a processing workflow that you would like to re-use for exploratory processing (e.g., for sorting particles or separating heterogeneity).
Use Workflows to save time, to maintain consistency and repeatability, and to capture important target-specific or facility-specific processing details and patterns so that new users can get a head start.
Blueprints: Blueprints allow you to define custom parameters and annotations as a template for any job type. Apply the Blueprint to an existing job or create a new job from a Blueprint directly from the job builder, in any workspace or project.
Save time and avoid mistakes in entering parameter values by creating Blueprints for e.g., import jobs for specific microscope setups, classification and refinement jobs for specific protein types or sizes, settings for specific computational configurations, etc.
Create Blueprints to avoid searching across projects for the parameters of previous runs of a particular job type. Easily export, share, and import Blueprints from one CryoSPARC instance to another.
Reference Based Motion Correction (BETA) in CryoSPARC improves final resolution and map quality by re-estimating per-particle motion trajectories using a reference volume, similar to Bayesian Polishing (Zivanov et al, 2019).
Before running Reference Based Motion Correction, you must run Patch Motion on your input movies and take the extracted particles through a 3D refinement.
The reference density and particle poses from 3D refinement are used to go back to the input movies and re-extract particles with optimized motion trajectories. Reference Based Motion Correction is multi-GPU accelerated and optimized, and includes a new method for hyperparameter optimization.
The job also computes optimal empirical dose weights by comparing aligned particles against the reference density, and uses these to weight contribution from early and late frames of the input movies. This dose weighting is often responsible for a large part of the improvement from Reference Based Motion Correction. In CryoSPARC v4.4, Patch Motion correction can now accept empirical dose weights as input, e.g. from previous processing of a similar dataset, so that empirical dose weights can be applied right from the start of the processing pipeline.
Orientation Diagnostics, a job that rapidly computes a set of easily interpretable diagnostics to identify orientation bias and anisotropic signal content in 3D refinement results. Quickly identify potential data collection, sample preparation, or particle picking/curation issues with two complementary statistics: Conical FSC Area Ratio (cFAR), sensitive to anisotropy in signal content and alignments and Sampling Compensation Factor(SCF*), sensitive to anisotropy in alignments (Baldwin and Lyumkis, 2020).
Several new diagnostic plots allow for in-depth analysis of directional variation in FSCs and particle scale factors. The generated diagnostics include the values and plots previously produced by the 3DFSC wrapper job, which is now a legacy job.
Non-Uniform Refinement is up to 2x faster: the core computation for non-uniform cross-validation regularization is optimized on GPU to be 7x-10x faster. This translates to approximately 2x faster overall run time for Non-Uniform refinement jobs, and Helical and Local Refinement jobs with non-uniform regularization enabled.
New
3D Flex Reconstruction job now supports CTF aberrations during high-resolution reconstruction. Also, there is now an option to read particles from SSD cache or from project directories during reconstruction, rather than loading all particles into RAM. This option substantially decreases the CPU RAM requirements for reconstruction.
In addition, 3D Flex dependencies are now bundled with CryoSPARC and no longer require a separate online installation step. 3D Flex now supports the latest NVIDIA GPU compute capabilities 8.9 and 9.0.
New
Symmetry Relaxation: There is now an option to relax the pose search over symmetry-related views in Homogeneous and Non-Uniform Refinement.
A “break symmetry” option is available in Homogeneous Reconstruction that randomizes particle poses over the chosen symmetry group.
New
float16 Support: Particles and motion-corrected micrographs can now be output in 16-bit floating point format, via a new parameter in motion correction and extraction jobs. This can cut disk usage in half for large projects. The option to output float16 can also be set as a project-level default parameter.
New
AFIS Beam Shift Import: In Import Movies and Import Micrographs, beam shift values for data collected in EPU via the Aberration-Free Image Shift (AFIS) method can now be directly imported, and exposure groups created based on clusters of beam shift values.
For existing exposure datasets in CryoSPARC, add beam shift information using the new Import Beam Shift job. In the Exposure Group Utilities job, cluster exposures into exposure groups based on imported beam shift values automatically.
New
Regroup 3D job quickly creates a set of ‘superclasses’ from a larger set of upstream 3D Classification or Heterogeneous Refinement classes. It uses a lightweight spectral clustering algorithm that can aid in identifying dominant conformational variations.
New
Heterogeneous Reconstruction Only job reconstructs multiple classes from upstream 3D Classification, Heterogeneous Refinement, or Regroup 3D outputs. It uses weighted back-projection via upstream class posteriors, with an option to apply hard classification. It includes similar reconstruction options as the Homogeneous Reconstruct Only job (including symmetry, box size, etc.).
Experimental SSD cache system with significantly improved reliability and support for network drives. To enable, add export CRYOSPARC_IMPROVED_SSD_CACHE=true to cryosparc_worker/config.sh. See the guide for more details.
New
Improved behaviour of the Data Cleanup Tool to streamline compaction of projects and workspaces:
All options are unchecked by default.
When running the Cleanup Data tool on a single workspace within a project, jobs that are linked to other workspaces will no longer be deleted by the tool; instead those jobs will be unlinked from the workspace being cleaned and will remain present in the other workspaces.
CryoSPARC Live jobs, which are excluded from the cleanup process, are marked as “Skipped” and their size is now displayed.
The total project directory size is now displayed in the cleanup tool along with the total size of data that the tool can clear, which may be different due to imported and manually copied files that may be present in the project directory.
New
2D Classification implementation that reduces memory usage and provides speedups with larger datasets. The original implementation is available as a legacy job type.
New
Reconstruct 2D Classes job that reconstructs 2D class averages from previously classified particles.
New
Ability to set custom parameters within the Live session view when configuring custom Streaming 2D Classification, Ab-inito and Streaming Refinement jobs.
New
Job cards now indicate warnings and errors, along with a text preview. If a job contains warnings or encountered an error, an icon will display in the cards and tree view. Hovering over the icon will display more information about the warning/error. A subset of the error will be visible directly on the job card view if applicable.
New
Ability to copy parameter code name via the ‘Inputs and Parameters’ tab when previewing a job. This name can be used in cryosparc-tools scripts.
New
If a job exists in more than one workspace, an icon and informational tooltip will be show on the cards view. In addition, jobs that display in the tree view but are not explicitly linked to the selected workspace are marked with an icon and informational tooltip for additional context.
New
Ability to star a Live session from within the session view through the primary dropdown menu.
New
Import Particles contains a new set of parameters to enable build-time output group generation and validation. This allows Import Particles to be used within Workflows for automated processing.
New
Simulate Data job that is now GPU accelerated. The previous implementation is available as a legacy job.
Update
The Performance Benchmarks system now captures benchmarks from the core algorithms used in 2D Classification and 3D Classification jobs. In addition, reference GPU Benchmarks have been added for the RTX 4090, A100 PCIE-40GB, RTX 3090, and A4000 GPUs, as well as for AWS EC2 g5 instances with FSx for Lustre and local NVME SSDs.
Update
The Validation (FSC) job now outputs an additional FSC figure with square plot markers along with a line, to more clearly see each resolution shell.
Update
3D Variability Display now includes an option to create intermediate reconstructions with equal numbers of particles in each frame. The job now also allows up to 100 input components instead of 20.
Update
2D Classification now has an option to set a minimum alignment resolution that provides the ability to high-pass filter references during classification and can be useful for smaller particles in some cases. The job also has an option to display a scale bar on the 2D class averages figure.
Update
Heterogeneous Refinement now correctly accounts for the Backprojection resolution factor parameter; the job card image display is improved when there are classes of very low resolution; the Initial random assignment iterations parameter now applies only if non-distinct initial volumes are provided; and the job now outputs a volume series containing all the classes that were produced.
Update
3D Classification now filters the consensus volume according to its FSC and outputs the result for downstream use.
Update
Sharpening Tools job now estimates a B-factor using a Guinier plot if a sharpening B-factor is not provided, with options to include mask and FSC weights prior to estimation.
Update
Downsample Particles now allows ‘uncropping’: when the desired box size is larger than the input particle box size, the particles are padded in real space and the padded region is filled with the mean value of the particle.
Update
Downsample particles now has an option to re-center particles according to their shifts.
Update
The Rebalance 2D Classes job now displays a plot of the pairwise affinity matrix between 2D classes, and also updates particle’s 2D alignments to be in-register within each superclass. Also, the job now always outputs combined datasets for particles and templates, even if “Split outputs” was activated.
Update
The legacy web application (deprecated since CryoSPARC v3.0 in December 2021) that could be spawned via cryosparcm start app_legacy and operated at BASE_PORT + 7 is no longer bundled with CryoSPARC.
Update
The CryoSPARC database now has journaling turned on by default, increasing durability in the event of a failure.
Update
When a job is in building status, any custom parameters set are visible on the job card in the tree view.
Update
CryoSPARC Live sessions now use 10% less storage space per micrograph in the database by reducing the number of plots created during preprocessing. To turn on all plotting, enable the “Enable Plotting in RTP Worker” session parameter.
Update
CryoSPARC Live Session UIDs now increment monotonically, independent from Workspace UID.
Update
Micrograph images with pick locations overlayed can now be downloaded from the micrograph viewer in CryoSPARC Live.
Update
The job details sidebar now shows the last time a job was exported and the path to which it was exported, to make it easy to find exported data.
Update
The recent jobs sidebar can now be toggled to show either the user’s jobs only or all instance jobs, as well as showing completed, failed, and killed jobs.
Update
In Blob Picker and Filament Tracer, the templates are now available as outputs, along with the particles.
Update
The Template Picker now has a parameter to disable automatic re-centering of the template image, which has been known to cause issues in some cases.
Update
Exposure Sets Tools and Particle Sets Tools now generate their output groups before they are run, allowing these jobs to be part of pre-defined Workflows.
Update
Volume Tools now can invert masks without applying a threshold.
Update
Volume Tools now specifies that the units for the dilation radius and the padding width are in pixels, and the default soft mask padding is set to 12 pixels.
Update
Volume Alignment Tools now has an option to reassign particles’ UIDs, for assisting the refinement of particles with custom symmetry operators.
Update
Homogeneous Reconstruction Only now includes the posterior precision as an output, as part of the the volume group.
Update
DeepEMhancer job now includes an option to enable dynamic allocation of GPU memory, obviating the need to set an environment variable.
Update
DeepEMhancer model path can now be specified as a project-level default parameter.
Update
The Deep Picker job now runs with Tensorflow v2.8.
Update
Many jobs now use a common progress bar in the event log.
Update
When queueing a job via the sidebar, the last used lane will now automatically be selected, if applicable.
Update
When importing a job, the path will now default to the project directory of the selected project.
Update
Initial loading of the job tree view is improved.
Update
Summary data shown on job cards is improved across CryoSPARC.
Update
Additional quick actions are now available for swift creation of commonly run jobs.
Update
Additional project-level filters can now be used to show archived or detached projects.
Fixed
PatchCTF can now correctly handle high-magnification micrographs without throwing AssertionError: Get rid of patches that are too close to edges.
Fixed
2D classification no longer displays classes upside-down relative to the image plotting convention used in the rest of CryoSPARC.
Fixed
Resolved an issue where a list of all jobs are not available via the header quick switcher when viewing all jobs within a single project.
Fixed
Resolved a bug where it is possible to create Session configuration profiles with the same name.
Fixed
Resolved a bug where when deleting a workspace the interface does not redirect to the project level.
Fixed
Resolved an issue where the number of GPUs a job is utilizing is formatted incorrectly on the job card.
Fixed
When queueing a job via the sidebar, the cluster submission script variable configuration panel is now hidden if no custom variables are defined in the script.
Fixed
Resolved an issue on Firefox where the exposure feed in a Live session view can not be controlled via the mouse scrollbar.
Fixed
Resolved an issue on the job cards display where the border outlining parents, children and cloned jobs is too thin.
Fixed
Fixed a issue where queued jobs can be accidentally deleted via the multi-action sidebar panel.
Fixed
Resolved an issue where the ‘Build’ button in the job preview dialog does not function correctly in certain cases.
Fixed
Resolved an issue in the jobs tree view where the final or final ancestor status of a job is not visible until selected.
Fixed
Fixed an issue in 2D Classification where the greyscale background of class averages is incorrectly computed from the values in the corners of the average.
Fixed
Fixed pymongo.errors.WriteError: Size must be between 0 and 16793600(16MB) when processing a live session with a large number (>10,000) of movies.
Fixed
It is now possible to specify —cpus in cryosparcw connect without also having to specify —update.
Fixed
Exposures processed by CryoSPARC Live now have their average intensity computed.
Fixed
Fixed a bug where Particle Subtraction does not process all of the particles if the half sets are not of equal sizes.
Fixed
Resolved a bug where 3D Variability’s tile image displays components in the wrong order after job completion.
Fixed
2D Classification no longer erroneously removes duplicates by default for particle datasets with filament present.
Fixed
Fixed bug where Helical Refinement in building mode doesn’t allow modifications to the “enable FSC noise-substitution” parameter
Fixed
Fixed bug where flipping the hand of a volume (in volume tools, homogeneous reconstruction, Align 3D, and 3D Variability Display jobs) causes symmetric volumes to no longer be aligned to the conventional symmetry axes.
Fixed
Fixed bug causing 2D Classification to calculate the wrong inter-particle distance when removing duplicates when downsampled particles are provided as input.
Fixed
Fixed bug causing an UnboundLocalError in Patch CTF Extraction.
Fixed
Disabled menu items no longer erroneously open sub menus when hovering.
Fixed
Fixed issue where the spacebar shortcut for opening job dialogs stops working due to conflicting event handlers.
Fixed
Fixed an issue where the scatter plot in Manually Curate Exposures does not updating the rejected status of micrographs correctly.
Fixed
Deleting a job while the queue slideover is open no longer causes it to lose context and go blank.
Fixed
Select 2D job now correctly reports error messages when it cannot connect to the backend.
Fixed
Installer no longer writes duplicate PATH entries for cryosparc_master/bin directory into .bashrc.
Fixed
CryoSPARC Live processing jobs no longer use more memory than needed when plotting.
Fixed
Errors that happen when CryoSPARC Live is watching for new files are now isolated so that errors in one session do not affect other running sessions.
Fixed
CryoSPARC Live can now run reconstruction processes (2D Classification, Refinement) on a lane with multiple one-GPU targets.
Fixed
The reliability of cryosparcm update is now improved in the case where the master is connected to itself as a worker.
Fixed
CTFFIND4 jobs no longer fail due to localization settings on some systems.
Fixed
Leading and trailing spaces are now automatically removed when setting parameters.
Fixed
Local Motion Correction no longer fails when no particles are extracted from one or more exposures.
Fixed
3D Classification no longer fails during noise estimation if the largest class contains fewer than 100 particles. A warning is also now printed if any per-particle scale factors become NaN.
Fixed
The new parameter Disable auto batchsize now allows disabling the automatic GPU processing batch size calculation in non-uniform and homogeneous refinement that can sometimes cause the jobs to run out of GPU RAM. Extraneous batch size parameters from local refinement are removed.
Fixed
Fixed bug where background subtraction of density maps during ab-initio and heterogeneous refinement is incorrect.
Fixed
Fixed erroneous component ordering in 3D Variability Analysis Display when using 10+ components.
Fixed
Fixed confusing labelling of variability component plots produced at the end of 3D Variability Analysis jobs.
Fixed
MRC format volume outputs from CryoSPARC now include mx/my/mz values in the header which fixes display issues in some visualization programs.
Fixed
Resolved an issue in the job builder where numerical inputs display inaccurate tooltip information on hover.
Fixed
Moving jobs between workspaces now correctly updates workspace statistics.
Fixed
The workspace count statistic is now correctly incremented when a new workspace is created in a project.
Fixed
Fixed an issue where the lock file within a project locked to another instance is overwritten rather than blocking the attachment of the project in another instance.
Fixed
Homogeneous Reconstruct Only job now correctly allows for turning off the real-space window on particles.
CryoSPARC v4.3.1
August 31, 2023
This release contains various stability improvements:
Fixed
Fixed an issue where in some cases, memory usage of the CryoSPARC master processes could grow without bound after restarting the instance.
Fixed
Sizes of job directories on disk are no longer updated more often than necessary.
Fixed
CryoSPARC Live sessions are now automatically paused when the instance is restarted.
Fixed
Fixed an issue where a project could be detached while a CryoSPARC Live session was still running.
Fixed
CryoSPARC Live no longer monitors worker jobs when not necessary.
Prevent “syntax error” following successful CryoSPARC update
CryoSPARC v4.3
August 9, 2023
We’re excited to announce another release of CryoSPARC! Version 4.3 includes new tools for data management, performance benchmark utilities, improvements to interactive jobs, many feature updates, performance optimizations and various stability enhancements.
Mark certain jobs as final and automatically clear everything except for jobs needed to replicate those results
Compact CryoSPARC Live sessions, removing preprocessing data but storing particle locations and parameters needed to restore the session later
Restack particles to remove filtered out junk particles and then automatically clear deterministicpreprocessing jobs that can be re-run later if needed
Clear intermediate results of iterative jobs across a project or workspace
Clear killed and failed jobs that take up disk space but can’t be further used
Featuring a comprehensive Cleanup Data user interface, on-demand refreshing of project and job size statistics, data usage summaries in the project details sidebar, and new right-click menu options for selecting ancestors and descendants of jobs manually. See the Data Cleanup guide for more details.
A new system for measuring and comparing key aspects of CryoSPARC performance across CPU, File system, and GPU operations. The benchmarking jobs break down critical elements of CryoSPARC high performance code paths and measure detailed timings to shed light on which parts of a system may be bottlenecking performance.
A new performance benchmark user interface allowing for visual comparison of different benchmark runs as well as reference benchmarks published along with CryoSPARC. The interface can display detailed timings from benchmarking jobs or job runtimes from full runs of the Extensive Validation(previously called Extensive Workflow) test set.
New
Particle caching in CryoSPARC jobs now support multiple threads (default 2), increasing copy speed on certain filesystems. To modify, specify CRYOSPARC_CACHE_NUM_THREADS in cryosparc_worker/config.sh. Read the guide for more information.
New plots, displays, and interactive tools across all interactive jobs, including a new exposure image grid view in the Exposure Curation job for ease of assessing and manually curating exposures, and the ability to set threshold parameters when starting the job to bypass interaction and automatically filter data.
Several usability enhancements, including:
Extraction is now optional in the manual picker
Datasets without pick statistics can be used in Inspect Picks
Charts in Exposure Curation remember their expand/collapse state when reloaded
Exposure Curation now displays particle picks if available
Optimized micrograph image loading for faster response times
New
Extensive Validation (previously called Extensive Workflow) job extended to support multiple datasets and also has a new “Benchmark” mode where timing results are reported and browsable using the new Performance Benchmarking system.
New
The cryosparcm compact command can reduce the amount of disk space used by the MongoDB database. This, along with other steps, may be useful for reducing used storage space in instances with large database sizes. See the Guide: Reduce Database Size for more details.
Update
Interactive job improvements:
Flipped Select 2D resolution threshold selection option for clarity
Added total picks across all exposures as info badge in Manual Picker top bar
Added filter to Select 2D job to allow showing of only selected or unselected classes (sorting options can be applied to filtered classes)
Added info badge to Inspect Picks job to display the total number of particles within the applied thresholds
Manual Picker and Inspect Picks jobs now display all available data fields in the micrograph table
Added an exposure image preview when hovering exposure table rows in interactive jobs
Individual tab has been renamed to diagnostics in Exposure Curation job for clarity of function
Thresholds of 0 now filter micrographs correctly in Exposure Curation job
Support added for right clicking to remove picks in Manual Picker job
Update
Non-uniform Refinement implementation speedups up to 15% on certain GPUs and box sizes.
Update
Import Particles job is now 50-70% faster.
Update
Improved speed of the caching system by reducing the number of file lookups, and improved logging of the cache system while a job is running.
Update
CTFFIND4 is updated to v4.1.14.
Update
Import and utility jobs that previously ran only on the master node can now optionally be launched on any worker node or cluster lane. The CRYOSPARC_DISABLE_IMPORT_ON_MASTER environment variable is no longer used.
Update
Added star button to top navigation bar in browse system to allow starring a workspace from the jobs page.
Update
Quick access panel will now remember selections on a user basis rather than a browser tab basis.
Update
The load all jobs view option is now saved per-user rather than based on the browser.
Update
Event log view options have been moved into a new view options menu to minimize interface clutter.
Update
Sessions can now be marked as completed from the session browser view.
Update
Archived and detached projects no longer show within the session browser.
Update
An Additional spotlight action added to navigate to the job history page.
Update
The ‘view job’ button in the sidebar can be ctrl/cmd clicked to open in a new tab
Update
For job types that take random seed parameters, the random seed that was used is now saved in the input parameters of the job for easy reuse. The random seed will stay consistent if the job is cleared and re-run.
Update
2D Classification, 3D Classification, and 3D Variability Analysis jobs will now produce intermediate results at each iteration, but will delete intermediate results from previous iterations as each new iteration completes. This behaviour can be turned off with a project-level or job-level parameter that will keep all intermediate results.
Update
Jobs that fail to report heartbeat to the master node within the required timeout but continue running in the background will now be killed on their worker node, rather than being allowed to continue running after having been marked as failed due to missing heartbeat.
Update
Worker installation job launch test (cryosparcm test workers) timeout increased to 2 minutes from 10 seconds.
Update
The config.sh file with environment variables will now be included in the cryosparcm snaplogs bundle of logs.
Update
CryoSPARC running version is now available in the Job PDF download.
Update
MongoDB database journaling can be enabled by adding export CRYOSPARC_MONGO_EXTRA_FLAGS=” ” to cryosparc_master/config.sh.
Fixed
Topaz Extract job no longer extracts particles from denoised micrographs by default.
Fixed
3DFlex Generate job no longer ignores the Number of Frames parameter.
Fixed
Fixed unnecessary AssertionError in Flex Mesh Prep job when pixel size between input mask and volume match but have rounding error difference
Fixed
Heterogenous Refinement job no longer fails with AttributeError when “Refinement box size (Voxels)” parameter passed None as a value.
Fixed
Worker installation test (cryosparcm test workers) no longer fails when worker target does not have SSD configured.
Fixed
CryoSPARC Live Session compute configuration parameters no longer fails to be changed if a worker lane that was previously selected is removed from the CryoSPARC instance.
Fixed
Fixed select confirmation appearance in Edge browser on Windows.
Fixed
Fixed bug where Job PDF would show incorrect parameter values in some cases.
Fixed
Fixed issue where PDF event log download was crashing due to control characters being interpreted incorrectly during text cleaning.
Fixed
Corrected sorting order of the job event log PDF.
Fixed
Added optimistic job count update to stop the user interface loading state from glitching when deleting a workspace job when few exist.
Fixed
Resolved error where Import Result Group fails to import Topaz jobs correctly, causing downstream jobs to fail.
Fixed
Resolved error where Import 3D Volumes job fails with “Unable to connect to EMDB server” when trying to load volumes from EMBD.
Fixed
Allow retrieving job and workspace documents from the CLI without errors.
Fixed
Improved handling of missing job parent in tree view.
Fixed
Live no longer stops finding new exposures until the next restart.
Correct retrieval of legacy cluster configurations
Fixed
Live no longer stops finding new exposures
Fixed
SSD Cache system correctly retries up to 3 times on network timeout
CryoSPARC v4.2.1
March 15, 2023
We are excited to announce the stable release of CryoSPARC v4! This release contains various updates and performance improvements:
New
A project-level parameter now controls whether or not jobs output intermediate results, with default on. Turning this off will save disk space, without having to manually clear intermediate results after the jobs complete.
Update
In Particle Sets Tool and Exposure Sets Tool jobs, add note in event log indicating that result groups are taken from set A for the intersect dataset.
Update
In Average Power Spectra job, print particle pixel size and power spectra sample spacing to the event log.
Fixed
Manual Picker no longer fails when micrograph CTF input is not provided.
Fixed
Resolved an issue where loading external data for the homepage modules would fail if a non-standard response was received.
Fixed
Prevented a database error from occurring when no recent items existed for a user.
Fixed
Fixed issue where copying to the clipboard would not work in some cases depending on how CryoSPARC was hosted over an internal network.
Fixed
Priority field in the job queuing slide-over is now required and cannot be empty.
Fixed
Resolved an issue where a previously selected entity was still selected after creating a new project.
Fixed
Resolved an issue where clearing a job would not reset its queue priority.
Fixed
Resolved an issue where completed jobs that were deleted displayed within project, workspace and session cards.
Fixed
In 3D Classification, fixed KeyError with CTF fields when “Output results after every F-EM iteration” is enabled.
Fixed
Fixed an issue where re-ordering classes and outputting intermediate results in 3D Classification conflicted and caused the job to fail. These options are now mutually exclusive.
In Global CTF Refinement, throw a warning if the spherical aberration refines to a nonpositive value.
Update
In 3D Classification, add “Force re-do FSC split” parameter that forces re-splitting the particles into half-set splits from scratch. Disabling this allows for preserving the input splits, for particles from previous helical refinements, symmetry expansions, or local refinements.
Fixed
Fixed a problem in 2D classification where many classes would be blank or very faint if multiple GPUs were used.
Fixed
Fixed issues when running multiple jobs using the same SSD cache space.
Fixed
Fixed 3DFSC crashing due to list index out of range error.
Fixed
In 3D Classification, removed extraneous alignments3D entry in particles_all_classes dataset output which caused csparc2star.py to assign all particles to a single class.
CryoSPARC v4.2
February 27, 2023
We’re excited to announce another release of CryoSPARC packed with new features, performance optimizations and various stability enhancements:
New
CryoSPARC now fully supports CUDA 11.8, enabling support for NVIDIA Hopper and NVIDIA Ada Lovelace GPUs (NVIDIA H100, NVIDIA RTX 40 Series), including for 3DFlex.
New
The browse system has been updated with various incremental improvements including table header sorting, customizable view options, and upgraded CSV download functionality.
A view options menu can be accessed from the browse system control bar allowing card and table details to be shown or hidden to suit user preference.
The view options menu table section items can be dragged and dropped within the list to re-order the table view columns.
The CSV download feature now includes an additional dialog allowing the user to select what information to include or exclude and the ability to re-order the table columns.
New
Within the details sidebar of a job, you can click on a particular cloned, parent, or child job to view it in the current tab. Command/ctrl click it to open that job in a new tab.
New
Within the ‘Metadata’ tab in the job preview dialog, there is a button to download the JSON data of a job to your clipboard.
New
Added a quick action to perform CTF Estimation via CTTFIND4 from a Patch Motion Correction job.
Update
Introduced a workaround for a bug in CUDA 11.8 which formerly caused jobs relying on CUDA Fourier transforms (cuFFT) to run slower than they should (sometimes dramatically so).
Update
When importing movies and micrographs, the ‘Skip header check’ parameter is now enabled by default.
Update
Performance and stability improvements to all interactive job tables and the browse table in Live sessions.
Update
Improvements to the context menu for job cards within the current jobs dialog.
Update
Optimized current jobs tab within manage dialog and added a running timer for each job.
Fixed
Fixed bug in 3D Classification that caused csparc2star.py to produce erroneous star files if class reordering was turned on.
Fixed
If a job is running on a cluster, the available memory usage statistic is no longer displayed within the event log.
Fixed
If specified, the cropped box size is correctly displayed on the card of a Volume Tools job.
Fixed
Resolved cases where the outputs view mode toggle was triggered while attempting to perform a browser or operating system keyboard shortcut.
Fixed
Removed variables ram_gb, num_cpu, and num_gpu from appearing as custom variables in a cluster configuration.
The above variables are internal variables set by CryoSPARC and cannot be used/modified as custom variables
Updated guide with a list of internal cluster variables and instructions for modifying resources requested in a cluster submission script
Fixed
Fixed reading of .mrc.bz2 gain reference files in CryoSPARC Live.
Fixed
Fixed issue on some OSes where building pycuda would fail during installation.
Fixed
Fixed error message during job heartbeat failure to correctly specify heartbeat timeout.
Fixed
Fixed “wrong generation counter” error during some long-running jobs.
Fixed
Performance optimizations to the volume viewer in Live Sessions and the job preview dialog.
Fixed
Fixed 2D Classification to allow manual overrides to Force max over poses/shifts parameter if classes is set to 20 or fewer.
Fixed
Removed the parameter “Overall normalizer (override)” in particle extraction. The parameter had no effect.
Fixed
Resolved an issue where clearing intermediate results for projects triggered multiple notifications.
Fixed
Resolved an issue where viewing a job without selecting the card would select and pin it to the sidebar.
Fixed
Resolved an issue where the current jobs tab within the manage dialog would crash the application in some situations when interactive jobs were queued or active.
Fixed
Resolved a bug where it was not possible to define a threshold for the ‘Timestamp’ attribute of exposures in a Live Session.
Fixed
Resolved an issue that caused an ‘Invalid path’ error when building Topaz or DeepEMhancer jobs.
CryoSPARC v4.1.2
January 23, 2023
This release contains various features, updates and stability improvements:
New
Added a new Restack Particles job for consolidating particle files. More information available in the guide.
Added an option (default: on) to re-order classes by number of particles at the end of processing
Added maximum F-EM iterations which will terminate classification even if convergence criteria are not met
New
In addition to CPU memory used, available CPU memory can now be displayed in the job event log. It is also displayed in the job log PDF.
New
Added options to sort projects by their size.
New
Added a new project-level action to delete detached projects from the CryoSPARC database. More information is available in the updated data management guide.
New
Exposure preview context menus in CryoSPARC Live now feature a button to reject/unreject the exposure.
Update
Additional timestamp information added to job output log.
Update
The job preview dialog project and workspace breadcrumbs have been updated to native browser links to allow them to be opened in a new window or tab.
Update
Buttons with a corresponding keyboard shortcut will now show that shortcut in their tooltips.
Update
The tree view now supports using the shift key to toggle the outputs view on all job cards.
Update
Multi-select context menu improvements
The job card action menu trigger button will now always display that job’s specific action menu even with multiple jobs selected
Opening the action menu on a job card not included in a multi-selection will show that job’s specific action menu instead of the multi-select action menu
The multi-select action menu can now be opened by right clicking the workspace background and not just jobs within the selection
Fixed
In 3D classification, ensure solvent mask is generated from consensus volume if (only) a focus mask is supplied.
Fixed
Fix a memory leak in 3D Classification related to plotting. With the fix applied, peak memory usage should be significantly reduced, with jobs that have a large number (50+) of classes seeing the most benefit (up to a 75% reduction).
Fixed
Fix incorrect batch size during class ESS tuning in 3D Classification that required excessive amounts of GPU memory.
Fixed
Errors reading TIFF files during import movies could cause abnormal termination.
Fixed
Event log would flash and jump in a perpetual loading state in some cases.
Fixed
Job description would automatically close while being edited in some cases.
Fixed
Clearing intermediate results on jobs with empty output result groups would cause an error.
Fixed
Jobs exported from this version may now be correctly imported in older versions of CryoSPARC.
Fixed
Added a guard against creating redundant outputs in Extract From Micrographs (CPU).
Improved database startup reliability following updates.
Fixed
Prevent wrong generation dataset error for some jobs that process a large amount of micrographs or particles.
Fixed
Prevent Extract From Micrographs (GPU) job from using too much GPU memory, causing exposures to fail with cufftAllocFailed or cufftInternalError.
CryoSPARC v4.1.1
December 20, 2022
This release contains various stability improvements:
Fixed
Fixed a bug affecting some OSes in the dependency installation for 3DFlex jobs (cryosparcw install-3dflex) that caused a gcc compilation error during the PyCUDA build.
Fixed
Fixed a bug that caused 3DVA display to fail when using a subset of variability components.
Fixed
Fixed a bug causing restarted jobs with GPU requirements to override the scheduler and jump to the front of the queue on the specified lane.
CryoSPARC v4.1
December 12, 2022
We’re excited to announce another release of CryoSPARC packed with new features, performance optimizations and various stability enhancements:
New
3D Flexible Refinement (3DFlex) (Beta) is a motion-based deep generative model for continuous heterogeneity. It can model non-rigid motion and flexibility of a protein molecule across its conformational landscape, and can use the motion model to combine signal from particle images in different conformations to improve refinement resolution in flexible regions.
3DFlex jointly learns a single canonical 3D density map of the particle, latent coordinates for each particle image, and a flow generator network. Given a latent position, the flow generator outputs a 3D deformation field mapping the canonical density to a particular conformation
3DFlex requires tuning of some parameters and optionally can take custom inputs defining geometry and topology of the target molecule
Multi-select actions allow for cloning multiple jobs at once, cloning a chain of jobs between two selections, queuing, clearing or deleting multiple jobs at once, and more. Learn more about multi actions here.
New
Restart job action allows a job to be restarted from a killable or clearable state. This new action kills, clears, and queues the job to the same lane with the same parameters with which it was initially run.
New
Ability to configure custom variables for use in cluster submission scripts directly through the Cluster Configuration Tab in the Admin Panel. Custom variables can be configured at the instance level, target level and job level. See Guide: Configuring Custom Variables for Cluster Job Submission Scripts for more details.
New
Ability for admins to restrict users to view and run jobs on specific lanes using the Lane Restrictions Tab on the Admin Panel. See Guide: Lane Assignments and Restrictions for more details. By default, when a CryoSPARC user is created, they are assigned all existing lanes.
Per-particle scale optimization is now built-in to the job
New ‘class flow matrix’ diagram helps to visualize class changes across F-EM iterations
New ‘Δ’ text column lists class size differences across F-EM iterations
Updated aesthetics for class flow diagram to improve clarity of edges
Unique volumes are no longer required for initialization
Update
Job: Remove Duplicate Particles now allows for duplicate rejection based on 2D and 3D alignment error, in addition to NCC score.
Update
Job: Symmetry Expansion now includes a parameter that enables splitting the output particle stack by the applied symmetry operator.
Update
Job: Volume Alignment Tools now allows for application of arbitrary rotations to input volumes, particles, and masks.
Update
Job: Select 2D Classes now exports an .mrc stack containing only selected classes, and an .mrc stack containing only rejected classes.
Update
Job: 2D Classification now by default removes duplicate particles after classification based on a minimum separation distance between pick locations.
Update
2D Classification jobs that perform binary class re-centering now show the binary masks used, generated from the specified threshold.
Update
New project-level option to set the default path to a DeepEMhancer executable. If set, all new DeepEMhancer jobs will automatically be populated with the project-level path.
Update
Job: Import Particle Stack now copies rlnAutopickFigureOfMerit into CryoSPARC’s pick_stats/ncc_score upon import, if available.
Update
Job: Global CTF Refinement now allows for initializing the spherical aberration to a custom value, prior to fitting.
Update
Ability to prevent CryoSPARC from requesting external resources external HTTPS resources used to display information modules on the homepage by setting CRYOSPARC_DISABLE_EXTERNAL_REQUESTS environment variable to true. Details.
Update
The CryoSPARC updater (cryosparcm update) has several enhancements, including:
Ability to only download the master and worker update packages without doing a full update. Read more here
Ability to skip downloading the master and worker update packages, and update using the previously-downloaded packages. Read more here
The CryoSPARC instance is only shut down after confirming the master and worker update packages are downloaded and valid, reducing downtime during an update
The output log of the update process is now written to cryosparc_master/run/update.log, and can be read any time using cryosparcm log update.
Update
Ability for instances with the CRYOSPARC_DISABLE_IMPORT_ON_MASTER environment variable to queue jobs to the master node.
Project directories can now be renamed by using the edit button beside the “directory” field in the details section of the project sidebar.
Update
Job: Reassign Particles to Micrographs will now copy the exposure group data from the micrographs to the particles if the particles.ctf slot and the exposure.ctf or exposure.mscope_params slots are connected.
Update
Job: Import Movies info tag now displays the size of each exposure and number of frames.
Update
When viewing jobs in a flattened view (e.g., all jobs in a project) the clone job action sub-menu now provides all workspaces as options to clone into.
Update
Restrict some cryosparcm commands to owner UNIX account, including createuser, updateuser, resetpassword and patch.
Update
Close open file handles during Import jobs.
Update
User IDs are now visible in the output of cryosparcm listusers
Update
job_type variable is now added to available cluster submission script variables provided by CryoSPARC. This value can be used for tracking which CryoSPARC job types are running on a cluster lane or for determining resource requirements for a cluster script.
Update
qstat_cmd_tpl and qdel_cmd_tpl will have access to the same variables that are injected into send_cmd_tpl, including any user-configured custom variables.
Update
Additional logging for master cache functions.
Update
Interactive jobs now print out more informative errors in a scrollable, clearable log at the top of the interactive panel in the job dialog.
Fixed
Resolved an issue where when viewing all parameters of a job within the comparison dialog incorrect custom parameter values were shown.
Fixed
Jobs created via quick actions are no longer created in a different workspace or session than the one currently selected.
Fixed
NCC score slider in CryoSPARC Live no longer loses all picks when set to 0.
Fixed bug in Job: 3D Variability Analysis that prevented job from completing when number of components is set to 1.
Fixed
The job dialog outputs tab now correctly displays the name of low-level results.
Fixed
Fixed a bug that prevented correct download links for each output group of a job from being displayed within the sidebar details panel when a job is selected.
Fixed
Fixed a bug that prevented multiple entities from being selected via the checkbox within the table view.
Fixed
Scroll bars are no longer shown overtop of menus and dialogs in Safari.
Fixed
Hostname no longer appears twice on instance management cards.
Fixed
The sidebar action panel now collapses automatically after an option has been selected.
Fixed
Fixed issue where a “Not Found” message would flash in the sidebar on load.
Fixed
Fixed bug that resulted in incorrect cropping of some PNG figures in the event log.
Fixed
Fixed bug in Job: Simulate Data that caused incorrect particles’ CTF parameters when “apply CTF” was disabled.
Fixed bug in CryoSPARC Live that caused failed exposures to be double-counted as queued exposures.
Fixed
Fixed issue where retrieving CryoSPARC Live sessions via the CLI would fail.
Fixed
Fixed issue where slow database command response times would cause problems when configuring the database (e.g., updating CryoSPARC).
Fixed
Extensive Workflow now runs on a worker with 32GB of RAM.
Fixed
Installation and update are now cancelled if Python dependency installation fails.
Fixed
Prevent job heartbeat failures following multiple disconnects and reconnects.
Fixed
Fixed bug that raised a JSONDecodeError when calling cryosparcm eventlog on Job: 3D Variability Display.
Fixed
Fixed bug that caused command_core, command_rtp and command_vis logs to not be properly rotated.
Fixed
Event logs for motion correction and extraction jobs now count movies/micrographs starting from one instead of starting from zero (e.g. “completed 20/20” instead of “completed 19/20”).
Fixed
Fixed bug whereby ice thickness reported by Patch CTF (including in CryoSPARC Live) can be NaN.
Fixed
Prevent file reading operations from failing on systems that incorrectly seek
Fixed
Fixed bug that caused extracted particle count not to be displayed on the details panel when viewing a Live session
CryoSPARC v4.0.3
November 3, 2022
This release contains various stability improvements:
Fixed
Prevent interface from becoming unresponsive when viewing the homepage in cases where the web application cannot load external resources.
Fixed
Fixed issue where changelog grid styling was causing items to overlap on Safari.
Fixed
Prevented an issue where available CryoSPARC updates would not display on the homepage and navigation bar.
CryoSPARC v4.0.2
October 24, 2022
This release contains performance optimizations, various stability enhancements and all changes from Patch 221017:
New
When viewing jobs within a workspace there is an option to load all jobs at once rather than incrementally load as you scroll down the page.
New
New project-level option to set the default path to a Topaz executable. If set, all new Topaz jobs will automatically be populated with the specified project-level path.
New
Ability to use the spotlight to search for applicable jobs based on what project or workspace has been selected.
Update
In Job: Exposure Group Utilities, when specifying a regular expression to split exposures or particles into exposure groups, all unmatched items will now be grouped together into a single, separate exposure group.
Update
Added —offline and —skip_workers options to cryosparcm errorreport
—offline skips retrieval of database statistics and worker instance information (intended for use when the CryoSPARC instance is not running)
—skip_workers skips retrieval of worker instance information (intended for use when there are issues connecting to worker nodes)
Update
The first applicable lane is now pre-selected when queueing a job from the sidebar.
Update
systemd setup is now simplified, and correctly enables CryoSPARC environment during systemd process startup. Read the guide here.
Fixed
Resolved inconsistency between displayed values for accepted and rejected exposures (and threshold included and excluded) in the Curate Exposures job with specific threshold ranges set now match job outputs in all cases.
Fixed
Inspect Picks in v4+ now rejects outlier particles based on absolute value of power score rather than percentile (matching the behaviour of v3.3).
Fixed
Local Motion correction no longer fails when some movies have zero particles or when some movies are corrupt.
Fixed
Multiple targets within a single lane will now correctly display in the instance section of the manage dialog.
Fixed
Volume viewer will now properly resize to the available space when adjusting sidebars and resizing the window.
Fixed
Clicking checkboxes on the job table view will now correctly additively select them.
Fixed
Event log downloads will no longer break in cases where ASCII characters are not included in the pdf font set.
Fixed
Users can now log in with emails where the domain does not include a .
Fixed
The page will no longer automatically refresh when downloading files from the output actions menus in Firefox.
Fixed
When viewing a volume in Firefox, the selecting the up/down arrows within the threshold control number input field will retain focus in the same way other browsers do.
Fixed
Prevent Live session view from crashing if the session was started after being marked as completed.
Fixed
When making picks on the micrograph in the Manual Picker job, the picks column will correctly update in the corresponding table row.
Fixed
When attaching a project, workspaces no longer have an incorrect “Created By” field, but show the user that attached the project.
Fixed
Project title is now a required field when creating projects from the command line.
Fixed
Clear Intermediate Results no longer deletes non-final results from some refinement jobs when those results were used downstream.
Fixed
Improved error messages and error checking in Topaz Train, Extract and Cross Validation jobs.
Fixed
Topaz Denoise job now correctly allocates CPU resources when run without GPUs.
Fixed
Installation test (cryosparcm test install) no longer fails if only clusters are connected to the instance.
Fixed
Starting a Live session that was previously marked as “completed” no longer incorrectly pauses the session twice.
Fixed
Queue dialog now correctly applies titles and descriptions that were set.
Fixed
Queue dialog now has a tooltip to clarify the toggle to show or hide lanes with applicable targets based on job resource requirements.
Fixed
Manually Curate Exposures job no longer fails for infinite or NaN CTF parameters, outputs a warning instead.
Fixed
Ab-initio jobs no longer break display of job cards when there are many classes.
Fixed
Inspect Picks job no longer fails during CTF calibration if exposures do not contain enough particle pick locations.
Fixed
3D Classification no longer fails when Output Diagnostic Plots at every Interation is set to False.
Fixed
When using cryosparcm cluster connect, the final connected cluster configuration is printed as the output of the command (returning to v3.3 behaviour).
Fixed
The range values for threshold sliders and input controls in the Curate Exposures job will now round display values to expected precision in all cases.
Fixed
The reload browser shortcut will no longer manually reject selected exposure in the Curate Exposures job.
Fixed
Table index cells in the Curate Exposures job now correctly update their colour when setting thresholds or manually rejecting/un-rejecting.
Fixed
Filenames and other non-number values will now display correctly in the hover preview popovers in all interactive jobs.
Fixed
Input fields in the Curate Exposures job can now be incremented using the arrow keys without changing the selected exposure in the micrographs table.
Over 10x speedup in job completion processes (check and passthrough outputs) for jobs with many output groups (e.g., 3D Classification with 20+ classes). Also speeds up “Mark as Completed” action for jobs.
Update
Added more detailed logging message when manually taking over projects in an instance using cryosparcm cli “take_over_projects()“
Fixed
CryoSPARC environment no longer uses LD_PRELOAD to load libtiff, resolving issues where dependencies of libtiff (libLerc and libstdc++) would not be found on some OSes (e.g., CentOS 7).
CryoSPARC v4.0.1
October 6, 2022
This release contains performance optimizations and various stability enhancements:
Tensorflow test is now disabled by default during cryosparcm test workers. To enable, specify —test-tensorflow. Read the full installation test guide here.
Update
The instance tab within the admin panel now loads primary system logs by default.
Update
cryosparcm errorreport will now add the entire app, app_api, and app_legacy log files to the generated archive.
Update
Updated the Windows shortcut for multi-selecting to use the ctrl key instead of the meta key.
Redesigned interactive jobs (Inspect Picks, Exposure Curation, etc.) with significantly improved functionality and scalability. Easily handle large datasets (25K+ exposures)
And much more! Be sure to watch the video and read the guide
New
CryoSPARC Live is now integrated as part of the CryoSPARC interface. Seamlessly transition from the primary interface into a Live session within the same browser tab: How to Access CryoSPARC Live.
New project Attach and Detach actions to easily move projects between CryoSPARC instances
New project Archive and Unarchive actions simplify moving projects to archival storage or changing a project location on disk
For new projects, the generated project directory name is a hyphenated variant of the project title instead of PXX. A configurable prefix may be added to project directory names for easy identification
Projects now use an instance-lock to prevent accidental corruption due to multiple CryoSPARC instances modifying the same project simultaneously
Projects and Jobs no longer prefix output files with cryosparc_PXX_ on disk, since project numeric identifiers may change when detaching and re-attaching a project. The prefix is added when downloading files through the browser instead
Download a job error report bundle from the Event Log tab of the Job Dialog
Download a system error report bundle from the Instance Logs tab of the Admin Panel
From the CLI, the cryosparcm errorreportfunction generates a system level error report bundle when the web-app is inaccessible
New
cryosparcm filterlogcommand filters logs by days, date, name, function, and level and adopts a more consistent logging format including timestamps at the beginning of logs lines.
New
cryosparcm cluster validatefunction validates cluster configurations during installation.
New
View improved database backup functionality with recent activity records. Available in Instance Information → Backups.
All jobs which process masks and volumes now accept masks sampled at different box and/or pixel sizes relative to the volume, which are internally resampled when appropriate.
Job: Extract from Micrographs now supports extracting and downsampling to two different box sizes via Fourier crop to provide smaller particles for early classification and reconstruction steps.
Update
Job: Global CTF Refinement default parameters no longer fit fourth order aberrations (Spherical Aberration and Tetrafoil) unless specifically enabled.
Update
Job: Local Motion Correction now includes an option to downsample particles via Fourier crop after correcting per-particle motion.
Update
Job: Ab-initio Reconstruction now includes an unused particles output when not all particles were used for reconstruction in the single-class case.
Update
Job: Gctf is now a “Legacy” job. Turn on “Show legacy jobs” in the job builder to unhide the job. Gctf also includes a CUDA libraries parameter to override CUDA version for compatibility with available Gctf binaries.
Update
Job: 2D Classification may now run without CTF correction by turning off the Do CTF correction parameter, which may be useful for negative stain data.
Negative stain 2D class averages, with and without CTF correction, from the MTA-HDAC-MBD core complex (EMPIAR-10539)
Update
Improvements to CryoSPARC Live interface:
In the Live Overview tab, enter rejection thresholds with free-form numeric input
Display particle picks in more distinguishable colours
Volume viewer includes action buttons for resetting and zooming
Update
Added Maximum number of local maxima to consider picking parameter to Blob Picker and Template Picker in CryoSPARC Live.
Improved libcufft library initialization when only major version files (e.g., libcufft.so.10) are available.
Update
Updated Extensive Workflow to issue warning rather than timeout error caused by long-running jobs.
Update
Previous v3.3 web application (now running at BASE_PORT + 7) does not start up by default. To start it, run cryosparcm start app_legacy.
Update
CryoSPARC Live (liveapp service) is integrated into the main web application interface running at BASE_PORT and is no longer available at BASE_PORT + 6.
Update
Automatically update session data sizes when a session is paused or completed for improved data management.
Fixed
Exposure Sets Tools no longer fails to intersect on path if mscope_params result slot is connected to both Exposures_A and Exposures_B input groups.
Fixed
Heterogeneous Refinement jobs no longer fails when Force hard classification is turned on and some classes become empty.
Fixed
Additional corrupt exposure error handling for Full-Frame Motion Correction, Patch Motion Correction and Patch CTF jobs.
Fixed
Scheduler no longer fails for all queued jobs when a single queued job fails to run.
Fixed
cryosparcm restore no longer fails when restoring against an empty database directory.
Fixed
Show database connection error message in job log if one occurs during job startup.
Fixed
cryosparcm restart no longer restarts all services if an incorrect service name is provided as an argument.
Fixed
Long running background processes in one project no longer block actions across all projects in the system.
Fixed
Fixed phase_shift attribute in CryoSPARC Live overview plot that was previously displayed as radians instead of degrees with an incorrect label.
Fixed
FFTs on CPU no longer use a minimum of 8 threads that was causing contention on some systems.
cryoSPARC v3.3.2
April 8, 2022 (Latest patch: August 24, 2022)
We’re excited to announce another release of cryoSPARC packed with new features, performance optimizations and various stability enhancements:
New
New 3D Classification Beta job can be used to quickly classify particles into many 3D classes, without re-alignment of particle poses and shifts. Sort out heterogeneity from million+ particle datasets with 100+ classes. Read a tutorial here.
New Blob Picker Tuner determines optimal blob picker parameters from a minimal set of user-selected manual picks. Tunes blob size, shape, and picker score thresholds automatically. Read a tutorial here.
New
A technique for Ewald Sphere Correction implemented in homogeneous refinement and reconstruction jobs allows for higher resolution reconstructions of larger proteins. Read a tutorial here. The new technique corrects for the Ewald Sphere during pose alignment and during back-projection, and also correctly estimates/refines higher-order CTF aberrations (HOA) while accounting for the Ewald Sphere.
Substantial speed and memory improvements to FSC calculations in all job types, speeding up FSC calculation by over 10x, and providing an overall speedup of up to 20% for homogeneous refinements. A benchmark refinement of 100,000 Ribosome particles at 360 pixel box size (EMPIAR-10028) takes only 5.2 minutes on a single NVIDIA V100 GPU.
New
Substantially improved TIFF read performance for some filesystems. Learn more.
New
cryosparcm log provides an improved log format and new filter flags.
Specify path of directory containing preprocessed micrographs for Topaz Train and Extract Patch 211214
New
New 3D Classification advanced parameter “Output data after every full iter” to enable data outputs (volumes, particles, volume series) after every full iteration Patch 211214
New
cryosparcm cli “update_parents_and_children_for_project(project_uid)” function to fix broken tree view input/output connections in previously-imported projects Patch 220215
New
Exposure/Particle Sets Tool option to ignore leading UIDs when comparing paths for intersection mode Patch 220215
New
Support for reading .mrc files containing float16 data (datatype code 12) Patch 220215
New
Option to remove all duplicates in Remove Duplicate Particles job Patch 220315
New
Pre-processing jobs no longer fail immediately when a child process encounters a segmentation fault Patch 220518
New
Patch Motion jobs now use an improved IO library, providing substantial speedups on some cluster file systems Patch 220518
New
Blob Picker Tuner job now reports progress during optimization Patch 220518
New
Validation (FSC) job now takes in an optional Override Pixel Size (Å) parameter to allow adjusting the final pixel size and resolution estimate if incorrect pixel size was specified during processing Patch 220518
New
Align 3D job now takes in particle stack(s) and rotates/shifts the alignments of the particles to match the rotation and shift applied to the volumes during 3D alignment Patch 220518
New
Project deletion (and other long running actions) are now decoupled from instance-wide locks, allowing a cryoSPARC instance to continue to perform unrelated actions while a deletion is ongoing Patch 220518
New
Patch Motion Correction now supports Fourier crop ratios of 1/8 and 1/16 Patch 220824
New
3D Variability Display (Cluster Mode) now has a volume series output Patch 220824
CLI endpoint to safely update a project’s directory (update_project_directory(project_uid, new_project_dir)).
Update
Ability to use common path field (vs. UID) to intersect or split in Exposure Sets Tool and Particle Sets Tool.
Update
CryoSPARC Live Data Management now supports executing a script upon a datatype’s state change. Add export CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_ENABLE=true and export CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_PATH=/path/to/script.sh in cryosparc_master/config.sh to enable this feature.
Update
Particle subtraction now respects refined higher-order aberrations.
Update
Added FSC mask auto-tightening to the standalone Validation (FSC) job.
Update
Extraction Box Size in Local Motion Correction now uses the input particle box size by default.
Update
3D Classification volume outputs are now unmasked, fixing downstream refinements Patch 211214
Update
New printout for each 3D Classification iteration to clearly specify O-EM/F-EM iterations left Patch 211214
Update
3D Variability Display has more informative .mrc volume file names (simple/intermediates mode) Patch 220215
Update
Improved error details when a cli method call fails Patch 220315
Update
Improved logging in Extract From Micrographs (GPU), now shows number of particles rejected near edges of each micrograph Patch 220315
Update
Improved EER read performance for some file systems Patch 220315
Update
CryoSPARC Live Session Data Management: All datatype sizes are now updated when a session is “paused” or “completed”.
Update
Helpful error messages when calling cryosparcm/cli functions with incorrect arguments.
Update
Check for Corrupt Particles job now has an option to check for NaN values in the particle data files to test for data corruption on disk.
Update
Extraction jobs now have an option to force re-extraction of CTF values for each particle from the input micrographs.
Update
Outputs in the Sharpening Tools job are now separated into two output groups and can be directly connected to downstream Volume Tools jobs Patch 220518
Update
Warnings in the Import Particles job’s Overview tab are now more visible when fields are missing in the imported metadata files Patch 220518
Update
Warnings in the Check For Corrupt Particles job’s Overview tab are now more visible when corrupt files are found Patch 220518
Update
The Local Refinement (NEW!) job now explicitly requires an input mask Patch 220518
Update
Blob picker output images now show more contrast for negative stain data Patch 220518
Update
Further improvements to error messages when problems are encountered while reading particles or micrographs Patch 220824
Update
Volume series outputs in 3D Variability Display no longer require a zip system dependency Patch 220824
Update
3D Variability Display (Cluster Mode) 1D histogram now includes a legend Patch 220824
Update
Warning if all three dimensions of the input data for Import Templates match, which suggests that a cubic (3D) volume has been input instead of a stack of 2D templates Patch 220824
Update
Summary of rejected particle counts and reasons for rejection in Extract From Micrographs jobs Patch 220824
Update
Jobs now fail if they require more SSD cache space than the total available space or quota Patch 220824
Update
If available, show MRC file name in job log when file cannot be read Patch 220824
Update
Improved error formatting when running functions via cryosparcm cliPatch 220824
Fixed
Fixed contrast issues in Inspect Picks, Manual Picker and Curate Exposures jobs when viewing a micrograph that wasn’t motion-corrected in cryoSPARC.
Fixed
Fixed issue in Volume Tools that caused ringing artefacts when downsampling masks.
Fixed
Added resolution parameter to low-pass filter templates in Blob and Template Picker jobs.
Fixed
Fixed libtiff error when running the worker on a read-only file system.
Fixed
Corrected volume flip in cryoSPARC Live 3D volume web viewer.
Fixed
Fixed a bug that caused stale notifications to persist after restarting cryoSPARC.
Fixed
Fixed a bug where it was not possible to queue a job into a new workspace.
Fixed
Fixed a bug where topics from the cryoSPARC Discussion Forum would not display on the dashboard.
Fixed
Updated the job queuing interface to clarify which nodes certain jobs are launched on. Interactive jobs are always launched on the master node. Import jobs are launched on the master node unless CRYOSPARC_DISABLE_IMPORT_ON_MASTER is set to true, in which case import jobs can run on any lane/node selected. All other job types launch on a lane/node the user selects.
Fixed
Fixed bug where symmetry alignment during refinement often aligned volumes incorrectly if they were already aligned.
Fixed
Fixed bug where volumes are improperly aligned to symmetry axes in helical refinement.
Fixed
Fixed an error during startup that occurs when an instance has less than 1000 notifications v3.3.1
Fixed
Fixed plotting error during (legacy) local refinement Patch 211202
Fixed
Use correct license check SSL certificate configuration when running cryoSPARC in insecure mode Patch 211214
Fixed
Fixed Blob Tuner Error when Blob picker finds no particles Patch 211214
Fixed
Passthrough GS split in 3D Classification Patch 211214
Fixed
Check for empty splits in Homogenous Reconstruction job Patch 211214
Fixed
Prevent crash of Rigid Motion Correction, Local Motion Correction and Extract Particles jobs when encountering corrupt exposures. The jobs now continue after any error in a single exposure, and output all incomplete exposures as a separate output Patch 220118
Fixed
Set class field for 3D Classification particle outputs (fixes bug with downstream particles and csparc2star.py) Patch 220118
Fixed
In 3D Classification, viewing direction distribution plots work with empty classes Patch 220118
Fixed
In Topaz Extract, use provided preprocessed directory when no pre-trained model is specified Patch 220118
Fixed
Importing projects retain job tree view input/output connections Patch 220215
Fixed
Symmetry Search Utility no longer fails with KeyError Patch 220215
Fixed
Topaz Train imported jobs no longer reference preprocessed micrographs directory from old project Patch 220215
Fixed
Remove unused “Number of GPUs to parallelize” parameter from Local Motion Correction (non-multi) Patch 220215
Fixed
Include 1st and 99th percentile particles in Inspect Picks threshold range Patch 220315
Fixed
Use high-order aberration information in ResLog job Patch 220315
Fixed
Local Motion does not fail when some movies do not include particles Patch 220315
Fixed
Filter correct component reaction coordinates in 3D Variability Display job Patch 220315
Fixed
Prevent error in Local Motion Correction when zero-shift frame is set Patch 220315
Fixed
Improve multithreaded performance for some operations Patch 220315
Fixed
Local motion no longer fails when zero shift frame is not set by previous motion correction job.
Fixed
AttributeError raised in Filament Tracer job when only one particle pick is found on a given filament.
Fixed
GPU FSC calculation in Refinement jobs raises AssertionError when used with box sizes under 84.
Fixed
Topaz job now correctly fails when topaz subprocess fails.
Fixed
Micrograph contrast in Manual Picker job is sometimes set to a value that clips the image.
Fixed
Support for 3DVA Display clustering with one component.
Fixed
Return HTTP 404 response when GridFS image requests are not found.
Fixed
Allow clearing Extract Particles jobs created before the previous patch Patch 220518
Fixed
Prevent Motion Correction job failure due to low-level memory error from corrupt TIFF files Patch 220518
Fixed
Corrected stdout/stderr redirect options in cryosparcm cluster example scripts Patch 220518
Fixed
Manual picker and Inspect picks jobs no longer overly downsample rectangular (K3) images Patch 220518
Fixed
Job status timestamps are now correctly cleared when a job is cleared Patch 220518
Fixed
Patch CTF job no longer fails with data from spherical aberration-corrected microscopes Patch 220518
Fixed
Local Resolution Estimation job now reads the mask from the “Static Mask” input group Patch 220824
Fixed
.mrc.bz2 files with an MRC extended header now load correctly Patch 220824
Fixed
workspaces.json now correctly updated when a cryoSPARC Live session is running Patch 220824
Fixed
Prevent SSD cache system failure when file size is missing in cache files database collection Patch 220824
We’re excited to announce another release of cryoSPARC packed with new features, performance optimizations and various stability enhancements:
New
Added option for sorting jobs by date and title. Projects and workspaces can be sorted by title as well
New
Check Particles job for verifying data integrity of particle stacks Patch 210511
New
Computational batch size parameter in Heterogeneous Refinement job Patch 210511
New
cryosparcm changeport command and related utilities for changing the base port Patch 210601
New
New utility cryosparcm licensestatus to verify if a license is valid Patch 210629
New
cryosparcm status now reports patch version Patch 210713
New
Set environment variable CRYOSPARC_SSD_CACHE_LIFETIME_DAYS in cryosparc_master/config.sh to auto-remove cache files older than the given number of days Patch 210803
New
New TIFF reading strategy for improved performance on network file systems such as GPFS and Lustre. To enable, add export CRYOSPARC_TIFF_IO_SHM=true to cryosparc_worker/config.shPatch 210817
New
”Reassign Particles to Micrographs” job to reassign particles to their associated micrographs Patch 210831
New
Full Prefix/suffix cutting parameters for matching between input movies and micrographs in “Import Micrographs” job Patch 210831
New
Extend master/worker communication timeout with CRYOSPARC_CLIENT_TIMEOUT environment variable Patch 210831
New
Download EMDB-friendly XML files from FSC figures Patch 211012
Update
Substantially improved performance in iterative particle-processing jobs (2D Classification, Refinement) when particles are stored on some types of filesystems (including FSx for Lustre), resolving cases where some users found that jobs run in v3.1 were slower than in v2.15
Update
Deep Particle Picking jobs (Deep Picker Train, Deep Picker Inference) now require CUDA Toolkit 11+
Update
The ‘Metadata’ tab within the job preview dialog is more performant and features expandable sections and colour highlighting based on data type
Update
Import Particles no longer ignores the rlnImagePixelSize column while trying to import a particle .star file
Update
CryoSPARC processes are now less likely to become orphaned due to more robust supervisor management configuration
Update
Removed unused cuBLAS dependency that caused multiple GPU contexts on all GPUs in a system to be created even when using only a single GPU
Update
GPU status info is now updated when a worker is connected for the first time, or when a worker connection is updated
Update
New option to work around bug in CUDA on CentOS 7 that causes cuMemHostAlloc failed errors in multiple job types. To engage this, add export CRYOSPARC_NO_PAGELOCK=true to the cryosparc_worker/config.sh file.
Update
Patch CTF Estimation no longer fails when encountering an error on a specific micrograph. Failed micrographs are outputted to a separate group if exceptions are encountered and processing continues.
Update
Local Resolution job now prints local resolution statistics (minimum, median, maximum, 25th percentile, 75th percentile) in job event log Patch 210401
Update
CryoSPARC Live Data Management now supports executing a script upon a datatype’s state change. Add export CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_ENABLE=true and export CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_PATH=/path/to/script.sh in cryosparc_master/config.sh to enable this feature Patch 210401
Fixed
Issue in Patch Motion that causes dose weighting to be incorrectly applied in some cases. This was caused by the introduction of variable-dose support in v3.1.
Fixed
Patch CTF Estimation fails with IndexError: index 0 is out of bounds for axis 0 with size 0
Fixed
Patch CTF Estimation fails with ValueError: Axis limits cannot be NaN or Inf
Fixed
Patch CTF Estimation sometimes reports underestimated CTF fit resolution in cryoSPARC Live, in cases where the correlation between the fit and the actual signal is very poor at low resolutions
Fixed
Issue that can cause exposure groups set up at import to be lost during processing, specifically through wrapper jobs for GCTF or CTFFIND
Fixed
Issue where Deep Picker Train job would fail reporting that there are no training particles on any micrographs
Fixed
Topaz wrapper TypeError during negative stain processing: can’t multiple sequence by non-int of type numpy.float32
Fixed
Issue causing IndexError in wrapper for Motioncor2
Fixed
Issue in particle simulator job where all particles would be modulated by the same CTF regardless of specified input CTF parameters
Fixed
Issue in cryosparcm cli delete_user command that prevented an admin user from being authenticated
Fixed
Issue in Extract From Micrographs (CPU) job requiring a GPU dependency, causing it to fail on CPU-only workstations
Fixed
Import Job/Import Result Group fails with Unable to find data referenced by cs files when importing a mscope_params result group due to empty fields Patch 210401
Fixed
Local Motion correction with EER data Patch 210413
Fixed
Jobs no longer wait on cache forever after the cache is unlocked in other jobs Patch 210413
Fixed
Restore missing “Use Class Weights” parameter in Deep Picker Train job Patch 210413
Fixed
Prevent LD_PRELOAD warning when launching cluster jobs Patch 210413
Fixed
ValueError during 3DFSC job volume output stage Patch 210511
Fixed
Empty workspaces after importing project Patch 210511
Fixed
Prevent error when extracting particles generated by Topaz Patch 210601
Fixed
Prevent error in Deep Inference job during plotting Patch 210601
Improved error messages when reading invalid exposures or particle stacks Patch 210601
Fixed
Prevent numeric initialization error in Ab-Initio Reconstruction jobs for some systems Patch 210615
Fixed
Prevent numeric error in Inspect Picks job Patch 210615
Fixed
Statistic showing the total number of particles being processed by Ab-Initio Reconstruction job in cryoSPARC Live sometimes isn’t updated Patch 210629
Fixed
Prevent mrc_readmic (1) 447: Invalid argument errors that occur after extracting particles in a pipeline where micrographs were imported multiple times Patch 210629
Fixed
Prevent I/O exceptions from terminating the cryoSPARC Live Worker while processing exposures Patch 210713
Fixed
Allow shell operators during cluster job submission, enabling redirections required for LSF Clusters Patch 210713
Fixed
Exception handling for Clear Intermediate Results Patch 210831
Fixed
Corrected cryosparcm cluster example JSON output Patch 210831
Fixed
Consistent exposure/particle matching in Deep Picker Patch 211012
Fixed
Prevent startup error when system does not report CPU frequency Patch 211012
Fixed
Prevent loss reporting error during Topaz jobs Patch 211012
Fixed
Prevent create directory errors on some file systems Patch 211012
Helical refinement now supports Non-Uniform regularization
New
Symmetry expansion now supports helical symmetry
New
In helical refinement, shifts along the helical axis can be limited to the asymmetric unit, improving resolutions for many datasets
New
”z-clip” parameter added to helical refinement’s dynamic masking, to generate a mask covering only the central region along the z-axis
Stability and Performance Updates
New
New Extract from Micrographs (CPU) job, supporting particle extraction on CPU in parallel using multiple cores.
New
Admin users can now modify the list of users to whom a project is shared
Update
Major updates to SSD caching to avoid issues of Detected file change due to change in file size and improve performance
Update
Several robustness improvements for Patch-CTF Estimation. Previous behaviour can be enabled using “Classic mode” parameter.
Update
You can now quickly queue interactive jobs via the job details in the sidebar. The queue dialog has also been updated to clarify what machine the interactive jobs will run on.
Update
Import Movies and Patch Motion now skip bad files instead of failing. Skipped files are added as a separate output.
Update
Import Particles no longer ignores the data_optics section while trying to import a particle .star file. Exposure groups and CTF parameters other than high-order aberrations are retained from data_optics table.
Update
Rebalance 2D Classes now has a “Override maximum superclass size” parameter that can be set to override the maximum size of all superclasses
Update
When launching a job, hidden files (files that start with a ’.’) will be ignored when checking if the job directory is empty if the CRYOSPARC_IGNORE_HIDDEN_FILES environment variable exists (add to cryosparc_master/config.sh)
Update
Homogeneous Reconstruct now has the option to reconstruct with helical symmetry, and to reconstruct with an inverted hand
Update
Homogeneous Reconstruct now supports input alignments on any pixel size, regardless of the reconstruction pixel size
Update
cryoSPARC Job Scheduler will now ignore CPU requirements when queuing jobs to non-cluster lanes; only RAM and GPU will be considered
Update
Change the parameter “Remove leading UID in input micrograph path” in Import Particles to be optional
Fixed
Discussion forum topics have returned to the dashboard
Fixed
Some users were unable to create an account or reset their password in Chrome
Fixed
Fixed a bug in Import Particles that caused it will fail when trying to convert rlnOriginX/rlnOriginY to rlnOriginXAngst/rlnOriginYAngst while importing Relion 3.1 star files
Fixed
Fixed a bug that caused the cryoSPARC start process to not complete
Fixed
Fixed a bug that caused the string of characters appended to the micrograph file paths created by cryoSPARC to be a variable length (now 21 characters)
Fixed
Fixed a bug that caused the Import Project function to fail with _pickle.UnpicklingError: could not find MARK
Fixed
Filament tracer now eliminates particles beyond the micrograph’s edge
Fixed
Fixed a bug that prevented the Rebalance 2D Classes job from launching on clusters
Fixed
Fixed a bug in the Volume Tools job that gave erroneous output when the “Invert mask” and “Fill holes” options were both true
Fixed
Fixed a plotting issue in Heterogeneous Refinement causing orientation plots to fail with a ValueError
Fixed
Fixed issue that prevented pre-v3.0 Extract From Micrographs, 2D Classification, and Create Templates jobs from being cleared
Fixed
Fixed issue causing static masks to be improperly resized in Homogeneous Refinement New, Non-Uniform Refinement New, and Helical Refinement
Fixed
Fixed issue preventing Helical Refinement from running with static masks that were sampled on a different grid than the volume; fixed related bug with static masks being improperly rotated when the “Align input structure to helical (z) axis” parameter was activated
Fixed
Fixed bug where running “Clear Intermediate Results” on a project caused the outputs of Curate Exposures jobs within the project to be inadvertently deleted.
Fixed
Add support for pre-release versions of Topaz
Fixed
Fixed issue where job queue modal would display incorrect default lane/target info when ‘default’ lane doesn’t exist
Fixed
Fixed issue where systemd service used wrong python interpreter
Fixed
Fixed issue where deleted sessions still appear in cryoSPARC Live browse view
Fixed
Fixed issue in cryosparcm updateuser when using lastname flag
Fixed
Fixed IndexError in motioncor2 wrapper job
Fixed
Fixed issue where Topaz denoise would not work
CryoSPARC Live
Update
Clicking on the number of failed exposures in the navigation sidebar will open a dialog displaying the error details for each exposure
Fixed
Fixed a case where a deleted session would display in the browse page when selected
Fixed
Fixed a bug that caused manual picks on “Test” exposures to be excluded from session statistics and from being extracted
Fixed
Job cards in the “current jobs” and “target” modals will now open their respective job modals when clicked. Job cards will also display the job’s priority if one has been set
Fixed
Fixed a bug in the cryoSPARC Live Worker that caused it to fail repeatedly due to uncaught exceptions caused by IO errors
Fixed
Blob and template particle diameters are now correctly shown as Å values
cryoSPARC v3.0.1
December 15, 2020
Update
Disable existing conda environments before activating cryoSPARC’s conda environment
Fixed
Specifying initial user credentials in install.sh fails with “improper usage” because of incorrect arguments
Fixed
Command cryosparcm updateuser fails because of incorrect arguments
Fixed
cURL symbol lookup error on CentOS8 when starting cryoSPARC after update to v3.0.0
Fixed
Importing Projects from v2.15.0 or earlier into v3.0.0 fails with UnicodeDecodeError
Fixed
Command cryosparcw newcuda uses old pyCUDA binary that doesn’t exist
Fixed
Deleting an Extract From Micrographs job fails with KeyError
Fixed
Importing particle pose information in Import Particles fails with TypeError
Fixed
Command cryosparcm backup fails if cryoSPARC is turned off
Fixed
Inspect Picks job fails with ValueError if using micrographs imported with “Constant CTF” parameter
Fixed
Symmetry Search Utility job fails with TypeError in “rise” mode
cryoSPARC v3.0
December 9, 2020
We’re excited to announce another release of cryoSPARC packed with new features, performance optimizations and various stability enhancements:
CryoSPARC Live New
New
New and improved cryoSPARC Live app, redesigned and reworked for robust and seamless real-time end-to-end cryo-EM data processing. CryoSPARC Live is built for data collection facilities, cryo-EM cores, industry labs, microscope operators and individual users who want to perform real-time cryo-EM data quality assessment and decision-making based on streaming 2D and 3D results that update in time with new data. CryoSPARC Live is also now the best and fastest way for users to perform their first-cut processing and curation on already available data.
The ability to seamlessly reprocess data at any time, to curate exposures, test/refine picking strategies and the choice of particles for refinement, on the fly
Streaming 2D classification, in-line Ab-initio Reconstruction, and Streaming high-resolution 3D Refinement, which update automatically as new particles become available
Seamless integration with cryoSPARC, allowing for direct advanced processing from the results generated in cryoSPARC Live without any data shuffling
After extensive beta testing globally, cryoSPARC Live is now available free of charge for general academic use. To learn more and to access Live, please see: guide.cryosparc.com
Improved Non-Uniform Refinement algorithm NEW and job type Beta. Higher resolution results and reduced sensitivity to parameter changes. For more details, see the new paper published in Nature Methods
Plus:
Support for local and global CTF refinement during non-uniform refinement
Fast, new GPU implementation of non-uniform regularization developed with support from NVIDIA
Additionally, support for automatic adaptive marginalization over alignment variables (enabled by default) yielding further improvements for small molecules.
New
Upgraded to modern Python 3.7 runtime, including new updated dependencies and installation. Easy upgrades and downgrades between cryoSPARC v2 and v3.
New
Support for CUDA 11 + Ampere Architecture GPUs (3080, 3090, A100, etc.).
Full support for Thermo Fisher Scientific EER data in cryoSPARC and cryoSPARC Live. EER data is decompressed on the fly and can be upsampled and split into the desired number of fractions during processing.
New
Improved defect file and defect pixel support in motion correction jobs in cryoSPARC and cryoSPARC Live. Defect pixels can be specified in a .txt defect file or as zeros in the gain reference.
New
Helical ReconstructionBETA: New suite of tools for end-to-end helical processing and refinement, including:
New high-resolution Helical Refinement job that also supports non-uniform regularization, and enforcement and local optimization of helical symmetry
Extract From Micrographs will now re-center particles by default if they have already been extracted and aligned through 2D classification, 3D reconstruction, or refinement.
Ability to set priority values for jobs and users in cryoSPARC and cryoSPARC Live, allowing jobs to run in priority order instead of the order they were queued. Learn more about this feature with our Guide to Priority Job Queuing
New
CryoSPARC Live Session Data Management: View, delete and archive data created by cryoSPARC Live on a granular level. Save space taken up by data generated in old sessions and projects, while still being able to browse and view results.
New
A new Data Management tab is available in the Resource Manager in cryoSPARC, displaying details of all projects and workspaces available to the logged in user. At-a-glance information includes: project creation and ownership info, project directory location, last accessed/viewed, project size and other details.
New
Reduced GPU memory footprint in Patch Motion Correction. The overall GPU memory footprint of patch motion correction has been reduced. This enables most K3-sized image data to be processed on 11GB GPUs. Additionally, a “low-memory mode” switch is now available, which reduces it even further (at expense of speed).
New
NewRemove Duplicate Particles job checks for particles that are extracted too close together on a micrograph and keeps only one, to allow combining picks from multiple pickers without affecting FSC estimates.
New
BetaPatch Motion Correction in cryoSPARC and cryoSPARC Live now optionally support movie data that has been saved with a variable dose rate among frames (e.g., from Leginon)
New
NewSystemd Unit File to restart cryoSPARC after server shutdown: A systemd unit file is now included to allow running cryoSPARC as a service.
Update
Improvements to Patch CTF estimation for better CTF fit in cases where background or envelope estimation is difficult. Improved robustness in relative ice thickness measure.
Update
Downsample Particles and Extract from Micrographs will now correctly scale particle alignment shifts, fixing issues where downsample was used immediately prior to local refinement or 3D variability.
Update
Orientation distribution and posterior precision plots were added to Heterogeneous Refinement, and can be disabled by turning off the “Show viewing orientation distribution plots” parameter.
Update
Import Micrographs now has an option to output constant CTF values, so that subsequent steps will ignore the CTF entirely. Useful for negative stain and exotic data collection.
Update
Improved cryosparcm patch pre-apply validation.
Update
Reduced SSD-cache coordination bandwidth, improving the robustness of caching on multi-node systems.
Update
Smarter cluster job ID parsing following cluster submission. Ability to use shell redirects in cluster submission commands, enabling native support for the LSF cluster management system.
Update
Each project, workspace and job now records a ‘last accessed’ timestamp, updating whenever a user views the project, workspace or job through the web application interface. This allows for administrators to quickly identify and remove/archive data that is no longer used.
Update
Queuing interactive jobs always default to the master node.
Update
Ability to specify a first and last name for new users in the Admin panel and through the command line interface.
Update
Default umask is now 002, to allow write permissions by ‘group’ for folders created by cryoSPARC.
Update
Renamed internal installation directories from cryosparc2_ to cryosparc_
Fixed
Import Particles job fixes: Importing particle locations using micrographs created in v2.15+ failed because of a variable length UID prepended to the path of each micrograph; Using the “ignore raw data” parameter causes the job to fail with a missing pixel size error; Support for offsets specified in angstroms (_rlnOriginXAngst, _rlnOriginYAngst).
Fixed
Import result group fails if some path fields are blank.
Fixed
Inspect Picks “use_denoise” button failed to actually load denoised micrographs, even if they were connected.
Fixed
Local Refinement now respects the static masking and minimization-over-per-particle-scale parameters, which were previously not used correctly.
Fixed
Setting “Override number of iterations” to True without providing a number of iterations previously prevented Local Refinement from ever terminating.
Fixed
Non-Uniform Refinement: Local processing causes an error if masks are too close to the edge of the refinement box.
Fixed
Local Filtering had SSD:True hard-coded, causing it to fail when running on resources without SSDs.
Fixed
Usernames are now correctly displayed in the details panel for a job.
cryoSPARC v2.15
May 13, 2020
New cryoSPARC Guide Beta
New
First version of a new, comprehensive cryoSPARC guide! Available at guide.cryosparc.comBeta. This guide will consolidate all existing documentation and be populated with cryo-EM data processing tutorials, tips, descriptions of methods, case studies, etc.
Release Highlights
New
Use the new cryosparcm patch command to install bug fixes and improvements without a full update
New
It is now possible to set a project-level default value for whether or not to cache particle images to the local SSD before running jobs within a project. The parameter can be adjusted from the sidebar when a project is selected
New
New Particle Class Probability Filter job: Filter particles based on the probability of matching their assigned 2D or 3D classes
Update
In the Topaz Denoise job, the default values for the following parameters have been altered to better suit Topaz: The default value for the “shape of split micrographs” parameter has been changed to 1536. The default value for the “padding around each split micrograph” parameter has been changed to 256.
Update
The Topaz Cross-Validation job has been updated with various improvements: Added option to output models from subsidiary training jobs when running cross-validation. Added expected number of particles and training radius as cross-validation parameters. Improved GPU distribution for subsidiary training jobs.
Update
All Topaz jobs now store preprocessed micrographs to avoid unnecessary preprocessing in subsequent Topaz jobs.
Update
Added option for the Extensive Workflow to run additional job types, including Local Motion Correction, Multi-class ab-initio and 3D Variability. This can be used as a full system test for new installs and updates.
Update
Use dynamically-generated SSD cache paths by including environment variables in the CRYOSPARC_SSD_PATH definition for a worker’s config.sh
Update
Import Movies and Import Micrographs jobs are now parallelizable across multiple threads while reading the headers of all exposures to import
Update
Initial and final mini-batch size parameters for Ab-initio Reconstruction can now be set to zero to automatically tune based on the estimated particle size
Update
The project and workspace switchers, which allow you to quickly search and navigate, have been revamped. When you’re viewing a project or workspace, the switcher will highlight and scroll to it, allowing you to easily navigate relative to them. When viewing a job within a workspace, the workspace switcher will display a link icon next to every workspace that particular job is linked to.
Update
The job history tab within the resource manager page now shows only jobs that have not been deleted by default. Select the ‘show deleted’ toggle to show only jobs that have been deleted.
Update
Jobs can now only be imported from inside the project directory. This fixes a bug that causes particles to not be cached if they were imported from outside the project.
Update
Executing a cryosparcm command is now only allowed by the owner of the script and on the master node. You can override this parameter by setting CRYOSPARC_FORCE_USER or CRYOSPARC_FORCE_HOSTNAME to true in your shell (or permanently inside cryosparc2_master/config.sh).
Fixed
Bug where multiple jobs launched simultaneously using the same particles could result in endless “cache waiting” messages
Fixed
Full-frame Motion Correction runs out of memory on 8GB cards with K3 data. There is now a switch to reduce memory usage by approximately half
Fixed
ValueError was raised in 2D classification or Patch CTF Estimation jobs when movies that were imported separately but contain files with the same names get overwritten
Fixed
All refinement jobs now use the first (rather than last) FSC threshold crossing for filtering during iterations
Fixed
Bug in Particle Set Tools intersection when set A is smaller than set B
Fixed
Bug in Heterogeneous Refinement which fails if intermediate plots are disabled
Fixed
Manual Picker fails while calculating the contrast intensity range for a micrograph with an irregular shape
Fixed
Topaz Denoised micrographs display fail to display correctly in the Manual Picker job (as either gray or black images)
Fixed
Micrographs captured at a low magnification fail during Patch CTF Estimation with an IndexError
Fixed
Select 2D fails when trying to select classes from a particle .mrc file that is too large to fit in memory. Also fixed a bug where an IndexError is raised if a class has less than 3 particles in it.
Fixed
Exposure Group Utility job fails with a KeyError when an input dataset has a missing CTF result field
Fixed
Inspect Picks job fails if you try to extract locations and the box size set is not an even number
Fixed
Downsample Particles now exports particles in the correct order if the input particles are out of order on disk
Fixed
MotionCor2 Wrapper: total dose is incorrectly specified as the frame dose argument for dose weighting
Fixed
MotionCor2 Wrapper: output pixel size is incorrect after specifying a Fourier crop factor, causing CTF Estimates to fail
Fixed
Project directory sizes don’t update when a project is deleted
Fixed
Allow importing movies shot on high-aspect ratio detectors (2:1 aspect ratio or higher)
Fixed
Resolved version incompatibilities with Topaz version 0.2.4a
cryoSPARC v2.14
February 20, 2020
Release Highlights
New
The job builder includes a search bar to quickly find the job you’re looking for
New
“Cache Particles on SSD” Job that will cache particles on a node of your choice
Update
Many optimizations across cryoSPARC jobs - datasets containing a large number of items will see significant speedups
Update
Added option to use pre-trained initializations in Topaz Train
Update
Added option to use pre-trained model when running Topaz Extract
Update
All jobs that output FSC plots have been updated to allow for downloading the raw data for external plotting
Update
Curate Exposures job can be run without connecting micrographs, allowing for CTF-based curation using only movies
Fixed
Topaz Denoise fails when trained model is input
Fixed
Template Picker fails on K3 micrographs motion corrected using external software
Fixed
Turning on the “Generate new FSC Mask” parameter in the Sharpening Tools job caused an AssertionError
Fixed
“Particles Excluded” output group image in Select2D job was the same as the “Particles Included” output group image
Fixed
New Homogeneous Refinement job sometimes fails when resampling initial model because boxes were not even sized
Fixed
Performance regressions in the “Extract from Micrographs” job where the time to extract particles increased non-linearly with a larger number of exposures
Fixed
In 2.14.2: “Inspect Particle Picks” job fails with a KeyError when it encounters an exposure with no particle locations
cryoSPARC v2.13
January 28, 2020
v2.13 is packed with a number of updates including interactive 3D variability analysis, reduced memory requirements for refinement of large box sizes, seamless support for negative stain and phase plate data, and more!
New
The 3D Variability Display job output now includes an in-browser, in-line interactive 3D scatter plot to analyze clusters in the distribution of particle reaction coordinates! The user can select which three reaction coordinates to display, and individual clusters can be shown or hidden. Read the new full tutorial, including examples of results:
When inspecting particle picks from template or blob pickers, particle pick scores (NCC) and power scores are automatically calibrated with respect to the defocus of each micrograph. This ensures that setting an appropriate threshold on one micrograph will yield good picks on other micrographs with very different defocus.
Negative stain and phase-plate datasets are now more easy to process. A single toggle can be set during import and no other parameter changes are necessary in subsequent job types.
The blob picker can now use “ring” shaped blobs, which perform well in picking viruses and other spherical particles (eg. apoferritin).
New
Relion 3.1 format .star files can now be imported without errors, though optics group data is not currently parsed.
New
Manual picker interactive job now has adaptive robust contrast normalization to make seeing particles in imperfect images easier.
New
Manual Picker interactive job now includes a “Contrast Intensity Override” slider that allows users to manually select the contrast bounds when viewing a micrograph. The average, patch-based intensity is used by default for the Manual Picker and Exposure Curation jobs.
New
Validation (FSC) Job: Validation of a refinement result (i.e. half-maps) using the FSC is now available as a separate job type, with optional mask input.
Update
Fourier-space plots in Homogeneous Refinement display correctly.
Update
Multiple updates to Topaz-wrapped jobs in cryoSPARC:
Added multiprocessing to substantially improve preprocessing times in all Topaz jobs
Added parameter to select number of CPUs in the Topaz Extract job
Topaz jobs now automatically account for negative stain data
Topaz-extracted particles no longer cause 2D classification jobs to occasionally fail due to inconsistent CTF outputs
Update
Passthrough output files now include the project and job numbers for easier referencing (e.g., P1_J1_passthrough.cs)
Update
Manual Picker job now fails by default if the CTF inputs are not connected. An override switch has been added to allow manual picking without CTF estimates, which is not the recommended workflow.
Update
3D Variability Display can now flip the handedness of output volume series.
Fixed
CPU and GPU memory usage of Homogeneous Refinement is substantially reduced. In tests, it is now possible to refine a 1024 box size on a 11GB consumer-grade NVIDIA GPU (GTX 1080Ti), though this does require ~160GB of CPU RAM. A smaller 256 box size can run on a 4GB GPU and requires only 6GB of CPU RAM.
Fixed
The template picker now uses substantially less GPU memory for large micrographs (e.g., K3 super-resolution data).
Fixed
Local per-particle motion correction of K3 data now works correctly.
Fixed
Bug that would cause cufftInvalidPlan error during refinement.
Fixed
Bug where two Select 2D jobs are jobs connected to each other, the last Select 2D job shows incorrect class average thumbnails.
Fixed
Bug where the command_vis would repeatedly try to start but fails, using 100% CPU in the process.
Fixed
(Added in v2.13.2) Bug where mrc volumes were being saved as float64 files, which the mrc file format doesn’t support, causing the Sharpening Tools job to fail
Fixed
(Added in v2.13.2) Bug where Heterogenous Refinement and Multi-Class Ab-Initio jobs would fail due to a ValueError
cryoSPARC v2.12
November 29, 2019
Major updates to 3D Variability: Solve more modes, smaller motions, automatic classification of discrete classes, and more!
New
New “cluster” mode in 3D Variability Display allows for automatic hierarchical 3D classification based on variability coordinates: directly visualize and see how many discrete classes are present in a dataset, separate them by clustering, and reconstruct each separately. Then recursively perform 3D Variability to look for sub-conformations or flexibility within each cluster, or refine each cluster separately.
New
New “intermediates” mode in 3D Variability Display allows for weighted reconstructions of intermediate positions along each variability dimension: create more detailed videos of variability that minimize the effect of density appearing and disappearing during large motions. Also output weighted particle subsets for intermediate positions, for refinement of particular flexible conformations.
New
Default options in 3D Variability improve sensitivity to small variations in the 3D density. Plus support for coloured or white noise models, high-pass filter during optimization, per-particle scale optimization.
Fixed streaking issue so that (many) more variability modes can be solved simultaneously.
Major updates to 3D Refinement: On-the-fly CTF refinement, aberration correction, exposure groups, substantial speed improvements!
New
Per-particle defocus refinement implemented on GPU allows CTF refinement in minutes for 100,000’s of particles. Implemented as a standalone job and also as an in-line on-the-fly option during Homogeneous Refinement.
Global CTF Refinement and Local CTF Refinement jobs: CTF aberration support, also GPU implemented, corrects for beam tilt, trefoil, spherical aberration, and tetrafoil. Estimation of the new parameters can be done in a standalone job or as an in-line on-the-fly option during Homogeneous Refinement, yielding ultra-high resolution reconstructions.
New Exposure Group Utilities job allows users to split or combine datasets into multiple exposure groups for higher-order aberration correction. Create exposure groups using regular expressions and separators, and test your exposure group creation strategy.
New Topaz wrapper jobs for integrating Topaz’s particle picking and micrograph denoising functionality
Topaz Train, Topaz Cross Validation, and Topaz Extract jobs perform particle picking while the Topaz Denoise job performs micrograph denoising
Users only need to install Topaz to incorporate it within the cryoSPARC pipeline
Denoised micrographs are compatible with Manual Picker, Inspect Particle Picks, and Topaz’s particle picking pipeline, assisting users and machine learning models in observing micrographs
Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153–1160 (2019) doi:10.1038/s41592-019-0575-8
Other Updates
New
New Symmetry Expansion job expands particle poses around point-group symmetries, to support local refinement of subunits, as well as 3D variability of symmetric particles.
New
New Job Scheduler and Queuing Interface: Override the cryoSPARC scheduler to queue directly to a specific GPU
Ability to perform non-interactive threshold selection for particle picking and extraction, and non interactive selection of 2D classes, making it possible to process simple data completely end-to-end without manual intervention
New
Ability to interactively select all classes above or below a threshold in Select 2D job (right-click a class)
New
Multiprocessing robustness updates in preprocessing jobs (Patch Motion, Full Frame Motion Correction, Patch CTF Estimation) ensure that if a job fails or is interrupted, completed results are recovered and processing can be resumed.
Update
Orientation-precision plots in refinement are now coloured with minimum value zero so that differences in ratios can be easily spotted
Update
Relative ice thickness is now available in Exposure Curation
Update
2D Classification allows for changing the maximum alignment resolution and reconstruction resolution separately if desired
Update
Sharpening job now applied a spherical window by default
Update
Added option to skip header check when importing movies, which allows for faster importing on larger datasets
Update
Importing Movies, Micrographs or Particles will auto-populate the “exposure group ID” field of the dataset based on the project (each project increments from 0)
Update
Set the environment variable CRYOSPARC_DISABLE_IMPORT_ON_MASTER to allow master-direct jobs (import jobs, curate exposures, select 2D) to be queued on any machine you specify, and not only the master node
Update
If a job is queued to a specific GPU, it will be noted in the Resource Manager
Update
Job Cards now show queue message if placed in queue
Update
Job type is more prominently shown in the sidebar
Fixed
Ab-initio job with symmetry enforced fails with invalid memory access error
Fixed
Certain parameter descriptions in the Job Builder don’t show
Fixed
Better logging in Align 3D Job
Fixed
Error in non-uniform refinement when run with CPU option
Fixed
3DFSC job doesn’t accept mask inputs
Fixed
New ignore_pose option in import particles to enable import of locations when pose information is missing
Fixed
Deadlock in new job types like 3D var on some newer GPUs
Fixed
Select 2D fails when all classes are selected (i.e., if there are no particles in the templates_excluded output result group)
Fixed
White circles in 2D Class job when circular mask diameter parameter is too close to particle box size