Correct free SSD cache storage calculation to prevent infinite cache hang when there is enough available SSD storage

Correct retrieval of legacy cluster configurations

Live no longer stops finding new exposures

SSD Cache system correctly retries up to 3 times on network timeout

A project-level parameter now controls whether or not jobs output intermediate results, with default on. Turning this off will save disk space, without having to manually clear intermediate results after the jobs complete.

In Particle Sets Tool and Exposure Sets Tool jobs, add note in event log indicating that result groups are taken from set A for the intersect dataset.

In Average Power Spectra job, print particle pixel size and power spectra sample spacing to the event log.

Manual Picker no longer fails when micrograph CTF input is not provided.

Resolved an issue where loading external data for the homepage modules would fail if a non-standard response was received.

Prevented a database error from occurring when no recent items existed for a user.

Fixed issue where copying to the clipboard would not work in some cases depending on how CryoSPARC was hosted over an internal network.

Priority field in the job queuing slide-over is now required and cannot be empty.

Resolved an issue where a previously selected entity was still selected after creating a new project.

Resolved an issue where clearing a job would not reset its queue priority.

Resolved an issue where completed jobs that were deleted displayed within project, workspace and session cards.

In 3D Classification, fixed KeyError with CTF fields when “Output results after every F-EM iteration” is enabled.

Fixed an issue where re-ordering classes and outputting intermediate results in 3D Classification conflicted and caused the job to fail. These options are now mutually exclusive.

In Global CTF Refinement, throw a warning if the spherical aberration refines to a nonpositive value.

In 3D Classification, add "Force re-do FSC split” parameter that forces re-splitting the particles into half-set splits from scratch. Disabling this allows for preserving the input splits, for particles from previous helical refinements, symmetry expansions, or local refinements.

Fixed a problem in 2D classification where many classes would be blank or very faint if multiple GPUs were used.

Fixed issues when running multiple jobs using the same SSD cache space.

Fixed 3DFSC crashing due to list index out of range error.

In 3D Classification, removed extraneous alignments3D entry in particles_all_classes dataset output which caused csparc2star.py to assign all particles to a single class.

CryoSPARC now fully supports CUDA 11.8, enabling support for NVIDIA Hopper and NVIDIA Ada Lovelace GPUs (NVIDIA H100, NVIDIA RTX 40 Series), including for 3DFlex.

The browse system has been updated with various incremental improvements including table header sorting, customizable view options, and upgraded CSV download functionality.

• A view options menu can be accessed from the browse system control bar allowing card and table details to be shown or hidden to suit user preference.
• The view options menu table section items can be dragged and dropped within the list to re-order the table view columns.
• The CSV download feature now includes an additional dialog allowing the user to select what information to include or exclude and the ability to re-order the table columns.

Within the details sidebar of a job, you can click on a particular cloned, parent, or child job to view it in the current tab. Command/ctrl click it to open that job in a new tab.

Within the ‘Metadata’ tab in the job preview dialog, there is a button to download the JSON data of a job to your clipboard.

Added a quick action to perform CTF Estimation via CTTFIND4 from a Patch Motion Correction job.

Introduced a workaround for a bug in CUDA 11.8 which formerly caused jobs relying on CUDA Fourier transforms (cuFFT) to run slower than they should (sometimes dramatically so).

When importing movies and micrographs, the ‘Skip header check’ parameter is now enabled by default.

Performance and stability improvements to all interactive job tables and the browse table in Live sessions.

Improvements to the context menu for job cards within the current jobs dialog.

Optimized current jobs tab within manage dialog and added a running timer for each job.

Fixed bug in 3D Classification that caused csparc2star.py to produce erroneous star files if class reordering was turned on.

If a job is running on a cluster, the available memory usage statistic is no longer displayed within the event log.

If specified, the cropped box size is correctly displayed on the card of a Volume Tools job.

Resolved cases where the outputs view mode toggle was triggered while attempting to perform a browser or operating system keyboard shortcut.

Removed variables ram_gb, num_cpu, and num_gpu from appearing as custom variables in a cluster configuration.

• The above variables are internal variables set by CryoSPARC and cannot be used/modified as custom variables
• Updated guide with a list of internal cluster variables and instructions for modifying resources requested in a cluster submission script

Fixed reading of .mrc.bz2 gain reference files in CryoSPARC Live.

Fixed issue on some OSes where building pycuda would fail during installation.

Fixed error message during job heartbeat failure to correctly specify heartbeat timeout.

Fixed “wrong generation counter” error during some long-running jobs.

Performance optimizations to the volume viewer in Live Sessions and the job preview dialog.

Fixed 2D Classification to allow manual overrides to Force max over poses/shifts parameter if classes is set to 20 or fewer.

Removed the parameter “Overall normalizer (override)” in particle extraction. The parameter had no effect.

Resolved an issue where clearing intermediate results for projects triggered multiple notifications.

Resolved an issue where viewing a job without selecting the card would select and pin it to the sidebar.

Resolved an issue where the current jobs tab within the manage dialog would crash the application in some situations when interactive jobs were queued or active.

Resolved a bug where it was not possible to define a threshold for the ‘Timestamp’ attribute of exposures in a Live Session.

Resolved an issue that caused an ‘Invalid path’ error when building Topaz or DeepEMhancer jobs.

Added a new Restack Particles job for consolidating particle files. More information available in the guide.

New 3D Classification parameters

• Added an option (default: on) to re-order classes by number of particles at the end of processing
• Added maximum F-EM iterations which will terminate classification even if convergence criteria are not met

In addition to CPU memory used, available CPU memory can now be displayed in the job event log. It is also displayed in the job log PDF.

Added options to sort projects by their size.

Added a new project-level action to delete detached projects from the CryoSPARC database. More information is available in the updated data management guide.

Exposure preview context menus in CryoSPARC Live now feature a button to reject/unreject the exposure.

Additional timestamp information added to job output log.

The job preview dialog project and workspace breadcrumbs have been updated to native browser links to allow them to be opened in a new window or tab.

Buttons with a corresponding keyboard shortcut will now show that shortcut in their tooltips.

The tree view now supports using the shift key to toggle the outputs view on all job cards.

Multi-select context menu improvements

• The job card action menu trigger button will now always display that job’s specific action menu even with multiple jobs selected
• Opening the action menu on a job card not included in a multi-selection will show that job’s specific action menu instead of the multi-select action menu
• The multi-select action menu can now be opened by right clicking the workspace background and not just jobs within the selection

In 3D classification, ensure solvent mask is generated from consensus volume if (only) a focus mask is supplied.

Fix a memory leak in 3D Classification related to plotting. With the fix applied, peak memory usage should be significantly reduced, with jobs that have a large number (50+) of classes seeing the most benefit (up to a 75% reduction).

Fix incorrect batch size during class ESS tuning in 3D Classification that required excessive amounts of GPU memory.

Errors reading TIFF files during import movies could cause abnormal termination.

Event log would flash and jump in a perpetual loading state in some cases.

Job description would automatically close while being edited in some cases.

Clearing intermediate results on jobs with empty output result groups would cause an error.

Jobs exported from this version may now be correctly imported in older versions of CryoSPARC.

Added a guard against creating redundant outputs in Extract From Micrographs (CPU).

Improved database startup reliability following updates.

Prevent wrong generation dataset error for some jobs that process a large amount of micrographs or particles.

Prevent Extract From Micrographs (GPU) job from using too much GPU memory, causing exposures to fail with cufftAllocFailed or cufftInternalError.

Fixed a bug affecting some OSes in the dependency installation for 3DFlex jobs (cryosparcw install-3dflex) that caused a gcc compilation error during the PyCUDA build.

Fixed a bug that caused 3DVA display to fail when using a subset of variability components.

Fixed a bug causing restarted jobs with GPU requirements to override the scheduler and jump to the front of the queue on the specified lane.

3D Flexible Refinement (3DFlex) (Beta) is a motion-based deep generative model for continuous heterogeneity. It can model non-rigid motion and flexibility of a protein molecule across its conformational landscape, and can use the motion model to combine signal from particle images in different conformations to improve refinement resolution in flexible regions.

• 3DFlex jointly learns a single canonical 3D density map of the particle, latent coordinates for each particle image, and a flow generator network. Given a latent position, the flow generator outputs a 3D deformation field mapping the canonical density to a particular conformation
• 3DFlex requires tuning of some parameters and optionally can take custom inputs defining geometry and topology of the target molecule
• Read an in-depth guide, including new example results using 3DFlex
• Watch a walkthrough video covering how to using the new algorithm in CryoSPARC
• Read the bioRxiv preprint from April 2021. An updated paper will be made available soon.

cryosparc-tools (Beta) is an open-source Python library that enables powerful scripting access to CryoSPARC.

• Programmatically read and write exposure, particle and volume data
• Easily perform advanced operations on metadata (alignments, CTF, etc) and programmatically insert modified data back into CryoSPARC
• Access project, workspace and job data
• Build and run jobs to orchestrate custom cryo-EM workflows
• Extend CryoSPARC functionality with third-party software packages
• cryosparc-tools is on GitHub and available via pip and can be used outside of the CryoSPARC environment in your own programs and tools
• Read the full documentation including example usage

Multi-select actions allow for cloning multiple jobs at once, cloning a chain of jobs between two selections, queuing, clearing or deleting multiple jobs at once, and more. Learn more about multi actions here.

Restart job action allows a job to be restarted from a killable or clearable state. This new action kills, clears, and queues the job to the same lane with the same parameters with which it was initially run.

Ability to configure custom variables for use in cluster submission scripts directly through the Cluster Configuration Tab in the Admin Panel. Custom variables can be configured at the instance level, target level and job level. See Guide: Configuring Custom Variables for Cluster Job Submission Scripts for more details.

Ability for admins to restrict users to view and run jobs on specific lanes using the Lane Restrictions Tab on the Admin Panel. See Guide: Lane Assignments and Restrictions for more details. By default, when a CryoSPARC user is created, they are assigned all existing lanes.

Multiple updates to Job: 3D Classification. Refer to the job guide and updated Tutorial: 3D Classification for details.

• Per-particle scale optimization is now built-in to the job
• New 'class flow matrix' diagram helps to visualize class changes across F-EM iterations
• New ‘Δ’ text column lists class size differences across F-EM iterations
• Updated aesthetics for class flow diagram to improve clarity of edges
• Unique volumes are no longer required for initialization

Job: Remove Duplicate Particles now allows for duplicate rejection based on 2D and 3D alignment error, in addition to NCC score.

Job: Symmetry Expansion now includes a parameter that enables splitting the output particle stack by the applied symmetry operator.

Job: Volume Alignment Tools now allows for application of arbitrary rotations to input volumes, particles, and masks.

Job: Select 2D Classes now exports an .mrc stack containing only selected classes, and an .mrc stack containing only rejected classes.

Job: 2D Classification now by default removes duplicate particles after classification based on a minimum separation distance between pick locations.

2D Classification jobs that perform binary class re-centering now show the binary masks used, generated from the specified threshold.

New project-level option to set the default path to a DeepEMhancer executable. If set, all new DeepEMhancer jobs will automatically be populated with the project-level path.

Job: Import Particle Stack now copies rlnAutopickFigureOfMerit into CryoSPARC's pick_stats/ncc_score upon import, if available.

Job: Global CTF Refinement now allows for initializing the spherical aberration to a custom value, prior to fitting.

Ability to prevent CryoSPARC from requesting external resources external HTTPS resources used to display information modules on the homepage by setting CRYOSPARC_DISABLE_EXTERNAL_REQUESTS environment variable to true. Details.

The CryoSPARC updater (cryosparcm update) has several enhancements, including:

• Ability to only download the master and worker update packages without doing a full update. Read more here
• Ability to skip downloading the master and worker update packages, and update using the previously-downloaded packages. Read more here
• The CryoSPARC instance is only shut down after confirming the master and worker update packages are downloaded and valid, reducing downtime during an update
• The output log of the update process is now written to cryosparc_master/run/update.log, and can be read any time using cryosparcm log update.

Ability for instances with the CRYOSPARC_DISABLE_IMPORT_ON_MASTER environment variable to queue jobs to the master node.

The signatures for the CLI functions delete_project and delete_workspace have changed. Please see the cryosparcm cli reference page on the CryoSPARC Guide for more information.

Project directories can now be renamed by using the edit button beside the "directory" field in the details section of the project sidebar.

Job: Reassign Particles to Micrographs will now copy the exposure group data from the micrographs to the particles if the particles.ctf slot and the exposure.ctf or exposure.mscope_params slots are connected.

Job: Import Movies info tag now displays the size of each exposure and number of frames.

When viewing jobs in a flattened view (e.g., all jobs in a project) the clone job action sub-menu now provides all workspaces as options to clone into.

Restrict some cryosparcm commands to owner UNIX account, including createuser, updateuser, resetpassword and patch.

Close open file handles during Import jobs.

User IDs are now visible in the output of cryosparcm listusers

job_type variable is now added to available cluster submission script variables provided by CryoSPARC. This value can be used for tracking which CryoSPARC job types are running on a cluster lane or for determining resource requirements for a cluster script.

qstat_cmd_tpl and qdel_cmd_tpl will have access to the same variables that are injected into send_cmd_tpl, including any user-configured custom variables.

Additional logging for master cache functions.

Interactive jobs now print out more informative errors in a scrollable, clearable log at the top of the interactive panel in the job dialog.

Resolved an issue where when viewing all parameters of a job within the comparison dialog incorrect custom parameter values were shown.

Jobs created via quick actions are no longer created in a different workspace or session than the one currently selected.

NCC score slider in CryoSPARC Live no longer loses all picks when set to 0.

Fixed ValueError in Manually Curate Exposures due to outliers during histogram creation.

Fixed bugs in Job: Gctf (Wrapper) that caused local CTF refinement to fail.

Fixed bug in Job: 3D Variability Analysis that prevented job from completing when number of components is set to 1.

The job dialog outputs tab now correctly displays the name of low-level results.

Fixed a bug that prevented correct download links for each output group of a job from being displayed within the sidebar details panel when a job is selected.

Fixed a bug that prevented multiple entities from being selected via the checkbox within the table view.

Scroll bars are no longer shown overtop of menus and dialogs in Safari.

Hostname no longer appears twice on instance management cards.

The sidebar action panel now collapses automatically after an option has been selected.

Fixed issue where a "Not Found" message would flash in the sidebar on load.

Fixed bug that resulted in incorrect cropping of some PNG figures in the event log.

Fixed bug in Job: Simulate Data that caused incorrect particles’ CTF parameters when “apply CTF” was disabled.

Job: Simulate Data no longer requires the SSD cache

Residual plots in Job: Global CTF Refinement now properly account for phase ambiguities of 2π.

Fixed KeyError in Inspect Particle Picks caused by micrographs that contain no particle picks.

Fixed bug in Job: Homogeneous Reconstruction Only with helical symmetry enforced and custom box size.

Fixed bug in CryoSPARC Live that caused failed exposures to be double-counted as queued exposures.

Fixed issue where retrieving CryoSPARC Live sessions via the CLI would fail.

Fixed issue where slow database command response times would cause problems when configuring the database (e.g., updating CryoSPARC).

Extensive Workflow now runs on a worker with 32GB of RAM.

Installation and update are now cancelled if Python dependency installation fails.

Prevent job heartbeat failures following multiple disconnects and reconnects.

Fixed bug that raised a JSONDecodeError when calling cryosparcm eventlog on Job: 3D Variability Display.

Fixed bug that caused command_core, command_rtp and command_vis logs to not be properly rotated.

Event logs for motion correction and extraction jobs now count movies/micrographs starting from one instead of starting from zero (e.g. “completed 20/20” instead of “completed 19/20”).

Fixed bug whereby ice thickness reported by Patch CTF (including in CryoSPARC Live) can be NaN.

Prevent file reading operations from failing on systems that incorrectly seek

Fixed bug that caused extracted particle count not to be displayed on the details panel when viewing a Live session

Prevent interface from becoming unresponsive when viewing the homepage in cases where the web application cannot load external resources.

Fixed issue where changelog grid styling was causing items to overlap on Safari.

Prevented an issue where available CryoSPARC updates would not display on the homepage and navigation bar.

When viewing jobs within a workspace there is an option to load all jobs at once rather than incrementally load as you scroll down the page.

New project-level option to set the default path to a Topaz executable. If set, all new Topaz jobs will automatically be populated with the specified project-level path.

Ability to use the spotlight to search for applicable jobs based on what project or workspace has been selected.

In Job: Exposure Group Utilities, when specifying a regular expression to split exposures or particles into exposure groups, all unmatched items will now be grouped together into a single, separate exposure group.

Added --offline and --skip_workers options to cryosparcm errorreport

• --offline skips retrieval of database statistics and worker instance information (intended for use when the CryoSPARC instance is not running)
• --skip_workers skips retrieval of worker instance information (intended for use when there are issues connecting to worker nodes)

The first applicable lane is now pre-selected when queueing a job from the sidebar.

systemd setup is now simplified, and correctly enables CryoSPARC environment during systemd process startup. Read the guide here.

Resolved inconsistency between displayed values for accepted and rejected exposures (and threshold included and excluded) in the Curate Exposures job with specific threshold ranges set now match job outputs in all cases.

Inspect Picks in v4+ now rejects outlier particles based on absolute value of power score rather than percentile (matching the behaviour of v3.3).

Local Motion correction no longer fails when some movies have zero particles or when some movies are corrupt.

Multiple targets within a single lane will now correctly display in the instance section of the manage dialog.

Volume viewer will now properly resize to the available space when adjusting sidebars and resizing the window.

Clicking checkboxes on the job table view will now correctly additively select them.

Event log downloads will no longer break in cases where ASCII characters are not included in the pdf font set.

Users can now log in with emails where the domain does not include a .

The page will no longer automatically refresh when downloading files from the output actions menus in Firefox.

When viewing a volume in Firefox, the selecting the up/down arrows within the threshold control number input field will retain focus in the same way other browsers do.

Prevent Live session view from crashing if the session was started after being marked as completed.

When making picks on the micrograph in the Manual Picker job, the picks column will correctly update in the corresponding table row.

When attaching a project, workspaces no longer have an incorrect “Created By” field, but show the user that attached the project.

Project title is now a required field when creating projects from the command line.

Clear Intermediate Results no longer deletes non-final results from some refinement jobs when those results were used downstream.

Improved error messages and error checking in Topaz Train, Extract and Cross Validation jobs.

Topaz Denoise job now correctly allocates CPU resources when run without GPUs.

Installation test (cryosparcm test install) no longer fails if only clusters are connected to the instance.

Starting a Live session that was previously marked as “completed” no longer incorrectly pauses the session twice.

Queue dialog now correctly applies titles and descriptions that were set.

Queue dialog now has a tooltip to clarify the toggle to show or hide lanes with applicable targets based on job resource requirements.

Manually Curate Exposures job no longer fails for infinite or NaN CTF parameters, outputs a warning instead.

Ab-initio jobs no longer break display of job cards when there are many classes.

Inspect Picks job no longer fails during CTF calibration if exposures do not contain enough particle pick locations.

3D Classification no longer fails when Output Diagnostic Plots at every Interation is set to False.

When using cryosparcm cluster connect, the final connected cluster configuration is printed as the output of the command (returning to v3.3 behaviour).

The range values for threshold sliders and input controls in the Curate Exposures job will now round display values to expected precision in all cases.

The reload browser shortcut will no longer manually reject selected exposure in the Curate Exposures job.

Table index cells in the Curate Exposures job now correctly update their colour when setting thresholds or manually rejecting/un-rejecting.

Filenames and other non-number values will now display correctly in the hover preview popovers in all interactive jobs.

Input fields in the Curate Exposures job can now be incremented using the arrow keys without changing the selected exposure in the micrographs table.

Over 10x speedup in job completion processes (check and passthrough outputs) for jobs with many output groups (e.g., 3D Classification with 20+ classes). Also speeds up “Mark as Completed” action for jobs.

Added more detailed logging message when manually taking over projects in an instance using cryosparcm cli "take_over_projects()"

CryoSPARC environment no longer uses LD_PRELOAD to load libtiff, resolving issues where dependencies of libtiff (libLerc and libstdc++) would not be found on some OSes (e.g., CentOS 7).

Job: DeepEMhancer (Wrapper) now accepts a mask input for DeepEMhancer’s binary mask normalization mode.

Job: Manually Curate Exposures now outputs a warning if not all input particles exist in the input micrographs.

Tensorflow test is now disabled by default during cryosparcm test workers. To enable, specify --test-tensorflow. Read the full installation test guide here.

The instance tab within the admin panel now loads primary system logs by default.

cryosparcm errorreport will now add the entire app, app_api, and app_legacy log files to the generated archive.

Updated the Windows shortcut for multi-selecting to use the ctrl key instead of the meta key.

Updated command_core request License-ID header check “Authentication failed” message with actionable suggestions.

Removed the cryosparcw executable path validation check by the master node when running cryosparcm cluster connect.

Fixed issue where running jobs could be marked as completed.

Prevent SSH and other command failures when system library versions don’t match embedded CryoSPARC libraries.

Project data management table now shows projects shared with the user.

The tree view no longer resets its zoom level when creating or modifying jobs.

Resolved an issue in the job builder where scrolling was locked when hovering over an input box.

Items with many tags now display correctly in the sidebar details panel and on the cards.

The browse view no longer auto-scrolls to a newly created job.

Resolved issue where job cards were not properly displaying images and info tags.

Allow clearing long-running notifications from Notifications panel.

Fixed cryosparcm fixdbport error.

All-new CryoSPARC interface designed for performance and productivity. Watch a video walkthrough of all the new features and read the comprehensive new guide.

• Redesigned interactive jobs (Inspect Picks, Exposure Curation, etc.) with significantly improved functionality and scalability. Easily handle large datasets (25K+ exposures)
  

  
• Integrated in-app 3D volume viewer to inspect volume outputs from all job types
  

  
• System for browsing and filtering projects, workspaces, sessions and jobs
  

  
• Tagging system to easily organize and record details about projects and jobs
  

  
• Multi-job selection and a new job comparison view
  

  
• Table view for projects, workspaces, sessions and jobs with CSV download support
  

  
• Two new ways to build jobs in addition to the drag-and-drop Job Builder:
  • Quick actions: right click a job and run common next steps (e.g., Blob Picker → Inspect Picks)
    

    
  • Job cart: select outputs directly from completed job cards to generate a filtered list of applicable jobs. Create and connect in a single click
    

    
• Download a PDF report of each job including key stats and the event log
  
  

  
  

  
• And much more! Be sure to watch the video and read the guide

CryoSPARC Live is now integrated as part of the CryoSPARC interface. Seamlessly transition from the primary interface into a Live session within the same browser tab: How to Access CryoSPARC Live.

Improved algorithm and diagnostics in 3D Classification (BETA).

• New tutorial covering multiple datasets and use cases
  
  

  
• FSC-based regularization to improve class quality and variety
• Two new convergence criteria (class switches, density change)
• Separate solvent and focus mask inputs
• Option for hard classification
• Better tuned and refactored default parameters
• Updated diagnostics, including:
  • class "flow"
    
    

    
  • focus mask overlay
    
    

    
  • difference-from-consensus
    
    

    

Wrapper for DeepEMhancer, a deep learning model for automatic masking-like and sharpening-like post-processing.

“Average Power Spectra” Job for averaging the power spectra of aligned, 2D-classified particles for helical symmetry exploration.

Example of averaged power spectra from 2D classes of the NLRP1-CARD filament (EMPIAR-10564)

Verify CryoSPARC installation integrity with cryosparcm test install and cryosparcm test workers. Read the Guide: Installation Testing with cryosparcm test.

“Maintenance Mode” setting which may be toggled on/off to prevent queued jobs from running while allowing running jobs to complete.

Administrator-configurable login message and instance banner to show notices to users.

Improved project data management system, simplifying common projects tasks such as transfer and archival.

• New comprehensive Data Management Guide for v4.0
• New project Attach and Detach actions to easily move projects between CryoSPARC instances
  
  

  
• New project Archive and Unarchive actions simplify moving projects to archival storage or changing a project location on disk
  

  
• For new projects, the generated project directory name is a hyphenated variant of the project title instead of PXX. A configurable prefix may be added to project directory names for easy identification
• Projects now use an instance-lock to prevent accidental corruption due to multiple CryoSPARC instances modifying the same project simultaneously
• Projects and Jobs no longer prefix output files with cryosparc_PXX_ on disk, since project numeric identifiers may change when detaching and re-attaching a project. The prefix is added when downloading files through the browser instead

Improved CryoSPARC database and API security with authenticated access control.

Error reporting tools to simplify troubleshooting.

• Download a job error report bundle from the Event Log tab of the Job Dialog
  

  
• Download a system error report bundle from the Instance Logs tab of the Admin Panel
  
  

  
• From the CLI, the cryosparcm errorreport function generates a system level error report bundle when the web-app is inaccessible

cryosparcm filterlog command filters logs by days, date, name, function, and level and adopts a more consistent logging format including timestamps at the beginning of logs lines.

cryosparcm cluster validate function validates cluster configurations during installation.

View improved database backup functionality with recent activity records. Available in Instance Information → Backups.

Job: Non-Uniform Refinement now supports optional Ewald Sphere Correction via an advanced parameter switch.

All jobs which process masks and volumes now accept masks sampled at different box and/or pixel sizes relative to the volume, which are internally resampled when appropriate.

Local Refinement and Particle Subtraction now issue a warning if binary masks are used.

Job: Extract from Micrographs now supports extracting and downsampling to two different box sizes via Fourier crop to provide smaller particles for early classification and reconstruction steps.

Job: Global CTF Refinement default parameters no longer fit fourth order aberrations (Spherical Aberration and Tetrafoil) unless specifically enabled.

Job: Local Motion Correction now includes an option to downsample particles via Fourier crop after correcting per-particle motion.

Job: Ab-initio Reconstruction now includes an unused particles output when not all particles were used for reconstruction in the single-class case.

Job: Gctf is now a “Legacy” job. Turn on “Show legacy jobs” in the job builder to unhide the job. Gctf also includes a CUDA libraries parameter to override CUDA version for compatibility with available Gctf binaries.

Job: 2D Classification may now run without CTF correction by turning off the Do CTF correction parameter, which may be useful for negative stain data.

Negative stain 2D class averages, with and without CTF correction, from the MTA-HDAC-MBD core complex (EMPIAR-10539)

Improvements to CryoSPARC Live interface:

• In the Live Overview tab, enter rejection thresholds with free-form numeric input
• Display particle picks in more distinguishable colours
• Volume viewer includes action buttons for resetting and zooming

Added Maximum number of local maxima to consider picking parameter to Blob Picker and Template Picker in CryoSPARC Live.

Improved cluster job status updates before a job is launched for better visibility.

Silent password input prompt in cryosparcm commands; will not be printed in the command output.

Added cryosparcm cli functions get_project_symlinks and get_job_symlinks to find moved symlinks in jobs. Read the guide for more details.

Reduced excessive verbiage in log files for certain job types that process particles.

Added CRYOSPARC_DB_CONNECTION_TIMEOUT_MS configurable environment variable for adjusting connection timeout during database startup.

Improved libcufft library initialization when only major version files (e.g., libcufft.so.10) are available.

Updated Extensive Workflow to issue warning rather than timeout error caused by long-running jobs.

Previous v3.3 web application (now running at BASE_PORT + 7) does not start up by default. To start it, run cryosparcm start app_legacy.

CryoSPARC Live (liveapp service) is integrated into the main web application interface running at BASE_PORT and is no longer available at BASE_PORT + 6.

Automatically update session data sizes when a session is paused or completed for improved data management.

Exposure Sets Tools no longer fails to intersect on path if mscope_params result slot is connected to both Exposures_A and Exposures_B input groups.

Heterogeneous Refinement jobs no longer fails when Force hard classification is turned on and some classes become empty.

Additional corrupt exposure error handling for Full-Frame Motion Correction, Patch Motion Correction and Patch CTF jobs.

Scheduler no longer fails for all queued jobs when a single queued job fails to run.

cryosparcm restore no longer fails when restoring against an empty database directory.

Show database connection error message in job log if one occurs during job startup.

cryosparcm restart no longer restarts all services if an incorrect service name is provided as an argument.

Long running background processes in one project no longer block actions across all projects in the system.

Fixed phase_shift attribute in CryoSPARC Live overview plot that was previously displayed as radians instead of degrees with an incorrect label.

FFTs on CPU no longer use a minimum of 8 threads that was causing contention on some systems.

New 3D Classification Beta job can be used to quickly classify particles into many 3D classes, without re-alignment of particle poses and shifts. Sort out heterogeneity from million+ particle datasets with 100+ classes. Read a tutorial here.

New ruler and scale bar allow for interactively determining the size of a particle in cryoSPARC Live.

New Blob Picker Tuner determines optimal blob picker parameters from a minimal set of user-selected manual picks. Tunes blob size, shape, and picker score thresholds automatically. Read a tutorial here.

A technique for Ewald Sphere Correction implemented in homogeneous refinement and reconstruction jobs allows for higher resolution reconstructions of larger proteins. Read a tutorial here. The new technique corrects for the Ewald Sphere during pose alignment and during back-projection, and also correctly estimates/refines higher-order CTF aberrations (HOA) while accounting for the Ewald Sphere.

Anisotropic magnification can now be estimated during CTF refinement, and is corrected for in Refinement, Reconstruction, 3D Classification, and 3D Variability jobs.

Substantial speed and memory improvements to FSC calculations in all job types, speeding up FSC calculation by over 10x, and providing an overall speedup of up to 20% for homogeneous refinements. A benchmark refinement of 100,000 Ribosome particles at 360 pixel box size (EMPIAR-10028) takes only 5.2 minutes on a single NVIDIA V100 GPU.

Substantially improved TIFF read performance for some filesystems. Learn more.

cryosparcm log provides an improved log format and new filter flags.

Import 3D Volumes now supports importing multiple volumes, including importing entries directly from EMDB.

FSC Plots are now accompanied by an EMDB-friendly XML file.

Improved performance in large projects. Faster Tree View updates and instantaneous input drag-and-drop connections.

New Volume Alignment Tools utility, supporting re-centering and symmetry alignment of 3D volumes.

Specify path of directory containing preprocessed micrographs for Topaz Train and Extract Patch 211214

New 3D Classification advanced parameter "Output data after every full iter" to enable data outputs (volumes, particles, volume series) after every full iteration Patch 211214

cryosparcm cli "update_parents_and_children_for_project(project_uid)" function to fix broken tree view input/output connections in previously-imported projects Patch 220215

Exposure/Particle Sets Tool option to ignore leading UIDs when comparing paths for intersection mode Patch 220215

Support for reading .mrc files containing float16 data (datatype code 12) Patch 220215

Option to remove all duplicates in Remove Duplicate Particles job Patch 220315

Pre-processing jobs no longer fail immediately when a child process encounters a segmentation fault Patch 220518

Patch Motion jobs now use an improved IO library, providing substantial speedups on some cluster file systems Patch 220518

Blob Picker Tuner job now reports progress during optimization Patch 220518

Validation (FSC) job now takes in an optional Override Pixel Size (Å) parameter to allow adjusting the final pixel size and resolution estimate if incorrect pixel size was specified during processing Patch 220518

Align 3D job now takes in particle stack(s) and rotates/shifts the alignments of the particles to match the rotation and shift applied to the volumes during 3D alignment Patch 220518

Project deletion (and other long running actions) are now decoupled from instance-wide locks, allowing a cryoSPARC instance to continue to perform unrelated actions while a deletion is ongoing Patch 220518

Patch Motion Correction now supports Fourier crop ratios of 1/8 and 1/16 Patch 220824

3D Variability Display (Cluster Mode) now has a volume series output Patch 220824

Exposure processing priority in a cryoSPARC Live session can now be modified. For more information, see: the Guide page on Session-level functions Patch 220824

Cache log now outputs average speed (MB/s) and estimated time to completion Patch 220824

cryosparcw call <cmd> enables execution of a command in a transient cryoSPARC worker environment Patch 220824

cryosparcm eventlog <project_uid> <job_uid> prints all text from a job's overview tab to stdout Patch 220824

3DVA Display job can now output particle sets corresponding to intermediate states.

Added option to select a subset of variability components in 3DVA Display job.

Volume Tools job now accepts a custom output filename. The default output name has also been updated to be more informative.

Display path of job working directory at the top of the event log.

Automatically delete SSD cache files that have not been used in more than 30 days during job caching steps.

Specify path to custom Certificate Authority bundle for external SSL requests.

CLI endpoint to safely update a project's directory (update_project_directory(project_uid, new_project_dir)).

Ability to use common path field (vs. UID) to intersect or split in Exposure Sets Tool and Particle Sets Tool.

CryoSPARC Live Data Management now supports executing a script upon a datatype's state change. Add export CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_ENABLE=true and export CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_PATH=/path/to/script.sh in cryosparc_master/config.sh to enable this feature.

Particle subtraction now respects refined higher-order aberrations.

Added FSC mask auto-tightening to the standalone Validation (FSC) job.

Extraction Box Size in Local Motion Correction now uses the input particle box size by default.

3D Classification volume outputs are now unmasked, fixing downstream refinements Patch 211214

New printout for each 3D Classification iteration to clearly specify O-EM/F-EM iterations left Patch 211214

3D Variability Display has more informative .mrc volume file names (simple/intermediates mode) Patch 220215

Improved error details when a cli method call fails Patch 220315

Improved logging in Extract From Micrographs (GPU), now shows number of particles rejected near edges of each micrograph Patch 220315

Improved EER read performance for some file systems Patch 220315

CryoSPARC Live Session Data Management: All datatype sizes are now updated when a session is “paused” or “completed”.

Helpful error messages when calling cryosparcm/cli functions with incorrect arguments.

Check for Corrupt Particles job now has an option to check for NaN values in the particle data files to test for data corruption on disk.

Extraction jobs now have an option to force re-extraction of CTF values for each particle from the input micrographs.

Outputs in the Sharpening Tools job are now separated into two output groups and can be directly connected to downstream Volume Tools jobs Patch 220518

Warnings in the Import Particles job's Overview tab are now more visible when fields are missing in the imported metadata files Patch 220518

Warnings in the Check For Corrupt Particles job's Overview tab are now more visible when corrupt files are found Patch 220518

The Local Refinement (NEW!) job now explicitly requires an input mask Patch 220518

Blob picker output images now show more contrast for negative stain data Patch 220518

Further improvements to error messages when problems are encountered while reading particles or micrographs Patch 220824

Volume series outputs in 3D Variability Display no longer require a zip system dependency Patch 220824

3D Variability Display (Cluster Mode) 1D histogram now includes a legend Patch 220824

Warning if all three dimensions of the input data for Import Templates match, which suggests that a cubic (3D) volume has been input instead of a stack of 2D templates Patch 220824

Summary of rejected particle counts and reasons for rejection in Extract From Micrographs jobs Patch 220824

Jobs now fail if they require more SSD cache space than the total available space or quota Patch 220824

If available, show MRC file name in job log when file cannot be read Patch 220824

Improved error formatting when running functions via cryosparcm cli Patch 220824

Fixed contrast issues in Inspect Picks, Manual Picker and Curate Exposures jobs when viewing a micrograph that wasn't motion-corrected in cryoSPARC.

Fixed issue in Volume Tools that caused ringing artefacts when downsampling masks.

Added resolution parameter to low-pass filter templates in Blob and Template Picker jobs.

Fixed libtiff error when running the worker on a read-only file system.

Corrected volume flip in cryoSPARC Live 3D volume web viewer.

Fixed a bug that caused stale notifications to persist after restarting cryoSPARC.

Fixed a bug where it was not possible to queue a job into a new workspace.

Fixed a bug where topics from the cryoSPARC Discussion Forum would not display on the dashboard.

Updated the job queuing interface to clarify which nodes certain jobs are launched on. Interactive jobs are always launched on the master node. Import jobs are launched on the master node unless CRYOSPARC_DISABLE_IMPORT_ON_MASTER is set to true, in which case import jobs can run on any lane/node selected. All other job types launch on a lane/node the user selects.

Fixed bug where symmetry alignment during refinement often aligned volumes incorrectly if they were already aligned.

Fixed bug where volumes are improperly aligned to symmetry axes in helical refinement.

Fixed an error during startup that occurs when an instance has less than 1000 notifications v3.3.1

Fixed plotting error during (legacy) local refinement Patch 211202

Use correct license check SSL certificate configuration when running cryoSPARC in insecure mode Patch 211214

Fixed Blob Tuner Error when Blob picker finds no particles Patch 211214

Passthrough GS split in 3D Classification Patch 211214

Check for empty splits in Homogenous Reconstruction job Patch 211214

Prevent crash of Rigid Motion Correction, Local Motion Correction and Extract Particles jobs when encountering corrupt exposures. The jobs now continue after any error in a single exposure, and output all incomplete exposures as a separate output Patch 220118

Set class field for 3D Classification particle outputs (fixes bug with downstream particles and csparc2star.py) Patch 220118

In 3D Classification, viewing direction distribution plots work with empty classes Patch 220118

In Topaz Extract, use provided preprocessed directory when no pre-trained model is specified Patch 220118

Importing projects retain job tree view input/output connections Patch 220215

Symmetry Search Utility no longer fails with KeyError Patch 220215

Topaz Train imported jobs no longer reference preprocessed micrographs directory from old project Patch 220215

Remove unused “Number of GPUs to parallelize” parameter from Local Motion Correction (non-multi) Patch 220215

Include 1st and 99th percentile particles in Inspect Picks threshold range Patch 220315

Use high-order aberration information in ResLog job Patch 220315

Local Motion does not fail when some movies do not include particles Patch 220315

Filter correct component reaction coordinates in 3D Variability Display job Patch 220315

Prevent error in Local Motion Correction when zero-shift frame is set Patch 220315

Improve multithreaded performance for some operations Patch 220315

Local motion no longer fails when zero shift frame is not set by previous motion correction job.

AttributeError raised in Filament Tracer job when only one particle pick is found on a given filament.

GPU FSC calculation in Refinement jobs raises AssertionError when used with box sizes under 84.

Topaz job now correctly fails when topaz subprocess fails.

Micrograph contrast in Manual Picker job is sometimes set to a value that clips the image.

Support for 3DVA Display clustering with one component.

Return HTTP 404 response when GridFS image requests are not found.

Allow clearing Extract Particles jobs created before the previous patch Patch 220518

Prevent Motion Correction job failure due to low-level memory error from corrupt TIFF files Patch 220518

Corrected stdout/stderr redirect options in cryosparcm cluster example scripts Patch 220518

Manual picker and Inspect picks jobs no longer overly downsample rectangular (K3) images Patch 220518

Job status timestamps are now correctly cleared when a job is cleared Patch 220518

Patch CTF job no longer fails with data from spherical aberration-corrected microscopes Patch 220518

Local Resolution Estimation job now reads the mask from the “Static Mask” input group Patch 220824

.mrc.bz2 files with an MRC extended header now load correctly Patch 220824

workspaces.json now correctly updated when a cryoSPARC Live session is running Patch 220824

Prevent SSD cache system failure when file size is missing in cache files database collection Patch 220824

Added option for sorting jobs by date and title. Projects and workspaces can be sorted by title as well

Check Particles job for verifying data integrity of particle stacks Patch 210511

Computational batch size parameter in Heterogeneous Refinement job Patch 210511

cryosparcm changeport command and related utilities for changing the base port Patch 210601

New utility cryosparcm licensestatus to verify if a license is valid Patch 210629

cryosparcm status now reports patch version Patch 210713

Set environment variable CRYOSPARC_SSD_CACHE_LIFETIME_DAYS in cryosparc_master/config.sh to auto-remove cache files older than the given number of days Patch 210803

New TIFF reading strategy for improved performance on network file systems such as GPFS and Lustre. To enable, add export CRYOSPARC_TIFF_IO_SHM=true to cryosparc_worker/config.sh Patch 210817

"Reassign Particles to Micrographs" job to reassign particles to their associated micrographs Patch 210831

Full Prefix/suffix cutting parameters for matching between input movies and micrographs in "Import Micrographs" job Patch 210831

Extend master/worker communication timeout with CRYOSPARC_CLIENT_TIMEOUT environment variable Patch 210831

Download EMDB-friendly XML files from FSC figures Patch 211012

Substantially improved performance in iterative particle-processing jobs (2D Classification, Refinement) when particles are stored on some types of filesystems (including FSx for Lustre), resolving cases where some users found that jobs run in v3.1 were slower than in v2.15

Deep Particle Picking jobs (Deep Picker Train, Deep Picker Inference) now require CUDA Toolkit 11+

The 'Metadata' tab within the job preview dialog is more performant and features expandable sections and colour highlighting based on data type

Import Particles no longer ignores the rlnImagePixelSize column while trying to import a particle .star file

CryoSPARC processes are now less likely to become orphaned due to more robust supervisor management configuration

Removed unused cuBLAS dependency that caused multiple GPU contexts on all GPUs in a system to be created even when using only a single GPU

GPU status info is now updated when a worker is connected for the first time, or when a worker connection is updated

New option to work around bug in CUDA on CentOS 7 that causes cuMemHostAlloc failed errors in multiple job types. To engage this, add export CRYOSPARC_NO_PAGELOCK=true to the cryosparc_worker/config.sh file.

Patch CTF Estimation no longer fails when encountering an error on a specific micrograph. Failed micrographs are outputted to a separate group if exceptions are encountered and processing continues.

Local Resolution job now prints local resolution statistics (minimum, median, maximum, 25th percentile, 75th percentile) in job event log Patch 210401

CryoSPARC Live Data Management now supports executing a script upon a datatype's state change. Add export CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_ENABLE=true and export CRYOSPARC_LIVE_DATA_MANAGEMENT_SCRIPT_PATH=/path/to/script.sh in cryosparc_master/config.sh to enable this feature Patch 210401

Issue in Patch Motion that causes dose weighting to be incorrectly applied in some cases. This was caused by the introduction of variable-dose support in v3.1.

Patch CTF Estimation fails with IndexError: index 0 is out of bounds for axis 0 with size 0

Patch CTF Estimation fails with ValueError: Axis limits cannot be NaN or Inf

Patch CTF Estimation sometimes reports underestimated CTF fit resolution in cryoSPARC Live, in cases where the correlation between the fit and the actual signal is very poor at low resolutions

Issue that can cause exposure groups set up at import to be lost during processing, specifically through wrapper jobs for GCTF or CTFFIND

Issue where Deep Picker Train job would fail reporting that there are no training particles on any micrographs

Topaz wrapper TypeError during negative stain processing: can't multiple sequence by non-int of type numpy.float32

Issue causing IndexError in wrapper for Motioncor2

Issue in particle simulator job where all particles would be modulated by the same CTF regardless of specified input CTF parameters

Issue in cryosparcm cli delete_user command that prevented an admin user from being authenticated

Issue in Extract From Micrographs (CPU) job requiring a GPU dependency, causing it to fail on CPU-only workstations

Import Job/Import Result Group fails with Unable to find data referenced by cs files when importing a mscope_params result group due to empty fields Patch 210401

Local Motion correction with EER data Patch 210413

Jobs no longer wait on cache forever after the cache is unlocked in other jobs Patch 210413

Restore missing "Use Class Weights" parameter in Deep Picker Train job Patch 210413

Prevent LD_PRELOAD warning when launching cluster jobs Patch 210413

ValueError during 3DFSC job volume output stage Patch 210511

Empty workspaces after importing project Patch 210511

Prevent error when extracting particles generated by Topaz Patch 210601

Prevent error in Deep Inference job during plotting Patch 210601

Prevent error caused by MotionCor2 wrapper outputting incorrect shape Patch 210601

Improved error messages when reading invalid exposures or particle stacks Patch 210601

Prevent numeric initialization error in Ab-Initio Reconstruction jobs for some systems Patch 210615

Prevent numeric error in Inspect Picks job Patch 210615

Statistic showing the total number of particles being processed by Ab-Initio Reconstruction job in cryoSPARC Live sometimes isn't updated Patch 210629

Prevent mrc_readmic (1) 447: Invalid argument errors that occur after extracting particles in a pipeline where micrographs were imported multiple times Patch 210629

Prevent I/O exceptions from terminating the cryoSPARC Live Worker while processing exposures Patch 210713

Allow shell operators during cluster job submission, enabling redirections required for LSF Clusters Patch 210713

Exception handling for Clear Intermediate Results Patch 210831

Corrected cryosparcm cluster example JSON output Patch 210831

Consistent exposure/particle matching in Deep Picker Patch 211012

Prevent startup error when system does not report CPU frequency Patch 211012

Prevent loss reporting error during Topaz jobs Patch 211012

Prevent create directory errors on some file systems Patch 211012

Non-Uniform regularization and adaptive pose marginalization

Alignment priors for soft penalization of unlikely poses

Reconstruction using higher-order CTF aberrations

Enforced point-group symmetry

Helical refinement now supports Non-Uniform regularization

Symmetry expansion now supports helical symmetry

In helical refinement, shifts along the helical axis can be limited to the asymmetric unit, improving resolutions for many datasets

"z-clip" parameter added to helical refinement's dynamic masking, to generate a mask covering only the central region along the z-axis

New Extract from Micrographs (CPU) job, supporting particle extraction on CPU in parallel using multiple cores.

Admin users can now modify the list of users to whom a project is shared

Major updates to SSD caching to avoid issues of Detected file change due to change in file size and improve performance

Several robustness improvements for Patch-CTF Estimation. Previous behaviour can be enabled using "Classic mode" parameter.

You can now quickly queue interactive jobs via the job details in the sidebar. The queue dialog has also been updated to clarify what machine the interactive jobs will run on.

Import Movies and Patch Motion now skip bad files instead of failing. Skipped files are added as a separate output.

Import Particles no longer ignores the data_optics section while trying to import a particle .star file. Exposure groups and CTF parameters other than high-order aberrations are retained from data_optics table.

Rebalance 2D Classes now has a "Override maximum superclass size" parameter that can be set to override the maximum size of all superclasses

When launching a job, hidden files (files that start with a '.') will be ignored when checking if the job directory is empty if the CRYOSPARC_IGNORE_HIDDEN_FILES environment variable exists (add to cryosparc_master/config.sh)

Homogeneous Reconstruct now has the option to reconstruct with helical symmetry, and to reconstruct with an inverted hand

Homogeneous Reconstruct now supports input alignments on any pixel size, regardless of the reconstruction pixel size

cryoSPARC Job Scheduler will now ignore CPU requirements when queuing jobs to non-cluster lanes; only RAM and GPU will be considered

Change the parameter "Remove leading UID in input micrograph path" in Import Particles to be optional

Discussion forum topics have returned to the dashboard

Some users were unable to create an account or reset their password in Chrome

Fixed a bug in Import Particles that caused it will fail when trying to convert rlnOriginX/rlnOriginY to rlnOriginXAngst/rlnOriginYAngst while importing Relion 3.1 star files

Fixed a bug that caused the cryoSPARC start process to not complete

Fixed a bug that caused the string of characters appended to the micrograph file paths created by cryoSPARC to be a variable length (now 21 characters)

Fixed a bug that caused the Import Project function to fail with _pickle.UnpicklingError: could not find MARK

Filament tracer now eliminates particles beyond the micrograph's edge

Fixed a bug that prevented the Rebalance 2D Classes job from launching on clusters

Fixed a bug in the Volume Tools job that gave erroneous output when the "Invert mask" and "Fill holes" options were both true

Fixed a plotting issue in Heterogeneous Refinement causing orientation plots to fail with a ValueError

Fixed issue that prevented pre-v3.0 Extract From Micrographs, 2D Classification, and Create Templates jobs from being cleared

Fixed issue causing static masks to be improperly resized in Homogeneous Refinement New, Non-Uniform Refinement New, and Helical Refinement

Fixed issue preventing Helical Refinement from running with static masks that were sampled on a different grid than the volume; fixed related bug with static masks being improperly rotated when the "Align input structure to helical (z) axis" parameter was activated

Fixed bug where running "Clear Intermediate Results" on a project caused the outputs of Curate Exposures jobs within the project to be inadvertently deleted.

Add support for pre-release versions of Topaz

Fixed issue where job queue modal would display incorrect default lane/target info when 'default' lane doesn't exist

Fixed issue where systemd service used wrong python interpreter

Fixed issue where deleted sessions still appear in cryoSPARC Live browse view

Fixed issue in cryosparcm updateuser when using lastname flag

Fixed IndexError in motioncor2 wrapper job

Fixed issue where Topaz denoise would not work

Clicking on the number of failed exposures in the navigation sidebar will open a dialog displaying the error details for each exposure

Fixed a case where a deleted session would display in the browse page when selected

Fixed a bug that caused manual picks on "Test" exposures to be excluded from session statistics and from being extracted

Job cards in the "current jobs" and "target" modals will now open their respective job modals when clicked. Job cards will also display the job's priority if one has been set

Fixed a bug in the cryoSPARC Live Worker that caused it to fail repeatedly due to uncaught exceptions caused by IO errors

Blob and template particle diameters are now correctly shown as Å values

Disable existing conda environments before activating cryoSPARC's conda environment

Specifying initial user credentials in install.sh fails with "improper usage" because of incorrect arguments

Command cryosparcm updateuser fails because of incorrect arguments

cURL symbol lookup error on CentOS8 when starting cryoSPARC after update to v3.0.0

Importing Projects from v2.15.0 or earlier into v3.0.0 fails with UnicodeDecodeError

Command cryosparcw newcuda uses old pyCUDA binary that doesn't exist

Deleting an Extract From Micrographs job fails with KeyError

Importing particle pose information in Import Particles fails with TypeError

Command cryosparcm backup fails if cryoSPARC is turned off

Inspect Picks job fails with ValueError if using micrographs imported with "Constant CTF" parameter

Symmetry Search Utility job fails with TypeError in "rise" mode

New and improved cryoSPARC Live app, redesigned and reworked for robust and seamless real-time end-to-end cryo-EM data processing. CryoSPARC Live is built for data collection facilities, cryo-EM cores, industry labs, microscope operators and individual users who want to perform real-time cryo-EM data quality assessment and decision-making based on streaming 2D and 3D results that update in time with new data. CryoSPARC Live is also now the best and fastest way for users to perform their first-cut processing and curation on already available data.

• Expedited and optimized preprocessing, enabling up to 600+ movies per hour per GPU (motion correction, CTF estimation, particle picking and extraction)
• The ability to seamlessly reprocess data at any time, to curate exposures, test/refine picking strategies and the choice of particles for refinement, on the fly
• Streaming 2D classification, in-line Ab-initio Reconstruction, and Streaming high-resolution 3D Refinement, which update automatically as new particles become available
• Seamless integration with cryoSPARC, allowing for direct advanced processing from the results generated in cryoSPARC Live without any data shuffling

After extensive beta testing globally, cryoSPARC Live is now available free of charge for general academic use. To learn more and to access Live, please see: guide.cryosparc.com

• Watch Now: CryoSPARC Live Walkthrough

Improved Non-Uniform Refinement algorithm NEW and job type Beta. Higher resolution results and reduced sensitivity to parameter changes. For more details, see the new paper published in Nature Methods

Plus:

• Support for local and global CTF refinement during non-uniform refinement
• Fast, new GPU implementation of non-uniform regularization developed with support from NVIDIA
• Additionally, support for automatic adaptive marginalization over alignment variables (enabled by default) yielding further improvements for small molecules.
  

  

Upgraded to modern Python 3.7 runtime, including new updated dependencies and installation. Easy upgrades and downgrades between cryoSPARC v2 and v3.

Support for CUDA 11 + Ampere Architecture GPUs (3080, 3090, A100, etc.).

• Please ensure you’re running the latest NVIDIA Driver compatible with your GPU and CUDA Toolkit version. You can download the latest driver for your GPUs here. Visit Troubleshooting for common GPU errors.

Full support for Thermo Fisher Scientific EER data in cryoSPARC and cryoSPARC Live. EER data is decompressed on the fly and can be upsampled and split into the desired number of fractions during processing.

Improved defect file and defect pixel support in motion correction jobs in cryoSPARC and cryoSPARC Live. Defect pixels can be specified in a .txt defect file or as zeros in the gain reference.

Helical Reconstruction BETA: New suite of tools for end-to-end helical processing and refinement, including:

• New high-resolution Helical Refinement job that also supports non-uniform regularization, and enforcement and local optimization of helical symmetry
  

  
• NEW Filament Tracer job tailored to the picking of filaments
  

  
• Updates to Inspect Particle Picks to allow pruning of filament picks by curvature
• New Symmetry search utility for exploring the symmetry space in a helical volume

Extract From Micrographs will now re-center particles by default if they have already been extracted and aligned through 2D classification, 3D reconstruction, or refinement.

New Homogeneous Reconstruction Only job type that performs 3D reconstruction from previously aligned particle images.

Ability to set priority values for jobs and users in cryoSPARC and cryoSPARC Live, allowing jobs to run in priority order instead of the order they were queued. Learn more about this feature with our Guide to Priority Job Queuing

CryoSPARC Live Session Data Management: View, delete and archive data created by cryoSPARC Live on a granular level. Save space taken up by data generated in old sessions and projects, while still being able to browse and view results.

A new Data Management tab is available in the Resource Manager in cryoSPARC, displaying details of all projects and workspaces available to the logged in user. At-a-glance information includes: project creation and ownership info, project directory location, last accessed/viewed, project size and other details.

Reduced GPU memory footprint in Patch Motion Correction. The overall GPU memory footprint of patch motion correction has been reduced. This enables most K3-sized image data to be processed on 11GB GPUs. Additionally, a "low-memory mode" switch is now available, which reduces it even further (at expense of speed).

New Remove Duplicate Particles job checks for particles that are extracted too close together on a micrograph and keeps only one, to allow combining picks from multiple pickers without affecting FSC estimates.

Beta Patch Motion Correction in cryoSPARC and cryoSPARC Live now optionally support movie data that has been saved with a variable dose rate among frames (e.g., from Leginon)

New Systemd Unit File to restart cryoSPARC after server shutdown: A systemd unit file is now included to allow running cryoSPARC as a service.

Improvements to Patch CTF estimation for better CTF fit in cases where background or envelope estimation is difficult. Improved robustness in relative ice thickness measure.

Downsample Particles and Extract from Micrographs will now correctly scale particle alignment shifts, fixing issues where downsample was used immediately prior to local refinement or 3D variability.

Orientation distribution and posterior precision plots were added to Heterogeneous Refinement, and can be disabled by turning off the "Show viewing orientation distribution plots" parameter.

Import Micrographs now has an option to output constant CTF values, so that subsequent steps will ignore the CTF entirely. Useful for negative stain and exotic data collection.

Improved cryosparcm patch pre-apply validation.

Reduced SSD-cache coordination bandwidth, improving the robustness of caching on multi-node systems.

Smarter cluster job ID parsing following cluster submission. Ability to use shell redirects in cluster submission commands, enabling native support for the LSF cluster management system.

Each project, workspace and job now records a 'last accessed' timestamp, updating whenever a user views the project, workspace or job through the web application interface. This allows for administrators to quickly identify and remove/archive data that is no longer used.

Queuing interactive jobs always default to the master node.

Ability to specify a first and last name for new users in the Admin panel and through the command line interface.

Default umask is now 002, to allow write permissions by 'group' for folders created by cryoSPARC.

Renamed internal installation directories from cryosparc2_* to cryosparc_*

Import Particles job fixes: Importing particle locations using micrographs created in v2.15+ failed because of a variable length UID prepended to the path of each micrograph; Using the "ignore raw data" parameter causes the job to fail with a missing pixel size error; Support for offsets specified in angstroms (_rlnOriginXAngst, _rlnOriginYAngst).

Import result group fails if some path fields are blank.

Inspect Picks "use_denoise" button failed to actually load denoised micrographs, even if they were connected.

Local Refinement now respects the static masking and minimization-over-per-particle-scale parameters, which were previously not used correctly.

Setting "Override number of iterations" to True without providing a number of iterations previously prevented Local Refinement from ever terminating.

Non-Uniform Refinement: Local processing causes an error if masks are too close to the edge of the refinement box.

Local Filtering had SSD:True hard-coded, causing it to fail when running on resources without SSDs.

Usernames are now correctly displayed in the details panel for a job.

First version of a new, comprehensive cryoSPARC guide! Available at guide.cryosparc.com Beta. This guide will consolidate all existing documentation and be populated with cryo-EM data processing tutorials, tips, descriptions of methods, case studies, etc.

Use the new cryosparcm patch command to install bug fixes and improvements without a full update

It is now possible to set a project-level default value for whether or not to cache particle images to the local SSD before running jobs within a project. The parameter can be adjusted from the sidebar when a project is selected

New Particle Class Probability Filter job: Filter particles based on the probability of matching their assigned 2D or 3D classes

In the Topaz Denoise job, the default values for the following parameters have been altered to better suit Topaz: The default value for the "shape of split micrographs" parameter has been changed to 1536. The default value for the "padding around each split micrograph" parameter has been changed to 256.

The Topaz Cross-Validation job has been updated with various improvements: Added option to output models from subsidiary training jobs when running cross-validation. Added expected number of particles and training radius as cross-validation parameters. Improved GPU distribution for subsidiary training jobs.

All Topaz jobs now store preprocessed micrographs to avoid unnecessary preprocessing in subsequent Topaz jobs.

Added option for the Extensive Workflow to run additional job types, including Local Motion Correction, Multi-class ab-initio and 3D Variability. This can be used as a full system test for new installs and updates.

Use dynamically-generated SSD cache paths by including environment variables in the CRYOSPARC_SSD_PATH definition for a worker's config.sh

Import Movies and Import Micrographs jobs are now parallelizable across multiple threads while reading the headers of all exposures to import

Initial and final mini-batch size parameters for Ab-initio Reconstruction can now be set to zero to automatically tune based on the estimated particle size

The project and workspace switchers, which allow you to quickly search and navigate, have been revamped. When you're viewing a project or workspace, the switcher will highlight and scroll to it, allowing you to easily navigate relative to them. When viewing a job within a workspace, the workspace switcher will display a link icon next to every workspace that particular job is linked to.

The job history tab within the resource manager page now shows only jobs that have not been deleted by default. Select the 'show deleted' toggle to show only jobs that have been deleted.

Jobs can now only be imported from inside the project directory. This fixes a bug that causes particles to not be cached if they were imported from outside the project.

Executing a cryosparcm command is now only allowed by the owner of the script and on the master node. You can override this parameter by setting CRYOSPARC_FORCE_USER or CRYOSPARC_FORCE_HOSTNAME to true in your shell (or permanently inside cryosparc2_master/config.sh).

Bug where multiple jobs launched simultaneously using the same particles could result in endless "cache waiting" messages

Full-frame Motion Correction runs out of memory on 8GB cards with K3 data. There is now a switch to reduce memory usage by approximately half

ValueError was raised in 2D classification or Patch CTF Estimation jobs when movies that were imported separately but contain files with the same names get overwritten

All refinement jobs now use the first (rather than last) FSC threshold crossing for filtering during iterations

Bug in Particle Set Tools intersection when set A is smaller than set B

Bug in Heterogeneous Refinement which fails if intermediate plots are disabled

Manual Picker fails while calculating the contrast intensity range for a micrograph with an irregular shape

Topaz Denoised micrographs display fail to display correctly in the Manual Picker job (as either gray or black images)

Micrographs captured at a low magnification fail during Patch CTF Estimation with an IndexError

Select 2D fails when trying to select classes from a particle .mrc file that is too large to fit in memory. Also fixed a bug where an IndexError is raised if a class has less than 3 particles in it.

Exposure Group Utility job fails with a KeyError when an input dataset has a missing CTF result field

Inspect Picks job fails if you try to extract locations and the box size set is not an even number

Downsample Particles now exports particles in the correct order if the input particles are out of order on disk

MotionCor2 Wrapper: total dose is incorrectly specified as the frame dose argument for dose weighting

MotionCor2 Wrapper: output pixel size is incorrect after specifying a Fourier crop factor, causing CTF Estimates to fail

Project directory sizes don't update when a project is deleted

Allow importing movies shot on high-aspect ratio detectors (2:1 aspect ratio or higher)

Resolved version incompatibilities with Topaz version 0.2.4a

The job builder includes a search bar to quickly find the job you're looking for

"Cache Particles on SSD" Job that will cache particles on a node of your choice

Many optimizations across cryoSPARC jobs - datasets containing a large number of items will see significant speedups

Added option to use pre-trained initializations in Topaz Train

Added option to use pre-trained model when running Topaz Extract

All jobs that output FSC plots have been updated to allow for downloading the raw data for external plotting

Curate Exposures job can be run without connecting micrographs, allowing for CTF-based curation using only movies

Topaz Denoise fails when trained model is input

Template Picker fails on K3 micrographs motion corrected using external software

Turning on the "Generate new FSC Mask" parameter in the Sharpening Tools job caused an AssertionError

"Particles Excluded" output group image in Select2D job was the same as the "Particles Included" output group image

New Homogeneous Refinement job sometimes fails when resampling initial model because boxes were not even sized

Performance regressions in the "Extract from Micrographs" job where the time to extract particles increased non-linearly with a larger number of exposures

In 2.14.2: "Inspect Particle Picks" job fails with a KeyError when it encounters an exposure with no particle locations

The 3D Variability Display job output now includes an in-browser, in-line interactive 3D scatter plot to analyze clusters in the distribution of particle reaction coordinates! The user can select which three reaction coordinates to display, and individual clusters can be shown or hidden. Read the new full tutorial, including examples of results:

• Tutorial 3D Variability Analysis Tutorial: Part Two

When inspecting particle picks from template or blob pickers, particle pick scores (NCC) and power scores are automatically calibrated with respect to the defocus of each micrograph. This ensures that setting an appropriate threshold on one micrograph will yield good picks on other micrographs with very different defocus.

• Tutorial Particle Picking Calibration

Negative stain and phase-plate datasets are now more easy to process. A single toggle can be set during import and no other parameter changes are necessary in subsequent job types.

• Tutorial Negative Stain and Phase Plate Data Workflows in cryoSPARC

The blob picker can now use "ring" shaped blobs, which perform well in picking viruses and other spherical particles (eg. apoferritin).

Relion 3.1 format .star files can now be imported without errors, though optics group data is not currently parsed.

Manual picker interactive job now has adaptive robust contrast normalization to make seeing particles in imperfect images easier.

Manual Picker interactive job now includes a "Contrast Intensity Override" slider that allows users to manually select the contrast bounds when viewing a micrograph. The average, patch-based intensity is used by default for the Manual Picker and Exposure Curation jobs.

Validation (FSC) Job: Validation of a refinement result (i.e. half-maps) using the FSC is now available as a separate job type, with optional mask input.

Fourier-space plots in Homogeneous Refinement display correctly.

Multiple updates to Topaz-wrapped jobs in cryoSPARC:

• Added multiprocessing to substantially improve preprocessing times in all Topaz jobs
• Added parameter to select number of CPUs in the Topaz Extract job
• Topaz jobs now automatically account for negative stain data
• Topaz-extracted particles no longer cause 2D classification jobs to occasionally fail due to inconsistent CTF outputs

Passthrough output files now include the project and job numbers for easier referencing (e.g., P1_J1_passthrough.cs)

Manual Picker job now fails by default if the CTF inputs are not connected. An override switch has been added to allow manual picking without CTF estimates, which is not the recommended workflow.

3D Variability Display can now flip the handedness of output volume series.

CPU and GPU memory usage of Homogeneous Refinement is substantially reduced. In tests, it is now possible to refine a 1024 box size on a 11GB consumer-grade NVIDIA GPU (GTX 1080Ti), though this does require ~160GB of CPU RAM. A smaller 256 box size can run on a 4GB GPU and requires only 6GB of CPU RAM.

The template picker now uses substantially less GPU memory for large micrographs (e.g., K3 super-resolution data).

Local per-particle motion correction of K3 data now works correctly.

Bug that would cause cufftInvalidPlan error during refinement.

Bug where two Select 2D jobs are jobs connected to each other, the last Select 2D job shows incorrect class average thumbnails.

Bug where the command_vis would repeatedly try to start but fails, using 100% CPU in the process.

(Added in v2.13.2) Bug where mrc volumes were being saved as float64 files, which the mrc file format doesn't support, causing the Sharpening Tools job to fail

(Added in v2.13.2) Bug where Heterogenous Refinement and Multi-Class Ab-Initio jobs would fail due to a ValueError

New "cluster" mode in 3D Variability Display allows for automatic hierarchical 3D classification based on variability coordinates: directly visualize and see how many discrete classes are present in a dataset, separate them by clustering, and reconstruct each separately. Then recursively perform 3D Variability to look for sub-conformations or flexibility within each cluster, or refine each cluster separately.

New "intermediates" mode in 3D Variability Display allows for weighted reconstructions of intermediate positions along each variability dimension: create more detailed videos of variability that minimize the effect of density appearing and disappearing during large motions. Also output weighted particle subsets for intermediate positions, for refinement of particular flexible conformations.

Default options in 3D Variability improve sensitivity to small variations in the 3D density. Plus support for coloured or white noise models, high-pass filter during optimization, per-particle scale optimization.

• Tutorial 3D Variability Analysis: Part One

Fixed streaking issue so that (many) more variability modes can be solved simultaneously.

Per-particle defocus refinement implemented on GPU allows CTF refinement in minutes for 100,000's of particles. Implemented as a standalone job and also as an in-line on-the-fly option during Homogeneous Refinement.

• Tutorial On-the-fly CTF refinement in homogeneous refinement

Global CTF Refinement and Local CTF Refinement jobs: CTF aberration support, also GPU implemented, corrects for beam tilt, trefoil, spherical aberration, and tetrafoil. Estimation of the new parameters can be done in a standalone job or as an in-line on-the-fly option during Homogeneous Refinement, yielding ultra-high resolution reconstructions.

• Tutorial Local CTF Refinement
• Tutorial Global CTF Refinement

New Exposure Group Utilities job allows users to split or combine datasets into multiple exposure groups for higher-order aberration correction. Create exposure groups using regular expressions and separators, and test your exposure group creation strategy.

• Tutorial Exposure Groups

First-pass optimization of GPU and CPU code for Homogeneous Refinement: now up to 5x faster, especially for symmetric particles.

New admin page with ability to change roles of existing users and create new users through the UI

• Tutorial Create new users through the UI
• Tutorial Manage existing users through the UI

Users can request to reset their password through the interface

• Tutorial Resetting a password through the UI

New Topaz wrapper jobs for integrating Topaz's particle picking and micrograph denoising functionality

• Topaz Train, Topaz Cross Validation, and Topaz Extract jobs perform particle picking while the Topaz Denoise job performs micrograph denoising
• Users only need to install Topaz to incorporate it within the cryoSPARC pipeline
• Denoised micrographs are compatible with Manual Picker, Inspect Particle Picks, and Topaz's particle picking pipeline, assisting users and machine learning models in observing micrographs
• Tutorial Topaz integration in cryoSPARC

Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153–1160 (2019) doi:10.1038/s41592-019-0575-8

New Symmetry Expansion job expands particle poses around point-group symmetries, to support local refinement of subunits, as well as 3D variability of symmetric particles.

New Job Scheduler and Queuing Interface: Override the cryoSPARC scheduler to queue directly to a specific GPU

• Tutorial Queueing Directly To A GPU

Ability to perform non-interactive threshold selection for particle picking and extraction, and non interactive selection of 2D classes, making it possible to process simple data completely end-to-end without manual intervention

Ability to interactively select all classes above or below a threshold in Select 2D job (right-click a class)

Multiprocessing robustness updates in preprocessing jobs (Patch Motion, Full Frame Motion Correction, Patch CTF Estimation) ensure that if a job fails or is interrupted, completed results are recovered and processing can be resumed.

Orientation-precision plots in refinement are now coloured with minimum value zero so that differences in ratios can be easily spotted

Relative ice thickness is now available in Exposure Curation

2D Classification allows for changing the maximum alignment resolution and reconstruction resolution separately if desired

Sharpening job now applied a spherical window by default

Added option to skip header check when importing movies, which allows for faster importing on larger datasets

Importing Movies, Micrographs or Particles will auto-populate the "exposure group ID" field of the dataset based on the project (each project increments from 0)

Set the environment variable CRYOSPARC_DISABLE_IMPORT_ON_MASTER to allow master-direct jobs (import jobs, curate exposures, select 2D) to be queued on any machine you specify, and not only the master node

If a job is queued to a specific GPU, it will be noted in the Resource Manager

Job Cards now show queue message if placed in queue

Job type is more prominently shown in the sidebar

Ab-initio job with symmetry enforced fails with invalid memory access error

Certain parameter descriptions in the Job Builder don't show

Better logging in Align 3D Job

Error in non-uniform refinement when run with CPU option

3DFSC job doesn't accept mask inputs

New ignore_pose option in import particles to enable import of locations when pose information is missing

Deadlock in new job types like 3D var on some newer GPUs

Select 2D fails when all classes are selected (i.e., if there are no particles in the templates_excluded output result group)

White circles in 2D Class job when circular mask diameter parameter is too close to particle box size

Exposure Sets Tool Job doesn't recognize particle inputs

The 'more' button in the Job History page doesn't load new rows

Copy to clipboard functionality is more reliable on various browsers

Back button in File Browser correctly expands path variables

Imported jobs caused the tree layout to fail or throw an error

Project directory sizes are calculated more accurately

When updating a worker's SSD configuration, the SSD option remained disabled even though user supplied a path to the SSD