Projects, Workspaces and Jobs
Projects are the high level container housing one or more Workspaces, and all the associated Jobs. A Workspace is the place where new Jobs are created, and you can use one or more Workspaces to separate out different portions of a Project, as appropriate.
Best practices for organizing Projects and Workspaces
A Project might be used for a particular dataset, e.g., the T20S proteasome. Within that Project, you might create one Workspace where you Import raw movies and perform pre-processing steps. In another Workspace within the T20S Project, you might perform multiple rounds of 2D classification, and once satisfied with the subset of particles, you might use a third Workspace to perform ab-initio reconstruction followed by refinement. Using different Workspaces is especially useful when you are iterating many times through a workflow and want to create branches or sub-groups of experiments to more easily keep track. You can have any combination of jobs and workspaces within a project, and can import new data multiple times within a project if collecting data in multiple sessions. All these steps can also be automated using the command line interface to cryoSPARC.
For simple projects, you can also run all Jobs in one Workspace if you think your processing will be fairly simple or linear.
Creating a Project
- Navigate to the Projects view by clicking on the drawer icon on the header, or from the Projects button in the footer.
- To create a project, you can either press
Non your keyboard, or click on the "+" on the header, which will bring up a modal window for the New Project details.
- Enter a project Title and select a location for the associated project directory from the File Browser. All files associated with the project will be stored inside the selected project directory. You may also wish to enter a Description for your project. The project directory should exist and be readable and writable on all master and worker nodes attached to cryoSPARC.
- Your new project now appears on the Projects page.
Open up a Project
To open a Project, you can click on the Px button on the top left hand side of a Project card:
Alternatively, you can open a Project by looking for it in the drop-down menu by searching or scrolling, and then clicking on the Project:
Inside the Project you might find a number of Workspaces, e.g.:
Delete a Project
While selected, you can delete a project from the 'Details' tab. A pop-up message will ask you to confirm the delete. Once you confirm, another popup will show you a list of jobs and workspaces that will also be deleted and ask you to confirm again. Deleted jobs are first cleared before deleting, meaning that their intermediate and final results are erased from disk. Note that unlike deleting jobs, deleting a project doesn't remove the project folder from the disk.
Create a new Workspace
Workspaces reside within Projects. Once you are inside your selected Project, create a New Workspace using the "+" button on the header, or, press
N on your keyboard.
Open up a Workspace
You can open a Workspace either by clicking on the Px - Wx button for a particular Workspace (pictured immediately above), or you can locate it using the Workspace search bar in the header. A Workspace view might look like this (in Tile view):
Delete a Workspace
While selected, you can delete a project from the 'Details' tab. A pop-up message will ask you to confirm the delete. Once you confirm, another popup will show you two lists: A. jobs only inside the selected workspace that will be fully deleted, and B. jobs that exist in the selected workspaces, and other workspaces. Jobs that exist in multiple workspaces (list B) will not be fully deleted. They will simply be disassociated from the workspace to be deleted. Note that deleted jobs are first cleared before deleting, meaning that their intermediate and final results are erased from disk.
You can view and edit Project and Workspace details (e.g., descriptions, notes), and view recent jobs, from the Details tab on the right sidebar. It can be helpful to set a title and description to indicate your intent or a summary of your results in the workspace, especially to make it easy to understand a workflow when looking back months later.
Card View and Tree View
CryoSPARC v2 allows you to view and interact with jobs using Tile View (pictured above, in 'Open up a Workspace') or Tree View. Tree View shows you a detailed flowchart of all the Jobs inside one Project (without regard to which Workspace they are in) or inside one Workspace.
Click on the flowchart icon from the header to open the Tree View associated with a particular Project or Workspace:
The Tree View displays all jobs and their connections. Click and drag to move the tree around. Hold
ALT and scroll to zoom in and out of the tree. You can also scroll up and down or left and right using a trackpad or mouse wheel.
As pictured above, all jobs are connected with each other, except for jobs in 'Building' status (purple) that have not been connected to anything, which will remain beside the tree on the right side until they are connected by dragging inputs (pictured above).
Clicking on a particular job card in Tree View will highlight its parent and children jobs, and fade out other jobs in tree, until you double-click elsewhere on the workspace, or click
ESC to de-select the job:
You can interact with job cards in Tree View and Tile View in exactly the same manner:
- Click on the job number to open the job card (inspect view). Alternatively, press the
SPACEBARwhile the job of interest is selected. Exit inspect view by clicking the
xor pressing the
- Toggle between active and inactive Building states by pressing
bwhile a Building job is selected, or clicking on the Building button on the job card
- Open the Job Builder to drag and drop inputs from job cards (described below)
Quick Navigation and Keyboard Shortcuts
- Create new Project or Workspace: To create a new workspace or project, you can press
N. This will bring up the workspace/project creation screen.
- Open and inspect a job: When a job is selected, you can press
SPACEBARto open the inspection panel. Press it again to close. Alternatively, you can click the job number on the top left hand corner of the job tile/card to open the job, and use the
xon the top right of the inspect view, to close it.
- Edit the parameters of a Building job: When a job is in "Building" mode, you can press
Bto open up the interactive job builder and edit its parameters. This works with recently cloned jobs as well.
- De-select a job: To de-select a job, press
ESC. If you are in tree view, this will show all connections between jobs again.
- Zoom into tree view: Hold
ALTand scroll to zoom in and out of the tree view
Use the Job Builder on the right sidebar to create and run new jobs. Navigate to the Job Builder by clicking the pickaxe icon on the top. Select the job you wish to run, e.g., Import Movies. The Job Builder populates with specific input(s) and parameters for that job type:
Drag and drop to connect jobs
Inputs for a job can come from the workspace in which you are actively working, or any other workspace within your current Project. Connect the required input(s) by dragging and dropping the output(s) from another job, like below.
If you have successfully dropped the output(s), you will see the job number appear below the input, as well as an option to Remove if you have accidentally connected the wrong output:
Default and spec parameters
CryoSPARC v2 populates default parameters for most jobs where necessary. To show additional parameters, toggle to Advanced Mode at the top right hand corner of the Job Builder:
All default parameters are indicated with a blue D. If you modify a parameter, it will be indicated with a green S. To restore the default value, click on the x to the right of the parameter you wish to restore.
Queueing and running jobs
Once satisfied that you have successfully dragged the required input(s) and modified any applicable parameters, click Queue to launch the job. A modal will appear where you can select a Lane on which to run the job. By default, new jobs you create will run in the current active workspace. If you wish to run the job and link it into a different workspace, you can also specify this under Run Job in.
Open/Inspect a Job and View the Event Log
When a job is selected, you can press
SPACEBAR to open the inspection panel. Press it again to close. Alternatively, you can click the job number on the top left hand corner of the job tile/card to open the job, and use the
x on the top right of the inspect view, to close it.
To navigate easily through the stream log without having to scroll, select a checkpoint from the top, or use the filters. Clicking on "Follow latest" will cause the streamlog to tail the end of the job's output as it runs in real time.
Recall/review which parameters you modified for a previous job
To quickly view which parameters you adjusted for a given job, open the job card and click on the 'Inputs and Parameters' tab. Parameters which were unchanged from their default values display with a blue D. Parameters you specified are outlined in green and display with a green S.
You can easily determine the status of jobs within workspace view looking at the coloured dots on the job card. Purple = building, blue = running, and green = completed. Hovering over a job's status circle will indicate the status as a tool tip.
An alternate way to examine job status, across multiple projects, is to glance at the Resource Manager. Navigate here by clicking on the Resource Manager button on the footer.
The Job Queue provides information on all jobs currently queued in each lane. Jobs can be queued for various reasons, that are displayed for each card.
The Active Jobs tab shows jobs that are running or waiting in each lane, and jobs can be killed from this view. You can find compute configuration details for workers in each lane, in the Compute Configuration tab.
You can also Kill jobs from the Queue:
For more on killing, cloning and linking jobs, see section 2.14 below.
Queue up jobs to run automatically
In cryoSPARC v2, it is not necessary to wait until a job has completed before setting up the next step in the chain of experiments. You can queue up a series of jobs manually or automatically, which, once queued, will commence as soon as the outputs become available from earlier jobs.
Automated Basic Workflow
This workflow is especially handy when processing a new set of movies for the first time. In the Job Builder, select Basic Workflow. Configure the job by connecting imported movies as input, and setting any parameters needed. When the Basic Workflow job is run, it will create other jobs, connecting them together, and queuing them. This pre-generated series of jobs will proceed through a series of processing steps commonly applied to a new dataset. Note: You must start the Basic Workflow by launching the first job to Import Movies, however, it will proceed on its own through the rest of the steps, waiting only where necessary for user input (e.g., Select 2D Classes).
Create your own Workflow
You can create your own workflow by launching your first job, e.g, Import Movies. While the job is still running, open the job card, and immediately drag and drop the Output to a new job in the Job Builder (e.g., Motion correction). Queue up the next job and it will appear in "Queued" state in the workspace view. You can chain up a number of jobs in this way to have them run autonomously.
You can easily see how jobs relate to each other by clicking on the job of interest (e.g., J411). The parent(s) of the selected job, J410 (i.e., the job which provided inputs) are highlighted with a surrounding dashed red line, while the child J415 (i.e., the job which used inputs from J411) is highlighted with a surrounding dashed green line. Other un-related jobs fade away until you click somewhere else on the workspace canvas.
Kill, Delete and Clone Jobs
Kill a Job
While running, you can navigate to the 'Details' tab and click 'Kill Job'. A pop-up message will allow you to confirm. Killed jobs will display as orange and will remain part of the workspace unless you delete them. Killed jobs will not have their intermediate results expunged until they are cleared.
Clear a job
When a job has been killed, has failed, or has been completed, it can be cleared, meaning that its results and outputs are all erased from the file system, but its connections to other jobs and parameters are retained. This makes it easy to correct a mistakely set parameter, or to re-run a job. Clearing also removes queued jobs from the queue and resets jobs to Building status.
Delete a Job
While selected, you can delete a job from the 'Details' tab. A pop-up message will ask you to confirm the delete. Once deleted, the job will disappear from the workspace (except in the Tree view, if it has children that need to be displayed). Note that job numbers within a workspace are unique, so a new job in the same workspace will not be assigned the same number as a previously run or previously deleted job. Deleted jobs are first cleared before deleting, meaning that their intermediate and final results are erased from disk.
Clone a Job
Cloning is particularly useful when you wish to process subsequent jobs in a new workspace for better organization of your experiment, or if you wish to quickly replicate a job (to try another setting of parameters) without dragging and dropping inputs into the Job Builder. Select the job you wish to clone, and from the 'Details' tab, click 'Clone job'. By default, the job will be cloned in the current workspace. Click 'Queue' to launch the job. A modal will appear asking whether you wish to run the job in the current workspace or in a new workspace, and you can select as appropriate. Note that you can always edit the parameters of a new cloned job before Queuing. You can also change inputs by removing connected inputs and dragging in new inputs.
Moving and Linking Jobs
Moving a Job
When re-organizing your workspaces, you may find it necessary to move a job from one workspace to another. To do so, select the job and navigate to the 'Details' tab. Here, you can press the Move Job dropdown to select which workspace you would like to move the job to. The job will disappear from its original workspace and move to the new one.
Linking a Job
Occasionally, you may want to have the same job appear in multiple workspaces at the same time. This may be useful when you have a single job that will have many children, for example when you are refining different classes of the output of a multi-class Ab-Initio refinement. In this case, you can use the Link Job dropdown in the 'Details' tab to pick another workspace you would like the job to appear in. Any changes you make to the job in one workspace will be reflected in the other, as both workspaces will be displaying the exact same job. If a linked job is deleted from any workspace, all other instances of it in other workspaces will also be deleted. You can also unlink a job from a workspace if needed.
Download Job Outputs
Download outputs from the right hand side of the streamlog. Hovering over an output group (shown as a single draggable tile) will display a list of the available outputs within that group. Click the down arrow to select an output from the drop-down and download files directly. This is especially helpful for downloading refined volumes for inspection using tools like Chimera. Note that downloads happen over the network from your cryoSPARC master instance over to the local machine/laptop where you are browsing the interface. Downloaded results are always stored on the server so you can freely delete them from your local machine without losing them.