Having transparent, controllable cloud costs is a top priority for Terra. In response to feedback from our community, we have been working on three key initiatives to improve cost management on Terra:
- Cost Reporting: Tools to provide more clear and intuitive reports of cloud expenditures
- Cost Controls and Estimates: Tools to help prevent runaway workflow costs
- Cost Optimizations: Tools to reduce cloud spend
Overview of Cloud Costs in Terra
When you are working with data in Google Cloud, you are paying to store and analyze your data. Running a workflow or a Jupyter notebook all incur charges through your Google Cloud billing account. More details can be found on Terra’s support site around Managing Cloud Costs.
Initiative 1. Cost Reporting
Reminder: Set up spend reporting
We encourage all billing project owners to set up spend reporting for any new billing account. By completing the setup, you ensure that your billing data is captured and accessible in Terra, giving you greater control over your cloud expenses. Once set up, you can access reports for financial oversight and optimization of your cloud costs and resources.
Improved spend reports
Workspace owners now have a consolidated spend report to see costs for all of their workspaces across billing projects to identify what workspaces are costing them the most money. Billing project owners can also dive into the details of the workspaces in a billing project and see costs over time, aggregated daily to observe how costs are trending. For more details, see our roadmap article on reported costs.
Monitor and analyze workflow costs as they run
To reduce a workflow’s cost, it’s essential to know which tasks require the most resources, and which resources cost the most. You can now gather this information by selecting the “resource monitoring” option when configuring a workflow submission. Read Monitoring GCP cloud resources used in a workflow for more information and step-by-step instructions.
Once you’ve run a workflow with resource monitoring enabled, you can break down the resources and costs consumed by the workflow. Several tutorial notebooks demonstrate how to generate plots to visualize the time spent on each workflow task: CPU, memory, and disk usage, and the cost of each resource for each task. This information will help you optimize your workflow for cost and estimate the costs of similar workflows in the future.
To generate these plots for your own data, clone the Workflow Resource and Cost-Monitoring Guide workspace and follow the instructions on the dashboard.
Initiative 2. Cost Controls and Estimates
Workflow cost thresholds
This feature gives you the ability to set a cost threshold for a workflow to stop runaway costs from occurring. We recommend you use this feature more as a safety net to prevent runaway costs than a strict budgeting tool. Although our findings show high accuracy and reliability in our cost estimations, workflows may not terminate immediately upon hitting the defined threshold.
This feature is currently in preview. For more details, see our roadmap article.
Estimate costs of previous and in-progress workflows
For new workflows submitted after March 3, 2025, Terra shows estimated costs of workflows using the same estimates used to calculate workflow cost thresholds. These estimated workflow costs are displayed on the submission details page. If you have spend reporting set up, the estimates will convert to the actual costs from Google within 24 hours.
Estimate costs for commonly-used workflows
To estimate the cost of one of Terra’s featured workflows, refer to costs of selected features workflows. Note that your cost may differ from the estimates listed in this article due to differences in your data’s size and runtime settings like disk size.
Example workspaces for these workflows offer tools and workflows with benchmarked cost and time estimates in the workspace dashboards. For example: CellBender, one of our most commonly used single cell workflows, GWAS (using REGENIE), and GATK-SV single sample analysis workflow are among the many featured and public workspaces which contain cost estimates.
Initiative 3: Cost Optimizations
Autoclass storage
By default, all Terra workspaces and Terra Data Repository datasets now leverage Autoclass storage to automatically move infrequently-accessed data to a less expensive, “colder” storage tier (see Google’s storage pricing tables for details). Terra defaults to move files all the way to the Archive storage class for maximum savings. We estimate this will save users an average of 30% to 50% on annual storage costs.
Cleaning up intermediate workflow files
Workflows can generate many intermediate files that users do not wish to keep long-term. Workflows now include an option to delete intermediate files. This reduces the storage costs incurred by workflows configured to run with this setting.
For situations where you need to clean up intermediate files from workflows that were not run with this setting or after a period of time, Terra has introduced a new way to clean up intermediate files that you no longer need using Google lifecycle rules. Lifecycle rules will both save money on storage costs and let you better manage files that are produced as a part of your analysis. For more details, see our roadmap article.
Requester pays for data egress
While users with read-only access to your workspaces cannot run compute, they can incur costs to your account by downloading or copying the data. To avoid paying when other users download or copy data from your workspace, you can enable requester pays on your bucket. Read using Requester Pays workspaces for more information and step-by-step instructions. This option is particularly helpful when hosting large datasets on Terra.
We want to hear from you!
Working in the cloud opens up many opportunities to analyze data at scale. We appreciate the feedback we have received from our community about the importance of transparent costs and the ability to control them, and are dedicated to making improvements to better meet your needs. With these new tools in hand, it will be easier to track and control costs.
We’d like to continue to hear your feedback on these features and suggestions for the future! Check out our public Terra roadmap to see features that are planned for the near term or submit a new feature request on our public community forums to suggest improvements.
