New in Terra: Spend report per billing project
Adam Mullen
November 30, 2022
Learn more about the new spend reporting feature that provides more transparency into cloud computing costs incurred in Terra.
Menu
Menu
Learn more about the new spend reporting feature that provides more transparency into cloud computing costs incurred in Terra.
Adam Mullen, Software Product Manager in the Data Sciences Platform, explains a change to the default storage bucket setting for new Terra workspaces that will address the upcoming Google Cloud pricing changes to a number of services, including multi-regional storage buckets.
New Terra features give you greater control over data storage and computing resources by exposing the regional architecture of the Google Cloud Platform in key parts of Terra.
Starting the week of September 20th, each new Terra workspace will be associated with a dedicated Google project, which will allow us to improve the granularity of cost reporting in Terra.
This blog covers some key concepts and practical solutions for tackling the question of compute costs incurred by running workflows, which are typically the hardest to predict among the various types of cloud costs.
Many workflows generate intermediate files that you won’t ever use again once the pipeline has run to completion. You can reduce your data footprint — and your storage costs! — by getting Terra to delete them for you.
On the cloud, the rate you pay for computing resources is based on the technical specifications of the virtual machines you use. Our pipeline engineering team offers some tips for cutting costs by adjusting workflow resource allocations dynamically.
If you’re a PI switching from using institutional computing resources that are normally paid out of grant overhead, and until your institution lets you reallocate that money to cover cloud costs, you’re going to need to find new funding. The good news is that there are some funding sources out there dedicated to bridging that gap […]
Although Terra itself is free and open to all, you do have to pay for any actual work that you request from the underlying cloud service provider, which is currently Google Cloud Platform (GCP) (we’re working with Microsoft to add support for Azure). The good news is that you can get free credits to try out GCP and Terra, and see if this all works for you, without paying anything out of pocket.
If your day-to-day involves workflows with long-running tasks, I have good news for you: we just released a new checkpointing feature that makes it possible to save intermediate outputs for a task and resume work from that point if the task gets interrupted. For context, the Cromwell workflow engine we use in Terra already had a smart resume capability at the level of the workflow, called “call caching”.
Terra is developed by the Broad Institute of MIT and Harvard in collaboration with Microsoft and Verily.
