This week, we released one of those changes that looks small on the face of it but is actually a really big deal. Specifically, we upgraded the cloud environment (previously called “runtime”) that we provide in Terra for running Jupyter Notebooks to support persistent disk storage.
Until now, one of the limitations of our Notebook environment was that you had to manually save any outputs you cared about to a Google bucket (or other location of your choosing). For technical reasons, the storage space associated with the notebook was not guaranteed to stick around when you weren’t actively using it, and if you made certain configuration changes to your environment, the storage space was wiped and recreated from scratch.
Going forward, you’ll have the option to use what’s called a “detachable persistent disk” to store data that you plan to use as well as the outputs of any analyses you run in your notebooks. You can think of this as a sort of virtual USB thumb drive; you plug it in when you want to do some work, then detach it when you’re done, and keep it in your pocket until next time. (Just make sure you don’t leave it there when you do your laundry.)
In practice, the plugging in and detaching will be done automatically for you. When you reconfigure or delete your environment, the system will automatically detach and reattach your persistent disk, or save it for later if you don’t create a new environment right away.
You get one persistent disk per billing project, and you’ll be able to use that same persistent disk with notebooks in any of your workspaces within that billing project. You can see — and customize — the size of your persistent disk in the environment configuration panel, as shown in the screenshot below.
Note that the persistent disk is mounted to the directory /home/jupyter-user/notebooks, so please make sure that’s where you save your data and outputs! A benefit of this setup is that any software packages you install through the notebook will be saved there as well, reducing the amount of reinstallation you have to do when you change your environment configuration.
Interested in learning more? We’ve created this step by step guide to walk you through how and when to utilize this feature when launching a notebook. Please note this feature only supports cloud environments that use the “standard VM” option and will not apply to Spark or Hail application configurations.
We’d love to hear your thoughts on this new capability in the comments below! And as always, don’t hesitate to reach out to the Terra helpdesk if you run into any issues.
