How insightful feedback about the GATK showcase led to a flurry of updates and the creation of a suite of format conversion tools
Last June, just as we were putting the final touches on the teaching materials for a 4-day GATK workshop that we were planning to run entirely on Terra, we received a very insightful email from the workshop host, Matt Bashton of the University of Newcastle in the UK. A long-time user of the GATK, Matt had been trying out the Best Practices workspaces that we maintain on behalf of the GATK team in the Terra Showcase. In his email, he detailed a list of nine issues he saw as blocking researchers like him from moving their work onto Terra.
The workshop was scheduled for the following week, so my initial response was a bit of panic. Even though the workshop mainly uses Terra as a convenient environment for running the hands-on tutorials, it’s a unique opportunity for us to introduce the Best Practices workspaces as the recommended way to try out and evaluate the official GATK workflows. That wasn’t going to be terribly effective if there were major flaws in the resources involved.
Reading down the detailed list Matt had sent us, I understood his concerns: for example, he was absolutely right that it wasn’t clear how to convert sequence data to the uBAM format many GATK workflows used as input on the platform. Or how to deal with interleaved FASTQ files. Some issues we were already aware of, such as the lack of support for WDL 1.0 which had been blocking the publication of new and updated GATK workflows on Terra. In the case of WDL 1.0, resolution was imminent (in fact it’s now supported as of July 18th!), but others were a month or more away from being addressed. So while we wanted to make some concrete improvements in time for the workshop, my immediate question was, what could we possibly do in less than a week??
Turns out, quite a bit more than I was giving our team credit for.
That Friday afternoon, four members of our Support team sprang into action. It started with a phone call meeting between two team members in the office, one working remotely and me on my drive up to Maine for the weekend (don’t worry, I pulled off the road at a stop before taking the call). We went over the list of issues one by one and within an hour had a list of action items for the four of us to tackle before the workshop.
By that evening we were working away. We had a very basic file format conversion workspace already, so one team member went through those workflows, updating them to make sure they were configured to be runnable back-to-back (in the right order) without additional tweaks. Another developed an entirely new WDL that converts an interleaved FASTQ input into paired FASTQ files. I worked on the documentation to clarify the purpose and requirements of each workflow, define their input and output formats, and so on. As we finished each part of the action plan, our fourth teammate updated the tools, data, and workspaces in the public showcase.
Within five days we had completely revamped the barebones file format conversion workspace that we had started from. It now contained one completely new and four significantly updated workflows, as well as a clearer title and description card:
We shared the new workspace with Matt, and his feedback was deeply rewarding: “Thanks so much for creating this workspace it’s exactly what I was looking for I suspect it will really help with on-boarding of new people.” Nailed it! We don’t often pull a weekender like that — this was honestly an extreme case (seriously, we know about work-life balance) — but sometimes it’s just worth it, you know?
Of course, we know that we’re not done yet. In our line of work, you’re never really done. There’s always more to do to clear the path, lay down tracks and make the ride smoother. For my part, the next project I plan to tackle is making all the GATK workspaces follow the same template – more modular, with clearer descriptions and titles.
I wanted to share this story to encourage all of you to give us your feedback too. Be like Matt. Tell us what you need, what’s blocking you. I can’t promise we’ll always have the solutions to all your troubles, but I can promise we’ll do our best every time.
