Got a problem?

1. Search using the upper-right search box, e.g. using the error message.
2. Do you think FireCloud might be down? Please read this before posting.
3. Be as detailed as you can when describing your problem, i.e. tell us what happened or didn't happen rather than "it didn't work" or "it failed".
4. Include the full text of any error messages you encounter.
5. Include screenshots to show any abnormal display or indicators.
6. If the problem affects a workspace, use the Share button in FireCloud to share it with [email protected] Provide the name of the workspace as well as any submission and workflow identifiers in a forum post. By sharing with this group, you are giving access to the Data Sciences Platform's communication team and potentially engineers on the NCI Suitability Roster if further troubleshooting is required. Please consider whether your data is sensitive and should be shared with these individuals. If you have an Authorization Domain set, this group will not have access to the contents of the workspace even after sharing.

Get notifications!

You can opt in to receive email notifications, for example when your questions get answered or when there are new announcements, by following the instructions given here.

Formatting tip!

Wrap blocks of code, error messages and BAM/VCF snippets--especially content with hashes (#)--with lines with three backticks ( ``` ) each to make a code block as demonstrated here.

Hi @vickyhorst@francois_a@Chip , this error comes up when FireCloud is overloaded with activity. For example, a bunch of submissions have all completed and are overloading the system trying to write their outputs back to the database.

The best solution at the moment is to come back later (a couple of hours or the next day) and try again. This is not an ideal experience for you and I will make sure to communicate this to the team. Sorry for the inconvenience. We appreciate you letting us know and please continue to let us know if you hit this obstacle again.

Thanks @Tiffany_at_Broad -- can you provide a rough estimate for how long this issue will persist? Waiting for hours/days for jobs to get submitted isn't really an option for much of the work we're doing.

Hi everyone,
We really apologize for the inconvenience. There is a team that is diagnosing the issue since last night. we are going to work in the coming quarter on a more permanent solution that should address this problem. Our team will reach you by email today to resolve the issues you are currently experiencing.
Thanks for the understanding ,
Moran

This issue is really hindering CGA's ability to do its work. PLEASE give us more details regarding what is being done to address the issue and a time frame (more specific than "in the coming quarter") in which we can expect the problem to be fixed. This is a major regression in the scalability of the system.

Last night, we were able to restart one of the processes and lift the deadlock symptoms many were experiencing yesterday. It isn't a permanent fix, and our developers are still working on a long-term solution. We are actively monitoring FireCloud for deadlock issues, outages, and slowness. However, if you do experience any of these, please do report them here on the forum as well. Thank you for your patience.

Last night, a release went out which includes a fix for the behavior we observed with the outage discussed in this thread. If you are still experiencing the same errors, or if you encounter them again in the future, please let us know.

For further information on the fix included in the release, please read the release notes here.

Hi Firecloud,
I am getting this message this afternoon for workflow id f6a63ea6-8477-4942-aed0-1dde1ce41676 among others. Any thoughts on what could be causing this even after the April 10th update?
Thanks for your help.