Dear Experts
I’m working on a simulation using 10 spawns and have a quick question about extending cycles after running into some timeouts.
Initially, I ran 10 cycles, but the statistical uncertainties were still a bit high. To lower them, I extended the run from cycle 10 up to cycle 30. During that second run, 3 of the cycles timed out. I went ahead and merged the generated outputs in Flair, which brought the uncertainties down to some extent, but it raised two main concerns for me:
Are these merged results fully trustworthy given the 3 timed-out cycles? Does Flair automatically filter out incomplete files when processing the data, or could those broken cycles introduce errors into the results?
I’d like to add another 20 cycles to push the run from cycle 30 to 50 to further decrease the uncertainties. Is it safe to continue straight from cycle 30 after having timeouts? What is the best way to resume?
Timed-out cycles in Flair don’t always mean that they are not running; it just took more time than expected. You need to check if the output files are generated, or not.
If the cycles are really timed out due to a failure (which should be investigated), then no output files are generated; thus, they have no impact on the processed results.
If a cycle is failed, then they can’t be continued, because continuing expects the last cycle to be correctly finished.
It is always possible to start additional spawns from cycle 1, if more primaries are needed.
In my case, the output files for the timed-out cycles were indeed generated, and Flair was able to merge them without any issue.
What I’m still trying to understand is your statement:
“If the cycles are really timed out due to a failure (which should be investigated), then no output files are generated; thus, they have no impact on the processed results.”
Is it possible for one spawn to fail completely (producing no output files) while the other spawns continue and finish successfully?
Yes, it is possible to have only a fraction of spawns fail in a simulation. In theory, one spawn can encounter a rare bug in FLUKA, or – usually in an HPC environment – some spawns are stopped before producing results due to time or disk limit, etc.