/orcd/scratch/orcd/001/jahn/testreport/rocky8/MITgcm_intel-mpi-fast/verification/solid-body.cs-32x32x1/run
../../../tools/tst_2+2 -mpi All -command "mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv"
 
cmdEXE='mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv'
from previous run STDOUT.0000, lastPick='       25 ckptA' ; iter='25' ; sufx='ckptA'
prepare parameter file 'data.tst' :
prepare file 'data.tst' : done
 diff data.tst data
41,43c41
<  nIter0=25,
<  nTimeSteps=4,
< # nIter0=0,
---
>  nIter0=0,
48,49c46,47
< # pChkptFreq=0.,
< # chkptFreq=0.,
---
>  pChkptFreq=0.,
>  chkptFreq=0.,
52c50
< # nTimeSteps=25,
---
>  nTimeSteps=25,
 
 link back: temp_tst/pickup*.ckptA*
 rename ckptA -> 0000000025 for all: pickup
  rnp_loc: pickup.ckptA pickup.0000000025
 start-end iter: 25 , 27 , 29
 sufix: '0000000025' '0000000027' '0000000029'
 cmdEXE=mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv
==> START RUN 2 x 2 it
mv: cannot stat 'STDOUT.0000': No such file or directory
[mpiexec@node1613] Error: Unable to run bstrap_proxy (pid 3460634, exit code 256)
[mpiexec@node1613] poll_for_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:157): check exit codes error
[mpiexec@node1613] HYD_dmx_poll_wait_for_proxy_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:206): poll for event error
[mpiexec@node1613] HYD_bstrap_setup (../../../../../src/pm/i_hydra/libhydra/bstrap/src/intel/i_hydra_bstrap.c:1069): error waiting for event
[mpiexec@node1613] Error setting up the bootstrap proxies
[mpiexec@node1613] Possible reasons:
[mpiexec@node1613] 1. Host is unavailable. Please check that all hosts are available.
[mpiexec@node1613] 2. Cannot launch hydra_bstrap_proxy or it crashed on one of the hosts.
[mpiexec@node1613]    Make sure hydra_bstrap_proxy is available on all hosts and it has right permissions.
[mpiexec@node1613] 3. Firewall refused connection.
[mpiexec@node1613]    Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable.
[mpiexec@node1613] 4. slurm bootstrap cannot launch processes on remote host.
[mpiexec@node1613]    You may try using -bootstrap option to select alternative launcher.
srun: error: Unable to create step for job 22444790: Memory required by task is not available
empty or no output file: 'std_outp.2it' => exit