/orcd/scratch/orcd/001/jahn/testreport/rocky8/MITgcm_intel-mpi-fast/verification/short_surf_wave/run ../../../tools/tst_2+2 -mpi All -command "mpirun -env I_MPI_DEBUG 2 -n 4 ./mitgcmuv" cmdEXE='mpirun -env I_MPI_DEBUG 2 -n 4 ./mitgcmuv' from previous run STDOUT.0000, lastPick=' 12 ckptA' ; iter='12' ; sufx='ckptA' prepare parameter file 'data.tst' : prepare file 'data.tst' : done diff data.tst data 46,48c46 < nIter0=12, < nTimeSteps=4, < # nIter0=1, --- > nIter0=1, 50c48 < # nTimeSteps=11, --- > nTimeSteps=11, 55,56c53,54 < # pChkptFreq=0.0, < # chkptFreq=0.0, --- > pChkptFreq=0.0, > chkptFreq=0.0, link back: temp_tst/pickup*.ckptA* rename ckptA -> 0000000012 for all: pickup rnp_loc: pickup.ckptA pickup.0000000012 start-end iter: 12 , 14 , 16 sufix: '0000000012' '0000000014' '0000000016' cmdEXE=mpirun -env I_MPI_DEBUG 2 -n 4 ./mitgcmuv ==> START RUN 2 x 2 it [0] MPI startup(): Intel(R) MPI Library, Version 2021.13 Build 20240701 (id: 179630a) [0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved. [0] MPI startup(): library kind: release [0] MPI startup(): libfabric version: 1.20.1-impi [0] MPI startup(): libfabric provider: mlx [0] MPI startup(): Load tuning file: "/orcd/software/community/001/rocky8/intel/2024.2.1/mpi/2021.13/opt/mpi/etc/tuning_generic_shm-ofi_mlx_hcoll.dat" NORMAL END NORMAL END NORMAL END NORMAL END ==> END RUN 2 x 2 it listP= pickup rnp_loc: pickup.ckptA pickup.0000000016 move_outp: res_2it ==> START RUN 1iA mv: cannot stat 'STDOUT.0000': No such file or directory [mpiexec@node3108] Error: Unable to run bstrap_proxy (pid 1168504, exit code 256) [mpiexec@node3108] poll_for_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:157): check exit codes error [mpiexec@node3108] HYD_dmx_poll_wait_for_proxy_event (../../../../../src/pm/i_hydra/libhydra/demux/hydra_demux_poll.c:206): poll for event error [mpiexec@node3108] HYD_bstrap_setup (../../../../../src/pm/i_hydra/libhydra/bstrap/src/intel/i_hydra_bstrap.c:1069): error waiting for event [mpiexec@node3108] Error setting up the bootstrap proxies [mpiexec@node3108] Possible reasons: [mpiexec@node3108] 1. Host is unavailable. Please check that all hosts are available. [mpiexec@node3108] 2. Cannot launch hydra_bstrap_proxy or it crashed on one of the hosts. [mpiexec@node3108] Make sure hydra_bstrap_proxy is available on all hosts and it has right permissions. [mpiexec@node3108] 3. Firewall refused connection. [mpiexec@node3108] Check that enough ports are allowed in the firewall and specify them with the I_MPI_PORT_RANGE variable. [mpiexec@node3108] 4. slurm bootstrap cannot launch processes on remote host. [mpiexec@node3108] You may try using -bootstrap option to select alternative launcher. srun: error: Unable to create step for job 28278677: Memory required by task is not available empty or no output file: 'std_outp.1iA' => exit