/orcd/scratch/orcd/001/jahn/testreport/rocky8/darwin3_gfortran-mpi/verification_darwin/macromol_eccov3_2+1/run ../../../tools/tst_2+2 -mpi All -command "mpirun -n 72 ./mitgcmuv" cmdEXE='mpirun -n 72 ./mitgcmuv' from previous run STDOUT.0000, lastPick=' 24 ckptA' ; iter='24' ; sufx='ckptA' prepare parameter file 'data.tst' : prepare file 'data.tst' : done diff data.tst data 59,61c59 < nIter0=24, < nTimeSteps=4, < # nIter0=0, --- > nIter0=0, 63c61 < # nTimeSteps=24, --- > nTimeSteps=24, 70,71c68,69 < # pChkptFreq = 31104000., < # chkptFreq = 31104000., --- > pChkptFreq = 31104000., > chkptFreq = 31104000., changes in data.pkg 8c8 < # useMNC=.TRUE., --- > useMNC=.TRUE., link back: temp_tst/pickup*.ckptA* rename ckptA -> 0000000024 for all: pickup_darwin pickup_ptracers rnp_loc: pickup_darwin.ckptA pickup_darwin.0000000024 rnp_loc: pickup_ptracers.ckptA pickup_ptracers.0000000024 start-end iter: 24 , 26 , 28 sufix: '0000000024' '0000000026' '0000000028' cmdEXE=mpirun -n 72 ./mitgcmuv ==> START RUN 2 x 2 it Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END ==> END RUN 2 x 2 it listP= pickup_darwin pickup_ptracers rnp_loc: pickup_darwin.ckptA pickup_darwin.0000000028 rnp_loc: pickup_ptracers.ckptA pickup_ptracers.0000000028 move_outp: res_2it ==> START RUN 1iA Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END Note: The following floating-point exceptions are signalling: IEEE_UNDERFLOW_FLAG IEEE_DENORMAL STOP NORMAL END ==> END RUN 1iA rnp_loc: pickup_darwin.ckptA pickup_darwin.0000000026 rnp_loc: pickup_ptracers.ckptA pickup_ptracers.0000000026 move_outp: res_1iA ==> START RUN 1iB [node1616:128072:0:128072] ib_mlx5_log.c:179 Transport retry count exceeded on mlx5_0:1/IB (synd 0x15 vend 0x81 hw_synd 0/0) [node1616:128072:0:128072] ib_mlx5_log.c:179 DCI QP 0x17845 wqe[6969]: SEND s-e [rqpn 0xb1b0 rlid 1659] [inl len 14] ==== backtrace (tid: 128072) ==== 0 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/libucs.so.0(ucs_handle_error+0x294) [0x14999cbfcb94] 1 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/libucs.so.0(ucs_fatal_error_message+0xb2) [0x14999cbf9e92] 2 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/libucs.so.0(ucs_log_print_compact+0) [0x14999cbfe700] 3 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/libucs.so.0(ucs_log_dispatch+0xcc) [0x14999cbfea8c] 4 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/ucx/libuct_ib.so.0(uct_ib_mlx5_completion_with_err+0x50d) [0x14999c96504d] 5 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/ucx/libuct_ib.so.0(uct_dc_mlx5_ep_handle_failure+0xbe) [0x14999c99288e] 6 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/ucx/libuct_ib.so.0(uct_ib_mlx5_check_completion+0x35) [0x14999c965fd5] 7 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/ucx/libuct_ib.so.0(+0x55c67) [0x14999c994c67] 8 /orcd/software/core/001/spack/pkg/ucx/1.16.0/pmf3myv/lib/libucp.so.0(ucp_worker_progress+0x5a) [0x14999d2d2fba] 9 /orcd/software/core/001/spack/pkg/openmpi/4.1.4/zuyo6jx/lib/openmpi/mca_pml_ucx.so(mca_pml_ucx_recv+0x130) [0x14999d76e060] 10 /orcd/software/core/001/spack/pkg/openmpi/4.1.4/zuyo6jx/lib/libmpi.so.40(MPI_Recv+0x155) [0x1499aa7a10e5] 11 /orcd/software/core/001/spack/pkg/openmpi/4.1.4/zuyo6jx/lib/libmpi_mpifh.so.40(pmpi_recv_+0x71) [0x1499aaaaa691] 12 ./mitgcmuv() [0x64ef79] 13 ./mitgcmuv() [0x564e66] 14 ./mitgcmuv() [0x5ee5f9] 15 ./mitgcmuv() [0x5da1e3] 16 ./mitgcmuv() [0x6ef5e5] 17 ./mitgcmuv() [0x69ddbc] 18 ./mitgcmuv() [0x6aa4fe] 19 ./mitgcmuv() [0x6ea639] 20 ./mitgcmuv() [0x70b4cf] 21 ./mitgcmuv() [0x70b5c5] 22 ./mitgcmuv() [0x65959d] 23 ./mitgcmuv() [0x6597ff] 24 /lib64/libc.so.6(__libc_start_main+0xe5) [0x1499a9cf47e5] 25 ./mitgcmuv() [0x40308e] ================================= Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x1499a9d085af in ??? #1 0x1499a9d0852f in ??? #2 0x1499a9cdbe64 in ??? #3 0x14999cbf9e96 in ucs_fatal_error_message at debug/assert.c:38 #4 0x14999cbfe6ff in ucs_log_default_handler at debug/log.c:324 #5 0x14999cbfea8b in ucs_log_dispatch at debug/log.c:388 #6 0x14999c96504c in uct_ib_mlx5_completion_with_err at mlx5/ib_mlx5_log.c:179 #7 0x14999c99288d in uct_dc_mlx5_ep_handle_failure at dc/dc_mlx5_ep.c:1728 #8 0x14999c965fd4 in uct_ib_mlx5_check_completion at mlx5/ib_mlx5.c:477 #9 0x14999c994c66 in uct_ib_mlx5_poll_cq at /tmp/milechin/spack-stage/spack-stage-ucx-1.16.0-pmf3myvl62bxgvhrmrmzre2p4iwwze62/spack-src/src/uct/ib/mlx5/ib_mlx5.inl:146 #10 0x14999c994c66 in uct_dc_mlx5_poll_tx at dc/dc_mlx5.c:271 #11 0x14999c994c66 in uct_dc_mlx5_iface_progress at dc/dc_mlx5.c:317 #12 0x14999c994c66 in uct_dc_mlx5_iface_progress_ll at dc/dc_mlx5.c:327 #13 0x14999d2d2fb9 in ucs_callbackq_dispatch at /tmp/milechin/spack-stage/spack-stage-ucx-1.16.0-pmf3myvl62bxgvhrmrmzre2p4iwwze62/spack-src/src/ucs/datastruct/callbackq.h:215 #14 0x14999d2d2fb9 in uct_worker_progress at /tmp/milechin/spack-stage/spack-stage-ucx-1.16.0-pmf3myvl62bxgvhrmrmzre2p4iwwze62/spack-src/src/uct/api/uct.h:2787 #15 0x14999d2d2fb9 in ucp_worker_progress at core/ucp_worker.c:2991 #16 0x14999d76e05f in ??? #17 0x1499aa7a10e4 in ??? #18 0x1499aaaaa690 in ??? #19 0x64ef78 in ??? #20 0x564e65 in ??? #21 0x5ee5f8 in ??? #22 0x5da1e2 in ??? #23 0x6ef5e4 in ??? #24 0x69ddbb in ??? #25 0x6aa4fd in ??? #26 0x6ea638 in ??? #27 0x70b4ce in ??? #28 0x70b5c4 in ??? #29 0x65959c in ??? #30 0x6597fe in ??? #31 0x1499a9cf47e4 in ??? #32 0x40308d in ??? #33 0xffffffffffffffff in ??? -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 0 on node node1616 exited on signal 6 (Aborted). -------------------------------------------------------------------------- ==> END RUN 1iB rnp_loc: pickup_darwin.ckptA pickup_darwin.0000000028 ls: cannot access 'pickup_darwin.ckptA*': No such file or directory rnp_loc: pickup_ptracers.ckptA pickup_ptracers.0000000028 move_outp: res_1iB == compare cg2d_init_res : run 1iA: run 1iB: run 2it: == diff pickup files : end of 1rst run (2x2 it) & end of 3rd run (2nd 2 it) --> file=pickup_darwin.0000000028, listY=. diff res_2it/pickup_darwin.0000000028.data res_1iB diff: res_1iB/pickup_darwin.0000000028.data: No such file or directory cmp: res_1iB/pickup_darwin.0000000028.data: No such file or directory diff res_2it/pickup_darwin.0000000028.meta res_1iB diff: res_1iB/pickup_darwin.0000000028.meta: No such file or directory Diff outp= 2 ==> stop