/orcd/pool/004/jahn/centos7/MITgcm_ifort-mpi-fast_san/verification/dome/run ../../../tools/tst_2+2 -mpi All -command "mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv" cmdEXE='mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv' from previous run STDOUT.0000, lastPick=' 20 ckptA' ; iter='20' ; sufx='ckptA' prepare parameter file 'data.tst' : prepare file 'data.tst' : done diff data.tst data 58,60c58 < nIter0=20, < nTimeSteps=4, < # nIter0=0, --- > nIter0=0, 62c60 < # nTimeSteps=20, --- > nTimeSteps=20, 65,66c63,64 < # pChkptFreq=0.0, < # chkptFreq=100000.0, --- > pChkptFreq=0.0, > chkptFreq=100000.0, link back: temp_tst/pickup*.ckptA* rename ckptA -> 0000000020 for all: pickup pickup_orlanskiW rnp_loc: pickup.ckptA pickup.0000000020 rnp_loc: pickup_orlanskiW.ckptA pickup_orlanskiW.0000000020 start-end iter: 20 , 22 , 24 sufix: '0000000020' '0000000022' '0000000024' cmdEXE=mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv ==> START RUN 2 x 2 it [0] MPI startup(): Single-threaded optimized library [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): shm and dapl data transfer modes forrtl: No such file or directory forrtl: severe (29): file not found, unit 9, file /orcd/pool/004/jahn/centos7/MITgcm_ifort-mpi-fast_san/verification/dome/run/pickup_orlanskiW.ckptA.006.001.data Image PC Routine Line Source libifcoremt.so.5 00002AAAACEC91C2 for__io_return Unknown Unknown libifcoremt.so.5 00002AAAACEED2DF for_open Unknown Unknown mitgcmuv 00000000004AA2D5 Unknown Unknown Unknown mitgcmuv 00000000005414F5 Unknown Unknown Unknown mitgcmuv 0000000000534C56 Unknown Unknown Unknown mitgcmuv 0000000000675B6A Unknown Unknown Unknown mitgcmuv 000000000061A6C4 Unknown Unknown Unknown mitgcmuv 0000000000632333 Unknown Unknown Unknown mitgcmuv 0000000000671BF0 Unknown Unknown Unknown mitgcmuv 0000000000699CEB Unknown Unknown Unknown mitgcmuv 0000000000699EA9 Unknown Unknown Unknown mitgcmuv 00000000005B4A54 Unknown Unknown Unknown mitgcmuv 00000000004026EE Unknown Unknown Unknown libc-2.17.so 00002AAAAF3EE505 __libc_start_main Unknown Unknown mitgcmuv 00000000004025F9 Unknown Unknown Unknown forrtl: No such file or directory forrtl: severe (29): file not found, unit 9, file /orcd/pool/004/jahn/centos7/MITgcm_ifort-mpi-fast_san/verification/dome/run/pickup.ckptA.005.002.data Image PC Routine Line Source libifcoremt.so.5 00002AAAACEC91C2 for__io_return Unknown Unknown libifcoremt.so.5 00002AAAACEED2DF for_open Unknown Unknown mitgcmuv 00000000004A2244 Unknown Unknown Unknown mitgcmuv 0000000000541135 Unknown Unknown Unknown mitgcmuv 00000000006A8F4C Unknown Unknown Unknown mitgcmuv 000000000061A705 Unknown Unknown Unknown mitgcmuv 0000000000632333 Unknown Unknown Unknown mitgcmuv 0000000000671BF0 Unknown Unknown Unknown mitgcmuv 0000000000699CEB Unknown Unknown Unknown mitgcmuv 0000000000699EA9 Unknown Unknown Unknown mitgcmuv 00000000005B4A54 Unknown Unknown Unknown mitgcmuv 00000000004026EE Unknown Unknown Unknown libc-2.17.so 00002AAAAF3EE505 __libc_start_main Unknown Unknown mitgcmuv 00000000004025F9 Unknown Unknown Unknown ==> END RUN 2 x 2 it listP= pickup pickup_orlanskiW rnp_loc: pickup.ckptA pickup.0000000024 rnp_loc: pickup_orlanskiW.ckptA pickup_orlanskiW.0000000024 move_outp: res_2it ==> START RUN 1iA [0] MPI startup(): Single-threaded optimized library [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END ==> END RUN 1iA rnp_loc: pickup.ckptA pickup.0000000022 rnp_loc: pickup_orlanskiW.ckptA pickup_orlanskiW.0000000022 move_outp: res_1iA ==> START RUN 1iB [0] MPI startup(): Single-threaded optimized library [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes ABNORMAL END: S/R MDS_READ_FIELD ==> RUN 1iB STOP without writing pickup => exit