/orcd/pool/004/jahn/centos7/MITgcm_ifort-mpi-fast_san/verification/advect_cs/run ../../../tools/tst_2+2 -mpi All -command "mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv" cmdEXE='mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv' from previous run STDOUT.0000, lastPick=' 192 ckptA' ; iter='192' ; sufx='ckptA' prepare parameter file 'data.tst' : prepare file 'data.tst' : done diff data.tst data 30,33c30,31 < nIter0=192, < nTimeSteps=4, < # nIter0=0, < # endTime=518400., --- > nIter0=0, > endTime=518400., 38,39c36,37 < # pChkptFreq=0., < # chkptFreq=0., --- > pChkptFreq=0., > chkptFreq=0., link back: temp_tst/pickup*.ckptA* rename ckptA -> 0000000192 for all: pickup pickup_somS rnp_loc: pickup.ckptA pickup.0000000192 rnp_loc: pickup_somS.ckptA pickup_somS.0000000192 start-end iter: 192 , 194 , 196 sufix: '0000000192' '0000000194' '0000000196' cmdEXE=mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv ==> START RUN 2 x 2 it [0] MPI startup(): Single-threaded optimized library [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END ==> END RUN 2 x 2 it listP= pickup pickup_somS rnp_loc: pickup.ckptA pickup.0000000196 rnp_loc: pickup_somS.ckptA pickup_somS.0000000196 move_outp: res_2it ==> START RUN 1iA [0] MPI startup(): Single-threaded optimized library [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): shm and dapl data transfer modes [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes forrtl: No such file or directory forrtl: severe (29): file not found, unit 9, file /orcd/pool/004/jahn/centos7/MITgcm_ifort-mpi-fast_san/verification/advect_cs/run/pickup.ckptA.002.003.data Image PC Routine Line Source libifcoremt.so.5 00002AAAACEC91C2 for__io_return Unknown Unknown libifcoremt.so.5 00002AAAACEED2DF for_open Unknown Unknown mitgcmuv 0000000000509968 Unknown Unknown Unknown mitgcmuv 0000000000563A35 Unknown Unknown Unknown mitgcmuv 000000000066D2AA Unknown Unknown Unknown mitgcmuv 0000000000608D65 Unknown Unknown Unknown mitgcmuv 000000000061CC09 Unknown Unknown Unknown mitgcmuv 0000000000648E20 Unknown Unknown Unknown mitgcmuv 0000000000665A7B Unknown Unknown Unknown mitgcmuv 0000000000665C39 Unknown Unknown Unknown mitgcmuv 00000000005BE934 Unknown Unknown Unknown mitgcmuv 000000000040281E Unknown Unknown Unknown libc-2.17.so 00002AAAAF3EE505 __libc_start_main Unknown Unknown mitgcmuv 0000000000402729 Unknown Unknown Unknown ==> END RUN 1iA rnp_loc: pickup.ckptA pickup.0000000194 rnp_loc: pickup_somS.ckptA pickup_somS.0000000194 move_outp: res_1iA ==> START RUN 1iB [0] MPI startup(): Single-threaded optimized library [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes ABNORMAL END: S/R MDS_READ_FIELD ABNORMAL END: S/R MDS_READ_FIELD ==> RUN 1iB STOP without writing pickup => exit