/orcd/pool/004/jahn/centos7/MITgcm_ifort-mpi-fast_san/verification/so_box_biogeo/tr_run.saphe ../../../tools/tst_2+2 -mpi All -command "mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv" cmdEXE='mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv' from previous run STDOUT.0000, lastPick=' 10 ckptA' ; iter='10' ; sufx='ckptA' prepare parameter file 'data.tst' : prepare file 'data.tst' : done diff data.tst data 45,47c45 < nIter0=10, < nTimeSteps=4, < # nIter0=0, --- > nIter0=0, 49c47 < # nTimeSteps = 10, --- > nTimeSteps = 10, 56,57c54,55 < # pChkptFreq = 311040000., < # chkptFreq = 31104000., --- > pChkptFreq = 311040000., > chkptFreq = 31104000., link back: temp_tst/pickup*.ckptA* rename ckptA -> 0000000010 for all: pickup pickup_dic pickup_ptracers rnp_loc: pickup.ckptA pickup.0000000010 rnp_loc: pickup_dic.ckptA pickup_dic.0000000010 rnp_loc: pickup_ptracers.ckptA pickup_ptracers.0000000010 start-end iter: 10 , 12 , 14 sufix: '0000000010' '0000000012' '0000000014' cmdEXE=mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv ==> START RUN 2 x 2 it [0] MPI startup(): Single-threaded optimized library [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): shm and dapl data transfer modes [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END ==> END RUN 2 x 2 it listP= pickup pickup_dic pickup_ptracers rnp_loc: pickup.ckptA pickup.0000000014 rnp_loc: pickup_dic.ckptA pickup_dic.0000000014 rnp_loc: pickup_ptracers.ckptA pickup_ptracers.0000000014 move_outp: res_2it ==> START RUN 1iA [0] MPI startup(): Single-threaded optimized library [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [0] MPI startup(): shm and dapl data transfer modes [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END ==> END RUN 1iA rnp_loc: pickup.ckptA pickup.0000000012 rnp_loc: pickup_dic.ckptA pickup_dic.0000000012 rnp_loc: pickup_ptracers.ckptA pickup_ptracers.0000000012 move_outp: res_1iA ==> START RUN 1iB [0] MPI startup(): Single-threaded optimized library [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed QQ load dic parameters, initial fixed ABNORMAL END: S/R MDS_READ_FIELD ABNORMAL END: S/R MDS_READ_FIELD ABNORMAL END: S/R MDS_READ_FIELD ==> RUN 1iB STOP without writing pickup => exit