/orcd/pool/004/jahn/centos7/MITgcm_ifort-mpi-fast_san/verification/fizhi-gridalt-hs/run ../../../tools/tst_2+2 -mpi All -command "mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv" cmdEXE='mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv' from previous run STDOUT.0000, lastPick=' 5 ckptA' ; iter='5' ; sufx='ckptA' prepare parameter file 'data.tst' : prepare file 'data.tst' : done diff data.tst data 54,56c54 < nIter0=5, < nTimeSteps=4, < # nIter0=0, --- > nIter0=0, 58c56 < # nTimeSteps=5, --- > nTimeSteps=5, 65,66c63,64 < # pChkptFreq=2592000.0, < # chkptFreq=0.0, --- > pChkptFreq=2592000.0, > chkptFreq=0.0, link back: temp_tst/pickup*.ckptA* rename ckptA -> 0000000005 for all: pickup pickup_fizhi pickup_vegtiles rnp_loc: pickup.ckptA pickup.0000000005 rnp_loc: pickup_fizhi.ckptA pickup_fizhi.0000000005 rnp_loc: pickup_vegtiles.ckptA pickup_vegtiles.0000000005 start-end iter: 5 , 7 , 9 sufix: '0000000005' '0000000007' '0000000009' cmdEXE=mpirun -env I_MPI_DEBUG 2 -n 6 ./mitgcmuv ==> START RUN 2 x 2 it [0] MPI startup(): Single-threaded optimized library [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [1] MPI startup(): shm and dapl data transfer modes [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart Opening pickup_vegtiles.0000000005.001.001.data Opening pickup_vegtiles.0000000005.005.001.data Opening pickup_vegtiles.0000000005.003.001.data Opening pickup_vegtiles.0000000005.002.001.data Opening pickup_vegtiles.0000000005.004.001.data Opening pickup_vegtiles.0000000005.006.001.data Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 122230 Current Date 20040301 Current Time 122230 Current Date 20040301 Current Time 122230 Current Date 20040301 Current Time 122230 Current Date 20040301 Current Time 122230 Current Date 20040301 Current Time 122230 Current Date 20040301 Current Time 123000 Current Date 20040301 Current Time 123000 Current Date 20040301 Current Time 123000 Current Date 20040301 Current Time 123000 Current Date 20040301 Current Time 123000 Current Date 20040301 Current Time 123000 Writing datetime file: NYMD = 20040301 NHMS = 123000 Writing datetime file: NYMD = 20040301 NHMS = 123000 Writing datetime file: NYMD = 20040301 NHMS = 123000 Writing datetime file: NYMD = 20040301 NHMS = 123000 Writing datetime file: NYMD = 20040301 NHMS = 123000 Writing datetime file: NYMD = 20040301 NHMS = 123000 NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END ==> END RUN 2 x 2 it listP= pickup pickup_fizhi pickup_vegtiles rnp_loc: pickup.ckptA pickup.0000000009 rnp_loc: pickup_fizhi.ckptA pickup_fizhi.0000000009 rnp_loc: pickup_vegtiles.ckptA pickup_vegtiles.0000000009 move_outp: res_2it ==> START RUN 1iA [0] MPI startup(): Single-threaded optimized library [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [1] MPI startup(): shm and dapl data transfer modes [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart Opening pickup_vegtiles.0000000005.001.001.data Opening pickup_vegtiles.0000000005.006.001.data Opening pickup_vegtiles.0000000005.005.001.data Opening pickup_vegtiles.0000000005.004.001.data Opening pickup_vegtiles.0000000005.003.001.data Opening pickup_vegtiles.0000000005.002.001.data Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 120730 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Current Date 20040301 Current Time 121500 Writing datetime file: NYMD = 20040301 NHMS = 121500 Writing datetime file: NYMD = 20040301 NHMS = 121500 Writing datetime file: NYMD = 20040301 NHMS = 121500 Writing datetime file: NYMD = 20040301 NHMS = 121500 Writing datetime file: NYMD = 20040301 NHMS = 121500 Writing datetime file: NYMD = 20040301 NHMS = 121500 NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END NORMAL END ==> END RUN 1iA rnp_loc: pickup.ckptA pickup.0000000007 rnp_loc: pickup_fizhi.ckptA pickup_fizhi.0000000007 rnp_loc: pickup_vegtiles.ckptA pickup_vegtiles.0000000007 move_outp: res_1iA ==> START RUN 1iB [0] MPI startup(): Single-threaded optimized library [0] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [1] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [2] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [0] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [0] MPI startup(): shm and dapl data transfer modes [1] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [1] MPI startup(): shm and dapl data transfer modes [2] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [2] MPI startup(): shm and dapl data transfer modes [3] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [4] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [5] DAPL startup(): trying to open DAPL provider from I_MPI_DAPL_PROVIDER: ofa-v2-mlx5_0-1u [3] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [3] MPI startup(): shm and dapl data transfer modes [4] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [4] MPI startup(): shm and dapl data transfer modes [5] MPI startup(): DAPL provider ofa-v2-mlx5_0-1u [5] MPI startup(): shm and dapl data transfer modes Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 Begin Date 20040301 Begin Time 120000 Current Date 20040301 Current Time 120000 In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart In fizhi_init_vars: Read from restart Opening pickup_vegtiles.0000000007.001.001.data Opening pickup_vegtiles.0000000007.003.001.data Opening pickup_vegtiles.0000000007.002.001.data Opening pickup_vegtiles.0000000007.004.001.data Opening pickup_vegtiles.0000000007.006.001.data Opening pickup_vegtiles.0000000007.005.001.data forrtl: No such file or directory forrtl: severe (29): file not found, unit 9, file /orcd/pool/004/jahn/centos7/MITgcm_ifort-mpi-fast_san/verification/fizhi-gridalt-hs/run/pickup_vegtiles.0000000007.005.001.data Image PC Routine Line Source libifcoremt.so.5 00002AAAACEC91C2 for__io_return Unknown Unknown libifcoremt.so.5 00002AAAACEED2DF for_open Unknown Unknown mitgcmuv 0000000000542CB4 Unknown Unknown Unknown mitgcmuv 0000000000404866 Unknown Unknown Unknown mitgcmuv 000000000076B250 Unknown Unknown Unknown mitgcmuv 000000000075C6B8 Unknown Unknown Unknown mitgcmuv 000000000078A0F8 Unknown Unknown Unknown mitgcmuv 000000000078A339 Unknown Unknown Unknown mitgcmuv 00000000006C0464 Unknown Unknown Unknown mitgcmuv 000000000040292E Unknown Unknown Unknown libc-2.17.so 00002AAAAF3EE505 __libc_start_main Unknown Unknown mitgcmuv 0000000000402839 Unknown Unknown Unknown ==> RUN 1iB STOP without writing pickup => exit