+ /opt/ddn/mvapich/bin/mpiexec -ppn 4 -np 8 -genv MV2_NUM_HCAS 1 -genv MV2_CPU_BINDING_LEVEL core -genv MV2_CPU_BINDING_POLICY scatter --hosts isc17-c04,isc17-c05 /esfs/jtacquaviva/software/install/ior/git-ddn/bin/ior -i 3 -s 1 -t 16384 -b 35295068160 -D 120 -a POSIX -F -e -g -z -k -o /esfs/jtacquaviva/ioperf/file_write -w + tee -a ./output/COUNT:1#NN:2#PPN:4#API:POSIX#T:16384.txt IOR-3.0.1: MPI Coordinated Test of Parallel I/O Began: Mon Oct 22 19:26:35 2018 Command line used: /esfs/jtacquaviva/software/install/ior/git-ddn/bin/ior -i 3 -s 1 -t 16384 -b 35295068160 -D 120 -a POSIX -F -e -g -z -k -o /esfs/jtacquaviva/ioperf/file_write -w Machine: Linux isc17-c04 Test 0 started: Mon Oct 22 19:26:35 2018 Summary: api = POSIX test filename = /esfs/jtacquaviva/ioperf/file_write access = file-per-process ordering in a file = random offsets ordering inter file= no tasks offsets clients = 8 (4 per node) repetitions = 3 xfersize = 16384 bytes blocksize = 32.87 GiB aggregate filesize = 262.97 GiB Using stonewalling = 120 second(s) access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- WARNING: Expected aggregate file size = 282360545280. WARNING: Stat() of aggregate file size = 282360545280. WARNING: Using actual aggregate bytes moved = 42085253120. write 310.04 34467840 16.00 0.000940 129.45 0.000283 129.45 0 WARNING: Expected aggregate file size = 282360545280. WARNING: Stat() of aggregate file size = 282358759424. WARNING: Using actual aggregate bytes moved = 34805186560. write 250.84 34467840 16.00 0.010233 132.32 0.000290 132.33 1 WARNING: Expected aggregate file size = 282360545280. WARNING: Stat() of aggregate file size = 282359365632. WARNING: Using actual aggregate bytes moved = 37848924160. write 270.72 34467840 16.00 0.019886 133.31 0.000331 133.33 2 Max Write: 310.04 MiB/sec (325.10 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum write 310.04 250.84 277.20 24.60 131.70394 0 8 4 3 1 0 1 0 0 1 35295068160 16384 42085253120 POSIX 0 Finished: Mon Oct 22 19:33:14 2018 + /opt/ddn/mvapich/bin/mpiexec -ppn 4 -np 8 -genv MV2_NUM_HCAS 1 -genv MV2_CPU_BINDING_LEVEL core -genv MV2_CPU_BINDING_POLICY scatter --hosts isc17-c04,isc17-c05 /esfs/jtacquaviva/git/ime-evaluation/drop_caches.sh + /opt/ddn/mvapich/bin/mpiexec -ppn 4 -np 8 -genv MV2_NUM_HCAS 1 -genv MV2_CPU_BINDING_LEVEL core -genv MV2_CPU_BINDING_POLICY scatter --hosts isc17-c04,isc17-c05 /esfs/jtacquaviva/software/install/ior/git-ddn/bin/ior -i 3 -s 1 -t 16384 -b 35295068160 -D 120 -a POSIX -F -e -g -z -k -o /esfs/jtacquaviva/indread2/file -r + tee -a ./output/COUNT:1#NN:2#PPN:4#API:POSIX#T:16384.txt IOR-3.0.1: MPI Coordinated Test of Parallel I/O Began: Mon Oct 22 19:33:25 2018 Command line used: /esfs/jtacquaviva/software/install/ior/git-ddn/bin/ior -i 3 -s 1 -t 16384 -b 35295068160 -D 120 -a POSIX -F -e -g -z -k -o /esfs/jtacquaviva/indread2/file -r Machine: Linux isc17-c04 Test 0 started: Mon Oct 22 19:33:25 2018 Summary: api = POSIX test filename = /esfs/jtacquaviva/indread2/file access = file-per-process ordering in a file = random offsets ordering inter file= no tasks offsets clients = 8 (4 per node) repetitions = 3 xfersize = 16384 bytes blocksize = 32.87 GiB aggregate filesize = 262.97 GiB Using stonewalling = 120 second(s) access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter ------ --------- ---------- --------- -------- -------- -------- -------- ---- WARNING: Expected aggregate file size = 282360545280. WARNING: Stat() of aggregate file size = 1129442181120. WARNING: Using actual aggregate bytes moved = 1890615296. read 15.01 34467840 16.00 0.000837 120.13 0.000239 120.13 0 WARNING: Expected aggregate file size = 282360545280. WARNING: Stat() of aggregate file size = 1129442181120. WARNING: Using actual aggregate bytes moved = 2103050240. read 16.70 34467840 16.00 0.000250 120.12 0.000203 120.12 1 WARNING: Expected aggregate file size = 282360545280. WARNING: Stat() of aggregate file size = 1129442181120. WARNING: Using actual aggregate bytes moved = 2139176960. read 16.98 34467840 16.00 0.000236 120.12 0.000190 120.12 2 Max Read: 16.98 MiB/sec (17.81 MB/sec) Summary of all tests: Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum read 16.98 15.01 16.23 0.87 120.12448 0 8 4 3 1 0 1 0 0 1 35295068160 16384 1890615296 POSIX 0 Finished: Mon Oct 22 19:39:26 2018 + set +x /esfs/jtacquaviva/ioperf stripe_count: 4 stripe_size: 1048576 stripe_offset: -1 /esfs/jtacquaviva/ioperf/file_write.00000006 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 2 obdidx objid objid group 2 12566550 0xbfc016 0 4 12508267 0xbedc6b 0 7 12456908 0xbe13cc 0 1 12638181 0xc0d7e5 0 /esfs/jtacquaviva/ioperf/file_write.00000002 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 6 obdidx objid objid group 6 12591812 0xc022c4 0 0 12420199 0xbd8467 0 2 12566552 0xbfc018 0 4 12508269 0xbedc6d 0 /esfs/jtacquaviva/ioperf/file_write.00000000 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 7 obdidx objid objid group 7 12456910 0xbe13ce 0 1 12638183 0xc0d7e7 0 3 12486496 0xbe8760 0 5 12378550 0xbce1b6 0 /esfs/jtacquaviva/ioperf/file_write.00000004 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 12420196 0xbd8464 0 2 12566549 0xbfc015 0 4 12508266 0xbedc6a 0 3 12486494 0xbe875e 0 /esfs/jtacquaviva/ioperf/file_write.00000007 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 4 obdidx objid objid group 4 12508268 0xbedc6c 0 7 12456909 0xbe13cd 0 1 12638182 0xc0d7e6 0 3 12486495 0xbe875f 0 /esfs/jtacquaviva/ioperf/file_write.00000003 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 12638180 0xc0d7e4 0 5 12378548 0xbce1b4 0 6 12591810 0xc022c2 0 0 12420197 0xbd8465 0 /esfs/jtacquaviva/ioperf/file_write.00000005 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 12638179 0xc0d7e3 0 3 12486493 0xbe875d 0 5 12378547 0xbce1b3 0 6 12591809 0xc022c1 0 /esfs/jtacquaviva/ioperf/file_write.00000001 lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_pattern: 1 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 12378549 0xbce1b5 0 6 12591811 0xc022c3 0 0 12420198 0xbd8466 0 2 12566551 0xbfc017 0