+ /opt/ddn/mvapich/bin/mpiexec -ppn 6 -np 96 -genv MV2_NUM_HCAS 1 -genv MV2_CPU_BINDING_LEVEL core -genv MV2_CPU_BINDING_POLICY scatter --hosts isc17-c01,isc17-c02,isc17-c03,isc17-c04,isc17-c05,isc17-c06,isc17-c07,isc17-c08,isc17-c09,isc17-c11,isc17-c12,isc17-c13,isc17-c14,isc17-c15,isc17-c18,isc17-c22 /esfs/jtacquaviva/software/install/ior/git-ddn/bin/ior -i 3 -s 1 -t 102400 -b 23530045440 -D 120 -a POSIX -F -e -g -z -k -o /esfs/jtacquaviva/ioperf/file_write -w + tee -a ./output/COUNT:1#NN:16#PPN:6#API:POSIX#T:102400.txt IOR-3.0.1: MPI Coordinated Test of Parallel I/O ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_18]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 18 [cli_1]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_19]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 19 [cli_5]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 5 [cli_7]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 7 [cli_20]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 20 [cli_9]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 9 [cli_84]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 84 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_21]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 21 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0 [cli_12]: [cli_37]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 37 [cli_8]: [cli_60]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 60 [cli_55]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 55 [cli_25]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 25 [cli_75]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 75 [cli_49]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 49 tee: [cli_86]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 86 standard output[cli_78]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 78 [cli_22]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 22 [cli_4]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 4 [cli_13]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 13 [cli_47]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 47 [cli_38]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 38 [cli_6]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 6 [cli_33]: [cli_61]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 61 [cli_59]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 59 [cli_27]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 27 [cli_72]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 72 [cli_51]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 51 [cli_68]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 68 [cli_90]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 90 [cli_89]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 89 [cli_79]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 79 : Resource temporarily unavailable [cli_23]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 23 [cli_15]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 15 [cli_42]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 42 [cli_39]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 39 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 8 [cli_35]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 35 [cli_63]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 63 [cli_54]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 54 [cli_29]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 29 ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) [cli_73]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 73 [cli_50]: [cli_70]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 70 [cli_91]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 91 [cli_87]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 87 [cli_80]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 80 [cli_2]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2 [cli_17]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 17 [cli_44]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 44 [cli_40]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 40 [cli_31]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 31 [cli_64]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 64 [cli_57]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 57 [cli_24]: [cli_74]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 74 [cli_52]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 52 [cli_66]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 66 [cli_92]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 92 [cli_88]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 88 [cli_82]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 82 [cli_3]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 3 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 12 [cli_43]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 43 [cli_41]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 41 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 33 [cli_62]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 62 [cli_58]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 58 [cli_26]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 26 [cli_76]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 76 [cli_48]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 48 [cli_67]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 67 [cli_93]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 93 [cli_83]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 83 [cli_16]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 16 [cli_45]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 45 [cli_36]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 36 [cli_10]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 10 [cli_30]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 30 [cli_65]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 65 [cli_56]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 56 [cli_28]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 28 [cli_77]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 77 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 50 [cli_71]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 71 [cli_94]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 94 [cli_85]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 85 [cli_81]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 81 [cli_14]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 14 [cli_46]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 46 [cli_11]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 11 [cli_34]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 34 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 24 [cli_53]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 53 [cli_69]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 69 [cli_95]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 95 [cli_32]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 32 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 15828 RUNNING AT isc17-c15 = EXIT CODE: 255 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:8@isc17-c09] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:912): assert (!closed) failed [proxy:0:8@isc17-c09] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:8@isc17-c09] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:0:9@isc17-c11] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:912): assert (!closed) failed [proxy:0:9@isc17-c11] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:9@isc17-c11] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:0:14@isc17-c18] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:912): assert (!closed) failed [proxy:0:14@isc17-c18] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:14@isc17-c18] main (pm/pmiserv/pmip.c:256): demux engine error waiting for event [mpiexec@isc17-c04] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@isc17-c04] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@isc17-c04] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@isc17-c04] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion tee: write error + /opt/ddn/mvapich/bin/mpiexec -ppn 6 -np 96 -genv MV2_NUM_HCAS 1 -genv MV2_CPU_BINDING_LEVEL core -genv MV2_CPU_BINDING_POLICY scatter --hosts isc17-c01,isc17-c02,isc17-c03,isc17-c04,isc17-c05,isc17-c06,isc17-c07,isc17-c08,isc17-c09,isc17-c11,isc17-c12,isc17-c13,isc17-c14,isc17-c15,isc17-c18,isc17-c22 /esfs/jtacquaviva/git/ime-evaluation/drop_caches.sh + tee -a ./output/COUNT:1#NN:16#PPN:6#API:POSIX#T:102400.txt + /opt/ddn/mvapich/bin/mpiexec -ppn 6 -np 96 -genv MV2_NUM_HCAS 1 -genv MV2_CPU_BINDING_LEVEL core -genv MV2_CPU_BINDING_POLICY scatter --hosts isc17-c01,isc17-c02,isc17-c03,isc17-c04,isc17-c05,isc17-c06,isc17-c07,isc17-c08,isc17-c09,isc17-c11,isc17-c12,isc17-c13,isc17-c14,isc17-c15,isc17-c18,isc17-c22 /esfs/jtacquaviva/software/install/ior/git-ddn/bin/ior -i 3 -s 1 -t 102400 -b 23530045440 -D 120 -a POSIX -F -e -g -z -k -o /esfs/jtacquaviva/indread16/file -r IOR-3.0.1: MPI Coordinated Test of Parallel I/O ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_66]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 66 [cli_60]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 60 [cli_72]: ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_68]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 68 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) [cli_61]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 61 [cli_74]: [cli_70]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 70 [cli_64]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 64 [cli_76]: [cli_12]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 12 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 72 [cli_13]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 13 [cli_1]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 1 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 74 [cli_16]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 16 [cli_53]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 53 [cli_4]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 4 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 76 [cli_17]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 17 [cli_49]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 49 [cli_5]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 5 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_48]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 48 [cli_52]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 52 [cli_0]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_78]: ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_80]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 80 [cli_82]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 82 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 78 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_50]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 50 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_3]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 3 [cli_14]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 14 [cli_81]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 81 [cli_2]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 2 [cli_77]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 77 [cli_15]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 15 [cli_67]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 67 ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_84]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 84 [cli_51]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 51 [cli_65]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 65 [cli_75]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 75 [cli_69]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 69 [cli_86]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 86 [cli_62]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 62 [cli_83]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 83 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) [cli_88]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 88 [cli_63]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 63 [cli_73]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 73 [cli_79]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 79 [cli_71]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 71 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_31]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 31 [cli_37]: [cli_43]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 43 [cli_25]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 25 [cli_7]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 7 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_33]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 33 [cli_39]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 39 [cli_45]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 45 [cli_27]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 27 [cli_9]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 9 [cli_55]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 55 [cli_35]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 35 [cli_41]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 41 [cli_47]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 47 [cli_29]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 29 [cli_11]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 11 [cli_57]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 57 ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) [cli_85]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 85 [cli_30]: [cli_36]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 36 [cli_42]: [cli_24]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 24 [cli_6]: [cli_56]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 56 [cli_87]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 87 [cli_32]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 32 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 37 [cli_44]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 44 [cli_26]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 26 [cli_8]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 8 [cli_58]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 58 [cli_89]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 89 [cli_34]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 34 [cli_38]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 38 [cli_46]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 46 [cli_28]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 28 [cli_10]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 10 [cli_59]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 59 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 30 [cli_40]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 40 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 42 aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 6 [cli_54]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 54 ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 0, Success (ior.c:2293) ior ERROR: block size must be a multiple of transfer size, errno 2, No such file or directory (ior.c:2293) [cli_90]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 90 [cli_92]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 92 [cli_93]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 93 [cli_94]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 94 [cli_95]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 95 [cli_91]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 91 [cli_19]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 19 [cli_23]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 23 [cli_21]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 21 [cli_20]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 20 [cli_22]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 22 [cli_18]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, -1) - process 18 =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 34458 RUNNING AT isc17-c09 = EXIT CODE: 255 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:11@isc17-c13] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:912): assert (!closed) failed [proxy:0:11@isc17-c13] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:11@isc17-c13] main (pm/pmiserv/pmip.c:256): demux engine error waiting for event [proxy:0:7@isc17-c08] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:912): assert (!closed) failed [proxy:0:7@isc17-c08] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:7@isc17-c08] main (pm/pmiserv/pmip.c:256): demux engine error waiting for event [proxy:0:6@isc17-c07] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:912): assert (!closed) failed [proxy:0:6@isc17-c07] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:6@isc17-c07] main (pm/pmiserv/pmip.c:256): demux engine error waiting for event [mpiexec@isc17-c04] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@isc17-c04] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@isc17-c04] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@isc17-c04] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion + set +x /esfs/jtacquaviva/ioperf stripe_count: 32 stripe_size: 1048576 stripe_offset: -1