podman intel mpi container
I run podman with intel mpi. But there is something wrong. Any help will be appreciated. --GHui $ mpirun --version Intel(R) MPI Library for Linux* OS, Version 2021.5 Build 20211102 (id: 9279b7d62) Copyright 2003-2021, Intel Corporation. $ mpirun -np 2 podman run --env-host --env-file envfile --network=host --pid=host --ipc=host -w /exports -v .:/exports:z centos/ /exports/mpitest [cli_0]: write_line error; fd=9 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=9 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143): MPID_Init(1221)......: MPIR_pmi_init(130)...: PMI_Get_appnum returned -1 [cli_0]: write_line error; fd=9 buf=:cmd=abort exitcode=1090575 : system msg for write_line failure : Bad file descriptor Attempting to use an MPI routine before initializing MPICH [cli_1]: write_line error; fd=10 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_1]: Unable to write to PMI_fd [cli_1]: write_line error; fd=10 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143): MPID_Init(1221)......: MPIR_pmi_init(130)...: PMI_Get_appnum returned -1 [cli_1]: write_line error; fd=10 buf=:cmd=abort exitcode=1090575 : system msg for write_line failure : Bad file descriptor Attempting to use an MPI routine before initializing MPICH ^C[mpiexec@ja0909.para.bscc] Sending Ctrl-C to processes as requested [mpiexec@ja0909.para.bscc] Press Ctrl-C again to force abort
$ mpirun -np 2 podman run --env-host --env-file envfile --network=host --pid=host --ipc=host -w /exports -v .:/exports:z centos/vasp:5.4.4 /exports/mpitest
Hi GHui, Thanks for reaching out. On Thu, Jun 16, 2022 at 11:12 AM None via Podman <podman@lists.podman.io> wrote:
I run podman with intel mpi. But there is something wrong. Any help will be appreciated. --GHui
$ mpirun --version Intel(R) MPI Library for Linux* OS, Version 2021.5 Build 20211102 (id: 9279b7d62) Copyright 2003-2021, Intel Corporation.
$ mpirun -np 2 podman run --env-host --env-file envfile --network=host --pid=host --ipc=host -w /exports -v .:/exports:z centos/ /exports/mpitest
Can you retry using `--userns=keep-id`? The following article elaborates on the details: https://podman.io/blogs/2019/09/26/podman-in-hpc.html Kind regards, Valentin
[cli_0]: write_line error; fd=9 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=9 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143): MPID_Init(1221)......: MPIR_pmi_init(130)...: PMI_Get_appnum returned -1 [cli_0]: write_line error; fd=9 buf=:cmd=abort exitcode=1090575 : system msg for write_line failure : Bad file descriptor Attempting to use an MPI routine before initializing MPICH [cli_1]: write_line error; fd=10 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_1]: Unable to write to PMI_fd [cli_1]: write_line error; fd=10 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143): MPID_Init(1221)......: MPIR_pmi_init(130)...: PMI_Get_appnum returned -1 [cli_1]: write_line error; fd=10 buf=:cmd=abort exitcode=1090575 : system msg for write_line failure : Bad file descriptor Attempting to use an MPI routine before initializing MPICH
^C[mpiexec@ja0909.para.bscc] Sending Ctrl-C to processes as requested [mpiexec@ja0909.para.bscc] Press Ctrl-C again to force abort _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
There is also failed. $ mpirun -np 2 podman run --env-host --env-file envfile --userns=keep-id --network=host --pid=host --ipc=host -w /exports -v .:/exports:z centos/vasp:5.4.4 /exports/mpitest [cli_1]: write_line error; fd=10 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_1]: Unable to write to PMI_fd [cli_1]: write_line error; fd=10 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143): MPID_Init(1221)......: MPIR_pmi_init(130)...: PMI_Get_appnum returned -1 [cli_1]: write_line error; fd=10 buf=:cmd=abort exitcode=1090575 : system msg for write_line failure : Bad file descriptor Attempting to use an MPI routine before initializing MPICH [cli_0]: write_line error; fd=9 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=9 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143): MPID_Init(1221)......: MPIR_pmi_init(130)...: PMI_Get_appnum returned -1 [cli_0]: write_line error; fd=9 buf=:cmd=abort exitcode=1090575 : system msg for write_line failure : Bad file descriptor Attempting to use an MPI routine before initializing MPICH
It looks like mpirun executes podman with extra open file descriptor(s) but they are not available to the container. In this Podman GitHub discussion thread https://github.com/containers/podman/discussions/13254 it was discussed if the option --preserve-fds could be used to pass in extra file descriptors from mpirun to the container. A showstopper was that the file descriptor(s) passed in were not continuous (sequential). On Fri, Jun 17, 2022 at 4:37 AM None via Podman <podman@lists.podman.io> wrote:
There is also failed.
$ mpirun -np 2 podman run --env-host --env-file envfile --userns=keep-id --network=host --pid=host --ipc=host -w /exports -v .:/exports:z centos/vasp:5.4.4 /exports/mpitest [cli_1]: write_line error; fd=10 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_1]: Unable to write to PMI_fd [cli_1]: write_line error; fd=10 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143): MPID_Init(1221)......: MPIR_pmi_init(130)...: PMI_Get_appnum returned -1 [cli_1]: write_line error; fd=10 buf=:cmd=abort exitcode=1090575 : system msg for write_line failure : Bad file descriptor Attempting to use an MPI routine before initializing MPICH [cli_0]: write_line error; fd=9 buf=:cmd=init pmi_version=1 pmi_subversion=1 : system msg for write_line failure : Bad file descriptor [cli_0]: Unable to write to PMI_fd [cli_0]: write_line error; fd=9 buf=:cmd=get_appnum : system msg for write_line failure : Bad file descriptor Abort(1090575) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(143): MPID_Init(1221)......: MPIR_pmi_init(130)...: PMI_Get_appnum returned -1 [cli_0]: write_line error; fd=9 buf=:cmd=abort exitcode=1090575 : system msg for write_line failure : Bad file descriptor Attempting to use an MPI routine before initializing MPICH _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-... In this article, it says $ mpirun -n <number-of-processes> -ppn <processes-per-node> -hostlist <hosts> singularity exec <container-name> <application> But there is no option to set PMI_FD. I think it cannot work.
participants (3)
-
Erik Sjölund -
ugiwgh＠qq.com -
Valentin Rothberg