inspect processes in a container with lsof, strace, ...
by Matthias Apitz
Hello,
We have containers with SuSE Linux SLES 15 SP6 and detected the
following issue: We can not inspect processes as user root, for example:
ebe6737da6e8:~ # ps ax | tail -3
10924 ? Ss 0:00 postgres: sisis sisis 127.0.0.1(35456) idle
11037 pts/0 R+ 0:00 ps ax
11038 pts/0 S+ 0:00 tail -3
ebe6737da6e8:~ # lsof -p 10924 | head -3
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postmaste 10924 postgres cwd unknown /proc/10924/cwd (readlink: Permission denied)
postmaste 10924 postgres rtd unknown /proc/10924/root (readlink: Permission denied)
It only works as the user who owns the process, in this case the PID 10924
the user 'postgres':
ebe6737da6e8:~ # su - postgres
postgres@ebe6737da6e8:~> lsof -p 10924 | head -3
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
postmaste 10924 postgres cwd DIR 0,60 1024 10816610 /data/postgresql151/data
postmaste 10924 postgres rtd DIR 0,60 1024 11934131 /
postgres@ebe6737da6e8:~>
Why 'root' is not allowed to do this?
matthias
--
Matthias Apitz, ✉ guru(a)unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub
Annalena Baerbock: "We are fighting a war against Russia ..." (25.1.2023)
I, Matthias, I am not at war with Russia.
Я не воюю с Россией.
Ich bin nicht im Krieg mit Russland.
10 months, 1 week
Podman candidate v5.3.0-rc3 Released
by Do Not Reply
Hi all,
Podman candidate v5.3.0-rc3 is now available. You may view the full details at
https://github.com/containers/podman/releases/tag/v5.3.0-rc3
Release candidate Notes:
--------------
### Features
- The `podman kube generate` and `podman kube play` commands can now create and run Kubernetes Job YAML ([#17011](https://github.com/containers/podman/issues/17011)).
- The `podman kube generate` command now includes information on the user namespaces for pods and containers in generated YAML. The `podman kube play` command uses this information to duplicate the user namespace configuration when creating new pods based on the YAML.
- The `podman kube play` command now supports Kubernetes volumes of type image ([#23775](https://github.com/containers/podman/issues/23775)).
- The service name of systemd units generated by Quadlet can now be set with the `ServiceName` key in all supported Quadlet files ([#23414](https://github.com/containers/podman/issues/23414)).
- Quadlets can now disable their implicit dependency on `network-online.target` via a new key, `DefaultDependencies`, supported by all Quadlet files ([#24193](https://github.com/containers/podman/issues/24193)).
- Quadlet `.container` and `.pod` files now support a new key, `AddHost`, to add hosts to the container or pod.
- The `PublishPort` key in Quadlet `.container` and `.pod` files can now accept variables in its value ([#24081](https://github.com/containers/podman/issues/24081)).
- Quadlet `.container` files now support two new keys, `CgroupsMode` and `StartWithPod`, to configure cgroups for the container and whether the container will be started with the pod it is part of ([#23664](htt\
ps://github.com/containers/podman/issues/23664) and [#24401](https://github.com/containers/podman/issues/24401)).
- Quadlet `.container` files can now use the network of another container by specifying the `.container` file of the container to share with in the `Network` key.
- Quadlet `.container` files can now mount images managed by `.image` files into the container by using the `Mount=type=image` key with a `.image` target.
- Quadlet `.pod` files now support six new keys, `DNS`, `DNSOption`, `DNSSearch`, `IP`, `IP6`, and `UserNS`, to configure DNS, static IPs, and user namespace settings for the pod ([#23692](https://github.com/co\
ntainers/podman/issues/23692)).
- Quadlet `.image` files can now give an image multiple times by specifying the `ImageTag` key multiple times ([#23781](https://github.com/containers/podman/issues/23781)).
- Quadlets can now be placed in the `/run/containers/systemd` directory as well as existing directories like `$HOME/containers/systemd` and `/etc/containers/systemd/users`.
- Quadlet now properly handles subdirectories of a unit directory being a symlink ([#23755](https://github.com/containers/podman/issues/23755)).
- The `podman manifest inspect` command now includes the manifest's annotations in its output.
- The output of the `podman inspect` command for containers now includes a new field, `HostConfig.AutoRemoveImage`, which shows whether a container was created with the `--rmi` option set.
- The output of the `podman inspect` command for containers now includes a new field, `Config.ExposedPorts`, which includes all exposed ports from the container, improving Docker compatibility.
- The output of the `podman inspect` command for containers now includes a new field, `Config.StartupHealthCheck`, which shows the container's startup healthcheck configuration.
- The `podman machine list` command now supports a new option, `--all-providers`, which lists machines from all supported VM providers, not just the one currently in use.
- VMs run by `podman machine` on Windows will now provide API access by exposing a Unix socket on the host filesystem which forwards into the VM ([#23408](https://github.com/containers/podman/issues/23408)).
- The `podman buildx prune` and `podman image prune` commands now support a new option, `--build-cache`, which will also clean the build cache.
- The Windows installer has a new radio button to select virtualization provider (WSLv2 or Hyper-V).
- The `--add-host` option to `podman create`, `podman run`, and `podman pod create` now supports specifying multiple hostnames, semicolon-separated (e.g. `podman run --add-host test1;test2:192.168.1.1`) ([#2377\
0](https://github.com/containers/podman/issues/23770)).
- The `podman run` and `podman create` commands now support three new options for configuring healthcheck logging: `--health-log-destination` (specify where logs are stored), `--health-max-log-count` (specify how many healthchecks worth of logs are stored), and `--health-max-log-size` (specify the maximum size of the healthcheck log).
### Changes
- Podman now uses the Pasta `--map-guest-addr` option by default which is used for the `host.containers.internal` entry in `/etc/hosts` to allow containers to reach the host by default ([#19213](https://github.com/containers/podman/issues/19213)).
- The names of the infra containers of pods created by Quadlet are changed to the pod name suffixed with `-infra` ([#23665](https://github.com/containers/podman/issues/23665)).
- The `podman system connection add` command now respects HTTP path prefixes specified with `tcp://` URLs.
- Proxy environment variables (e.g. `https_proxy`) declared in `containers.conf` no longer escape special characters in their values when used with `podman machine` VMs ([#23277](https://github.com/containers/p\
odman/issues/23277)).
- The `podman images --sort=repository` command now also sorts by image tag as well, guaranteeing deterministic output ordering ([#23803](https://github.com/containers/podman/issues/23803)).
- When a user has a rootless `podman machine` VM running and second rootful `podman machine` VM initialized, and the rootless VM is removed, the connection to the second, rootful machine now becomes the default as expected ([#22577](https://github.com/containers/podman/issues/22577)).
- Environment variable secrets are no longer contained in the output of `podman inspect` on a container the secret is used in ([#23788](https://github.com/containers/podman/issues/23788)).
- Podman no longer exits 0 on SIGTERM by default.
- Podman no longer explicitly sets rlimits to their default value, as this could lower the actual value available to containers if it had been set higher previously.
- Quadlet user units now correctly wait for the network to be ready to use via a new service, `podman-user-wait-network-online.service`, instead of the user session's nonfunctional `network-online.target`.
- Exposed ports in the output of `podman ps` are now correctly grouped and deduplicated when they are also published ([#23317](https://github.com/containers/podman/issues/23317)).
- Quadlet build units no longer use `RemainAfterExit=yes` by default.
### Bugfixes
- Fixed a bug where the `--build-context` option to `podman build` did not function properly on Windows, breaking compatibility with Visual Studio Dev Containers ([#17313](https://github.com/containers/podman/issues/17313)).
- Fixed a bug where Quadlet would generate bad arguments to Podman if the `SecurityLabelDisable` or `SecurityLabelNested` keys were used ([#23432](https://github.com/containers/podman/issues/23432)).
- Fixed a bug where the `PODMAN_COMPOSE_WARNING_LOGS` environment variable did not suppress warnings printed by `podman compose` that it was redirecting to an external provider.
- Fixed a bug where, if the `podman container cleanup` command was run on a container in the process of being removed, an error could be printed.
- Fixed a bug where rootless Quadlet units placed in `/etc/containers/systemd/users/` would be loaded for root as well when `/etc/containers/systemd` was a symlink ([#23483](https://github.com/containers/podman/issues/23483)).
- Fixed a bug where the remote Podman client's `podman stop` command would, if called with `--cidfile` pointing to a non-existent file and the `--ignore` option set, stop all containers ([#23554](https://github.com/containers/podman/issues/23554)).
- Fixed a bug where the `podman wait` would only exit only after 20 second when run on a container which rapidly exits and is then restarted by the `on-failure` restart policy.
- Fixed a bug where `podman volume rm` and `podman run -v` could deadlock when run simultaneously on the same volume ([#23613](https://github.com/containers/podman/issues/23613)).
- Fixed a bug where running `podman mount` on a container in the process of being created could cause a nonsensical error indicating the container already existed ([#23637](https://github.com/containers/podman/issues/23637)).
- Fixed a bug where the `podman stop` command could deadlock when run on containers with very large annotations ([#22246](https://github.com/containers/podman/issues/22246)).
- Fixed a bug where the `podman machine stop` command could segfault on Mac when a VM failed to stop gracefully ([#23654](https://github.com/containers/podman/issues/23654)).
- Fixed a bug where the `podman stop` command would not ensure containers created with `--rm` were removed when it exited ([#22852](https://github.com/containers/podman/issues/22852)).
- Fixed a bug where the `--rmi` option to `podman run` did not function correctly with detached containers.
- Fixed a bug where running `podman inspect` on a container on FreeBSD would emit an incorrect value for the `HostConfig.Device` field, breaking compatibility with the Ansible Podman module.
- Fixed a bug where rootless Podman could fail to start containers using the `--cgroup-parent` option ([#23780](https://github.com/containers/podman/issues/23780)).
- Fixed a bug where the `podman build -v` command did not properly handle Windows paths passed as the host directory.
- Fixed a bug where Podman could leak network namespace files if it was interrupted while creating a network namespace ([#24044](https://github.com/containers/podman/issues/24044)).
- Fixed a bug where the remote Podman client's `podman run` command could sometimes fail to retrieve a container's exit code for containers run with the `--rm` option.
- Fixed a bug where `podman machine` on Windows could fail to run VMs for certain usernames containing special characters.
- Fixed a bug where Quadlet would reject `RemapUsers=keep-id` when run as root.
- Fixed a bug where XFS quotas on volumes were not unique, meaning that all volumes using a quota shared the same maximum size and inodes (set by the most recent volume with a quota to be created).
- Fixed a bug where `Service` section of Quadlet files would only use defaults and not respect user input ([#24322](https://github.com/containers/podman/issues/24322)).
### API
- The Play API for Kubernetes YAML now supports `application/x-tar` compressed context directories ([#24015](https://github.com/containers/podman/pull/24015)).
- Fixed a bug in the Attach API for Containers (for both Compat and Libpod endpoints) which could cause inconsistent failures due to a race condition ([#23757](https://github.com/containers/podman/issues/23757)).
- Fixed a bug where the output for the Compat Top API for Containers did not properly split the output into an array ([#23981](https://github.com/containers/podman/issues/23981)).
- Fixed a bug where the Info API could fail when running `podman system service` via a socket-activated systemd service ([#24152](https://github.com/containers/podman/issues/24152)).
- Fixed a bug where the Events and Logs endpoints for Containers now send status codes immediately, as opposed to when the first event or log line is sent ([#23712](https://github.com/containers/podman/issues/23712)).
### Misc
- Podman now requires Golang 1.22 or higher to build.
- The output of `podman machine start` has been improved when trying to start a machine when another is already running ([#23436](https://github.com/containers/podman/issues/23436)).
- Quadlet will no longer log spurious ENOENT errors when resolving unit directories ([#23620](https://github.com/containers/podman/issues/23620)).
- The Docker alias shell script will now also honor the presence of `$XDG_CONFIG_HOME/containers/nodocker` when considering whether it should print its warning message that Podman is in use.
- The podman-auto-update systemd unit files have been moved into the `contrib/systemd/system` directory in the repo for consistency with our other unit files.
This message was generated by an automated system. Replies to the sender will bounce, be ignored and discarded.
10 months, 2 weeks
Speeding up podman by using cache
by Ganeshar, Puvi
Hello Podman team,
I am about explore this option so just wanted to check with you all before as I might be wasting my time.
I am in Platform Engineering team at DirecTV, and we run Go and Java pipelines on Jenkins using Amazon EKS as the workers. So, the process is that when a Jenkins build runs, it asks the EKS for a worker (Kubernetes pod) and the cluster would spawn one and the new pod would communicate back to the Jenkins controller. We use the Jenkins Kubernetes pod template to configure the communication. We are currently running the latest LTS of podman, v5.2.2, however still using cgroups-v1 for now, planning to migrate early 2025 by upgrading the cluster to use Amazon Linux 2023 which uses cgroups-v2 as default. Here’s the podman configuration details that we use:
host:
arch: arm64
buildahVersion: 1.37.2
cgroupControllers:
- cpuset
- cpu
- cpuacct
- blkio
- memory
- devices
- freezer
- net_cls
- perf_event
- net_prio
- hugetlb
- pids
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.1.12-1.el9.aarch64
path: /usr/bin/conmon
version: 'conmon version 2.1.12, commit: f174c390e4760883511ab6b5c146dcb244aeb647'
cpuUtilization:
idlePercent: 99.22
systemPercent: 0.37
userPercent: 0.41
cpus: 16
databaseBackend: sqlite
distribution:
distribution: centos
version: "9"
eventLogger: file
freeLocks: 2048
hostname: podmanv5-arm
idMappings:
gidmap: null
uidmap: null
kernel: 5.10.225-213.878.amzn2.aarch64
linkmode: dynamic
logDriver: k8s-file
memFree: 8531066880
memTotal: 33023348736
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.12.1-1.el9.aarch64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.12.1
package: netavark-1.12.2-1.el9.aarch64
path: /usr/libexec/podman/netavark
version: netavark 1.12.2
ociRuntime:
name: crun
package: crun-1.16.1-1.el9.aarch64
path: /usr/bin/crun
version: |-
crun version 1.16.1
commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20240806.gee36266-2.el9.aarch64
version: |
pasta 0^20240806.gee36266-2.el9.aarch64-pasta
Copyright Red Hat
GNU General Public License, version 2 or later
https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: false
path: /run/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.3.1-1.el9.aarch64
version: |-
slirp4netns version 1.3.1
commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 0
swapTotal: 0
uptime: 144h 6m 15.00s (Approximately 6.00 days)
variant: v8
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev,metacopy=on
graphRoot: /var/lib/containers/storage
graphRootAllocated: 107352141824
graphRootUsed: 23986397184
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Supports shifting: "true"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 1
runRoot: /run/containers/storage
transientStore: false
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 5.2.2
Built: 1724331496
BuiltTime: Thu Aug 22 12:58:16 2024
GitCommit: ""
GoVersion: go1.22.5 (Red Hat 1.22.5-2.el9)
Os: linux
OsArch: linux/arm64
Version: 5.2.2
We migrated to podman when Kubernetes deprecated docker and have been using podman for the last two years or so. Its working well, however since we run over 500 builds a day, I am trying to explore whether I can speed up the podman build process by using image caching. I wanted to see if I use an NFS file system (Amazon FSX) as the storage for podman (overlay-fs) would it improve podman performance by the builds completing much faster as of the already downloaded images on the NFS. Currently, podman in each pod on the EKS cluster would download all the required images every time so not taking advantage of the cached images.
These are my concerns:
1. Any race conditions, a podman processes colliding with each other during read and write.
2. Performance of I/O operations as NFS communication will be over the network.
Have any of you tried this method before? If so, can you share any pitfalls that you’ve faced?
Any comments / advice would be beneficial as I need to weigh up pros and cons before spending time on this. Also, if it causes outage due to storage failures it would block all our developers; so, I will have to design this in a way where we can recover quickly.
Thanks very much in advance and have a great day.
Puvi Ganeshar | @pg925u
Principal, Platform Engineer
CICD - Pipeline Express | Toronto
[Image]
10 months, 2 weeks
