Speeding up podman by using cache
by Ganeshar, Puvi
Hello Podman team,
I am about explore this option so just wanted to check with you all before as I might be wasting my time.
I am in Platform Engineering team at DirecTV, and we run Go and Java pipelines on Jenkins using Amazon EKS as the workers. So, the process is that when a Jenkins build runs, it asks the EKS for a worker (Kubernetes pod) and the cluster would spawn one and the new pod would communicate back to the Jenkins controller. We use the Jenkins Kubernetes pod template to configure the communication. We are currently running the latest LTS of podman, v5.2.2, however still using cgroups-v1 for now, planning to migrate early 2025 by upgrading the cluster to use Amazon Linux 2023 which uses cgroups-v2 as default. Here’s the podman configuration details that we use:
host:
arch: arm64
buildahVersion: 1.37.2
cgroupControllers:
- cpuset
- cpu
- cpuacct
- blkio
- memory
- devices
- freezer
- net_cls
- perf_event
- net_prio
- hugetlb
- pids
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.1.12-1.el9.aarch64
path: /usr/bin/conmon
version: 'conmon version 2.1.12, commit: f174c390e4760883511ab6b5c146dcb244aeb647'
cpuUtilization:
idlePercent: 99.22
systemPercent: 0.37
userPercent: 0.41
cpus: 16
databaseBackend: sqlite
distribution:
distribution: centos
version: "9"
eventLogger: file
freeLocks: 2048
hostname: podmanv5-arm
idMappings:
gidmap: null
uidmap: null
kernel: 5.10.225-213.878.amzn2.aarch64
linkmode: dynamic
logDriver: k8s-file
memFree: 8531066880
memTotal: 33023348736
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.12.1-1.el9.aarch64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.12.1
package: netavark-1.12.2-1.el9.aarch64
path: /usr/libexec/podman/netavark
version: netavark 1.12.2
ociRuntime:
name: crun
package: crun-1.16.1-1.el9.aarch64
path: /usr/bin/crun
version: |-
crun version 1.16.1
commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20240806.gee36266-2.el9.aarch64
version: |
pasta 0^20240806.gee36266-2.el9.aarch64-pasta
Copyright Red Hat
GNU General Public License, version 2 or later
https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: false
path: /run/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.3.1-1.el9.aarch64
version: |-
slirp4netns version 1.3.1
commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 0
swapTotal: 0
uptime: 144h 6m 15.00s (Approximately 6.00 days)
variant: v8
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mountopt: nodev,metacopy=on
graphRoot: /var/lib/containers/storage
graphRootAllocated: 107352141824
graphRootUsed: 23986397184
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Supports shifting: "true"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 1
runRoot: /run/containers/storage
transientStore: false
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 5.2.2
Built: 1724331496
BuiltTime: Thu Aug 22 12:58:16 2024
GitCommit: ""
GoVersion: go1.22.5 (Red Hat 1.22.5-2.el9)
Os: linux
OsArch: linux/arm64
Version: 5.2.2
We migrated to podman when Kubernetes deprecated docker and have been using podman for the last two years or so. Its working well, however since we run over 500 builds a day, I am trying to explore whether I can speed up the podman build process by using image caching. I wanted to see if I use an NFS file system (Amazon FSX) as the storage for podman (overlay-fs) would it improve podman performance by the builds completing much faster as of the already downloaded images on the NFS. Currently, podman in each pod on the EKS cluster would download all the required images every time so not taking advantage of the cached images.
These are my concerns:
1. Any race conditions, a podman processes colliding with each other during read and write.
2. Performance of I/O operations as NFS communication will be over the network.
Have any of you tried this method before? If so, can you share any pitfalls that you’ve faced?
Any comments / advice would be beneficial as I need to weigh up pros and cons before spending time on this. Also, if it causes outage due to storage failures it would block all our developers; so, I will have to design this in a way where we can recover quickly.
Thanks very much in advance and have a great day.
Puvi Ganeshar | @pg925u
Principal, Platform Engineer
CICD - Pipeline Express | Toronto
[Image]
4 weeks, 1 day
Podman candidate v5.3.0-rc2 Released
by Do Not Reply
Hi all,
Podman candidate v5.3.0-rc2 is now available. You may view the full details at
https://github.com/containers/podman/releases/tag/v5.3.0-rc2
Release candidate Notes:
--------------
This is the second release candidate for Podman v5.3.0. Preliminary release notes are below.
### Features
- The `podman kube generate` and `podman kube play` commands can now create and run Kubernetes Job YAML ([#17011](https://github.com/containers/podman/issues/17011)).
- The `podman kube generate` command now includes information on the user namespaces for pods and containers in generated YAML. The `podman kube play` command uses this information to duplicate the user namespace configuration when creating new pods based on the YAML.
- The `podman kube play` command now supports Kubernetes volumes of type image ([#23775](https://github.com/containers/podman/issues/23775)).
- The service name of systemd units generated by Quadlet can now be set with the `ServiceName` key in all supported Quadlet files ([#23414](https://github.com/containers/podman/issues/23414)).
- Quadlets can now disable their implicit dependency on `network-online.target` via a new key, `DefaultDependencies`, supported by all Quadlet files ([#24193](https://github.com/containers/podman/issues/24193)).
- Quadlet `.container` and `.pod` files now support a new key, `AddHost`, to add hosts to the container or pod.
- The `PublishPort` key in Quadlet `.container` and `.pod` files can now accept variables in its value ([#24081](https://github.com/containers/podman/issues/24081)).
- Quadlet `.container` files now support a new key, `CgroupsMode`, to configure cgroups for the container ([#23664](https://github.com/containers/podman/issues/23664)).
- Quadlet `.container` files can now use the network of another container by specifying the `.container` file of the container to share with in the `Network` key.
- Quadlet `.pod` files now support six new keys, `DNS`, `DNSOption`, `DNSSearch`, `IP`, `IP6`, and `UserNS`, to configure DNS, static IPs, and user namespace settings for the pod ([#23692](https://github.com/containers/podman/issues/23692)).
- Quadlet `.image` files can now give an image multiple times by specifying the `ImageTag` key multiple times ([#23781](https://github.com/containers/podman/issues/23781)).
- Quadlets can now be placed in the `/run/containers/systemd` directory as well as existing directories like `$HOME/containers/systemd` and `/etc/containers/systemd/users`.
- Quadlet now properly handles subdirectories of a unit directory being a symlink ([#23755](https://github.com/containers/podman/issues/23755)).
- The `podman manifest inspect` command now includes the manifest's annotations in its output.
- The output of the `podman inspect` command for containers now includes a new field, `HostConfig.AutoRemoveImage`, which shows whether a container was created with the `--rmi` option set.
- The output of the `podman inspect` command for containers now includes a new field, `Config.ExposedPorts`, which includes all exposed ports from the container, improving Docker compatibility.
- The output of the `podman inspect` command for containers now includes a new field, `Config.StartupHealthCheck`, which shows the container's startup healthcheck configuration.
- The `podman machine list` command now supports a new option, `--all-providers`, which lists machines from all supported VM providers, not just the one currently in use.
- VMs run by `podman machine` on Windows will now provide API access by exposing a Unix socket on the host filesystem which forwards into the VM ([#23408](https://github.com/containers/podman/issues/23408)).
- The `podman buildx prune` and `podman image prune` commands now support a new option, `--build-cache`, which will also clean the build cache.
- The Windows installer has a new radio button to select virtualization provider (WSLv2 or Hyper-V).
- The `--add-host` option to `podman create`, `podman run`, and `podman pod create` now supports specifying multiple hostnames, semicolon-separated (e.g. `podman run --add-host test1;test2:192.168.1.1`) ([#23770](https://github.com/containers/podman/issues/23770)).
- The `podman run` and `podman create` commands now support three new options for configuring healthcheck logging: `--health-log-destination` (specify where logs are stored), `--health-max-log-count` (specify how many healthchecks worth of logs are stored), and `--health-max-log-size` (specify the maximum size of the healthcheck log).
### Changes
- Podman now uses the Pasta `--map-guest-addr` option by default which is used for the `host.containers.internal` entry in `/etc/hosts` to allow containers to reach the host by default ([#19213](https://github.com/containers/podman/issues/19213)).
- The names of the infra containers of pods created by Quadlet are changed to the pod name suffixed with `-infra` ([#23665](https://github.com/containers/podman/issues/23665)).
- The `podman system connection add` command now respects HTTP path prefixes specified with `tcp://` URLs.
- Proxy environment variables (e.g. `https_proxy`) declared in `containers.conf` no longer escape special characters in their values when used with `podman machine` VMs ([#23277](https://github.com/containers/podman/issues/23277)).
- The `podman images --sort=repository` command now also sorts by image tag as well, guaranteeing deterministic output ordering ([#23803](https://github.com/containers/podman/issues/23803)).
- When a user has a rootless `podman machine` VM running and second rootful `podman machine` VM initialized, and the rootless VM is removed, the connection to the second, rootful machine now becomes the default as expected ([#22577](https://github.com/containers/podman/issues/22577)).
- Environment variable secrets are no longer contained in the output of `podman inspect` on a container the secret is used in ([#23788](https://github.com/containers/podman/issues/23788)).
- Podman no longer exits 0 on SIGTERM by default.
- Podman no longer explicitly sets rlimits to their default value, as this could lower the actual value available to containers if it had been set higher previously.
- Quadlet user units now correctly wait for the network to be ready to use via a new service, `podman-user-wait-network-online.service`, instead of the user session's nonfunctional `network-online.target`.
- Exposed ports in the output of `podman ps` are now correctly grouped and deduplicated when they are also published ([#23317](https://github.com/containers/podman/issues/23317)).
### Bugfixes
- Fixed a bug where the `--build-context` option to `podman build` did not function properly on Windows, breaking compatibility with Visual Studio Dev Containers ([#17313](https://github.com/containers/podman/issues/17313)).
- Fixed a bug where Quadlet would generate bad arguments to Podman if the `SecurityLabelDisable` or `SecurityLabelNested` keys were used ([#23432](https://github.com/containers/podman/issues/23432)).
- Fixed a bug where the `PODMAN_COMPOSE_WARNING_LOGS` environment variable did not suppress warnings printed by `podman compose` that it was redirecting to an external provider.
- Fixed a bug where, if the `podman container cleanup` command was run on a container in the process of being removed, an error could be printed.
- Fixed a bug where rootless Quadlet units placed in `/etc/containers/systemd/users/` would be loaded for root as well when `/etc/containers/systemd` was a symlink ([#23483](https://github.com/containers/podman/issues/23483)).
- Fixed a bug where the remote Podman client's `podman stop` command would, if called with `--cidfile` pointing to a non-existent file and the `--ignore` option set, stop all containers ([#23554](https://github.com/containers/podman/issues/23554)).
- Fixed a bug where the `podman wait` would only exit only after 20 second when run on a container which rapidly exits and is then restarted by the `on-failure` restart policy.
- Fixed a bug where `podman volume rm` and `podman run -v` could deadlock when run simultaneously on the same volume ([#23613](https://github.com/containers/podman/issues/23613)).
- Fixed a bug where running `podman mount` on a container in the process of being created could cause a nonsensical error indicating the container already existed ([#23637](https://github.com/containers/podman/issues/23637)).
- Fixed a bug where the `podman stop` command could deadlock when run on containers with very large annotations ([#22246](https://github.com/containers/podman/issues/22246)).
- Fixed a bug where the `podman machine stop` command could segfault on Mac when a VM failed to stop gracefully ([#23654](https://github.com/containers/podman/issues/23654)).
- Fixed a bug where the `podman stop` command would not ensure containers created with `--rm` were removed when it exited ([#22852](https://github.com/containers/podman/issues/22852)).
- Fixed a bug where the `--rmi` option to `podman run` did not function correctly with detached containers.
- Fixed a bug where running `podman inspect` on a container on FreeBSD would emit an incorrect value for the `HostConfig.Device` field, breaking compatibility with the Ansible Podman module.
- Fixed a bug where rootless Podman could fail to start containers using the `--cgroup-parent` option ([#23780](https://github.com/containers/podman/issues/23780)).
- Fixed a bug where the `podman build -v` command did not properly handle Windows paths passed as the host directory.
- Fixed a bug where Podman could leak network namespace files if it was interrupted while creating a network namespace ([#24044](https://github.com/containers/podman/issues/24044)).
- Fixed a bug where the remote Podman client's `podman run` command could sometimes fail to retrieve a container's exit code for containers run with the `--rm` option.
- Fixed a bug where `podman machine` on Windows could fail to run VMs for certain usernames containing special characters.
- Fixed a bug where Quadlet would reject `RemapUsers=keep-id` when run as root.
- Fixed a bug where XFS quotas on volumes were not unique, meaning that all volumes using a quota shared the same maximum size and inodes (set by the most recent volume with a quota to be created).
### API
- The Play API for Kubernetes YAML now supports `application/x-tar` compressed context directories ([#24015](https://github.com/containers/podman/pull/24015)).
- Fixed a bug in the Attach API for Containers (for both Compat and Libpod endpoints) which could cause inconsistent failures due to a race condition ([#23757](https://github.com/containers/podman/issues/23757)).
- Fixed a bug where the output for the Compat Top API for Containers did not properly split the output into an array ([#23981](https://github.com/containers/podman/issues/23981)).
- Fixed a bug where the Info API could fail when running `podman system service` via a socket-activated systemd service ([#24152](https://github.com/containers/podman/issues/24152)).
### Misc
- Podman now requires Golang 1.22 or higher to build.
- The output of `podman machine start` has been improved when trying to start a machine when another is already running ([#23436](https://github.com/containers/podman/issues/23436)).
- Quadlet will no longer log spurious ENOENT errors when resolving unit directories ([#23620](https://github.com/containers/podman/issues/23620)).
- The Docker alias shell script will now also honor the presence of `$XDG_CONFIG_HOME/containers/nodocker` when considering whether it should print its warning message that Podman is in use.
- The podman-auto-update systemd unit files have been moved into the `contrib/systemd/system` directory in the repo for consistency with our other unit files.
This message was generated by an automated system. Replies to the sender will bounce, be ignored and discarded.
1 month
Server down
by Morris, Adam (DELJIS)
Hello.
When running "podman machine init" is says the host server (140.82.113.3:443) is failing to respond:
[cid:765f8e93-8810-40ad-bd6d-9d2f9ba12c70]
Can you please resolve?
Thank you.
Regards.
-Adam
1 month
Rootless Networking
by openbidaaz
Hello,
I have setup 3 rootless containers in a pod and now want to expose ports on two containers to the outside world.
I have tried:
podman pod create \
--name "${ABCPOD}" \
--label "${ABCPOD}" \
--network slirp4netns:port_handler=slirp4netns \
--publish 8080:8080
but now the container exposing port 8080 is failing to get created.
How do I create rootless pods and then expose the networking?
Thank you in advance.
Regards
S
Sent with [Proton Mail](https://proton.me/mail/home) secure email.
1 month
Rootless Networking
by openbidaaz
Hello,
I have setup 3 rootless containers in a pod and now want to expose ports on two containers to the outside world.
I have tried:
podman pod create \
--name "${ABCPOD}" \
--label "${ABCPOD}" \
--network slirp4netns:port_handler=slirp4netns \
--publish 8080:8080
but now the container exposing port 8080 is failing to get created.
How do I create rootless pods and then expose the networking?
Thank you in advance.
Regards
S
1 month, 1 week
dangling images do not go away with 'podman image prune'
by Matthias Apitz
Hello,
I see a lot of so called dangling images with
$ podman images | more
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/sles15-sp6-v73 latest fcdbe8e4e245 21 hours ago 15.3 GB
<none> <none> e84f83f1637c 4 days ago 20.2 GB
<none> <none> d3921b019185 4 days ago 18.3 GB
...
$ podman images | grep '<none>' | wc -l
34
Yesterday, the number was even 57, then I did 'podman image prune' and
even 'podman image prune -a', the latter with the effect that I have had
to re-build my container 'localhost/sles15-sp6-v73'. But even after the
'-a' the dangling ones did not went away completely.
This causes a serious disk space problem. I run the 'podman images' into
a file, edited it a bit to sum up the used space with bc(1):
$ cat sum-space-with-bc.sh
echo '15300+ 20200+ 18300+ 13700+ 13600+ 13600+ 13300+ 13300+ 7670+ 7650+ 7630+ 7130+ 4330+ 4240+ 4240+ 4240+ 1530+ 568+ 568+ 568+ 568+ 568+ 568+ 568+ 7150+ 7600+ 7510+ 7200+ 6230+ 8840+ 5850+ 5520+ 5520+ 5520+ 5520+ 5520+ 8360+ 130' | bc
$ sh sum-space-with-bc.sh
260406
This number 260 GB matches nearly what I do see in the file system:
# du -sh storage/*
1.0K storage/defaultNetworkBackend
1.3M storage/libpod
266G storage/overlay
^^^^
2.1M storage/overlay-containers
8.3M storage/overlay-images
71M storage/overlay-layers
1.0K storage/storage.lock
0 storage/userns.lock
The podman version is 4.7.2 and the OS is SuSE SLES 15 SP6.
Should I file an issue in https://github.com/containers/podman/issues ?
Thanks
matthias
--
Matthias Apitz, ✉ guru(a)unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub
Annalena Baerbock: "We are fighting a war against Russia ..." (25.1.2023)
I, Matthias, I am not at war with Russia.
Я не воюю с Россией.
Ich bin nicht im Krieg mit Russland.
1 month, 1 week
podman && modify images
by Matthias Apitz
Hello,
We build our image based on the Dockerfile with something like
podman build -t sles15-sp6-v73 sles15-sp6-v73
and run it with
podman run -d -p 2022:22 ....sles15-sp6-v73
Building means that based on the Dockerfile we install the software
which should be come up on container start. This works all fine.
When we now add, install or configure something in addition in
the running container, all this gets lost on killing the container
and restart it again with 'podman run...'
Is it somehow possible to write this modified container to an image and
start this again, and not the image which was build with the Dockerfile?
Thanks
matthias
--
Matthias Apitz, ✉ guru(a)unixarea.de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub
1 month, 1 week
