journald log driver not working
Hello Everyone, I'm testing out using journald for logging from one of my containers. I think I configured it correctly, but I don't see anything in the path I passed to it. Here's the create command I used: CreateCommand": [ "podman", "run", "-d", "--name", "postgrey", "--pod", "mailman", "--volume", "postgrey:/var/spool/postfix/postgrey:Z", "--log-driver", "journald", "--log-opt", "path=/var/log/containers/postgrey.log", "postgrey" ], /var/log/containers is empty: # ls -l /var/log/containers/ total 0 Docker's docs on retrieving the logs with journalctl don't show me anything: journalctl CONTAINER_NAME=postgrey -- Logs begin at Wed 2022-01-19 17:04:20 EST, end at Tue 2022-01-25 18:40:27 EST. -- -- No entries -- Incidentally, I used podman to crate the systemd unit files for the pod and dumped them into /etc/systemd/system. I'm starting, stopping, etc the pod using systemctl. Using journald for logging means the container's logs get written to the host's journal, right? If that's true, what am I doing wrong? -- Ranbir
I played with the log drivers (I’m on a very old Podman … 1.6.2). What a found was journald didn’t use the file path at all. I used a less specific retrieval (along the lines of just journalctl). It wasn’t satisfactory to me. Instead, I chose the k8s driver. Despite the implications that this is somehow JSON-formatted output, it wasn’t. It wrote my output to the file I requested as plain text. -- Jim Melton
On Jan 25, 2022, at 4:55 PM, Ranbir <m3freak@thesandhufamily.ca> wrote:
Hello Everyone,
I'm testing out using journald for logging from one of my containers. I think I configured it correctly, but I don't see anything in the path I passed to it.
Here's the create command I used:
CreateCommand": [ "podman", "run", "-d", "--name", "postgrey", "--pod", "mailman", "--volume", "postgrey:/var/spool/postfix/postgrey:Z", "--log-driver", "journald", "--log-opt", "path=/var/log/containers/postgrey.log", "postgrey" ],
/var/log/containers is empty:
# ls -l /var/log/containers/ total 0
Docker's docs on retrieving the logs with journalctl don't show me anything:
journalctl CONTAINER_NAME=postgrey -- Logs begin at Wed 2022-01-19 17:04:20 EST, end at Tue 2022-01-25 18:40:27 EST. -- -- No entries --
Incidentally, I used podman to crate the systemd unit files for the pod and dumped them into /etc/systemd/system. I'm starting, stopping, etc the pod using systemctl.
Using journald for logging means the container's logs get written to the host's journal, right? If that's true, what am I doing wrong?
-- Ranbir _______________________________________________ Podman mailing list -- podman@lists.podman.io To unsubscribe send an email to podman-leave@lists.podman.io
On Tue, 2022-01-25 at 21:46 -0700, Jim Melton wrote:
I played with the log drivers (I’m on a very old Podman … 1.6.2). What a found was journald didn’t use the file path at all. I used a less specific retrieval (along the lines of just journalctl). It wasn’t satisfactory to me. Instead, I chose the k8s driver. Despite the implications that this is somehow JSON-formatted output, it wasn’t. It wrote my output to the file I requested as plain text.
I tried setting log-driver to "k8s-file" and a file was created in the location I set it to, but the file is empty. It's been about 3 hours since I did it and it's still empty. :/ -- Ranbir
On 1/26/22 21:09, Ranbir wrote:
On Tue, 2022-01-25 at 21:46 -0700, Jim Melton wrote:
I played with the log drivers (I’m on a very old Podman … 1.6.2). What a found was journald didn’t use the file path at all. I used a less specific retrieval (along the lines of just journalctl). It wasn’t satisfactory to me. Instead, I chose the k8s driver. Despite the implications that this is somehow JSON-formatted output, it wasn’t. It wrote my output to the file I requested as plain text. I tried setting log-driver to "k8s-file" and a file was created in the location I set it to, but the file is empty. It's been about 3 hours since I did it and it's still empty. :/
I think you are hitting very old bugs which will require you to update the podman package.
On Thu, 2022-01-27 at 08:08 -0500, Daniel Walsh wrote:
I think you are hitting very old bugs which will require you to update the podman package.
I'm running podman 3.3.1 on a Rocky Linux 8.5 host. I wanted to stay with the version included in the distribution. Should I move to the latest release from upstream (i.e. the podman project)? I suppose my other alternative is to just live with current logging situation. -- Ranbir
On 1/27/22 11:45, Ranbir wrote:
On Thu, 2022-01-27 at 08:08 -0500, Daniel Walsh wrote:
I think you are hitting very old bugs which will require you to update the podman package. I'm running podman 3.3.1 on a Rocky Linux 8.5 host. I wanted to stay with the version included in the distribution. Should I move to the latest release from upstream (i.e. the podman project)?
I suppose my other alternative is to just live with current logging situation.
No that should be new enough, I was going by a comment earlier on Podman 1.6. It could be a bug in that version, but it would be nice if you could try on the latest version. podman 3.4 should be showing up in RHEL8.5 in the next week or two, don't know how quickly it will show up in Rocky.
On Thu, 2022-01-27 at 13:38 -0500, Daniel Walsh wrote:
It could be a bug in that version, but it would be nice if you could try on the latest version.
I haven't been able to try on the latest upstream release.
podman 3.4 should be showing up in RHEL8.5 in the next week or two, don't know how quickly it will show up in Rocky.
I did try the latest version available on CentOS Stream: 3.4.1-dev. I tried setting journald as the log driver and gave the log path, but no file was created. When set to k8s-file, the file _does_ get created and written to. In addition, the log file looks exactly (minus the journald time stamps) like it does when flexget writes its own log. On a hunch, I checked the journal for a different container that's running sabnzbd and noticed its logs were in there. Yet, I hadn't specified anything for it. Is journald the default log driver? Anyway, the log looked the same, although the text colour was all red, which I've seen in the journal before for errors. Can this be changed? I switched the flexget container back to not specifying a log driver which sent its logs to the journal. But, this time the logs looked nothing like the k8s-file log file that was being written to before. It's actually missing information. See below for the comparisons. Here's a raw log file written by flexget itself: 2022-02-01 12:34:03 INFO task tv_shows Rerunning the task in case better resolution can be achieved. 2022-02-01 12:34:03 INFO task premieres Plugin next_series_episodes has requested task to be ran again after execution has completed. Reason: Look for next season 2022-02-01 12:34:04 INFO task premieres Rerunning the task in case better resolution can be achieved. Here's the contents of the k8s-file (not much activity before I changed it): 2022-01-31T23:35:54.992078575-05:00 stdout F 2022-01-31 23:35:54 INFO scheduler Starting scheduler 2022-01-31T23:39:35.081444166-05:00 stdout F 2022-01-31 23:39:35 INFO manager Got SIGTERM. Shutting down. Here's the journal for the same container: $ sudo journalctl CONTAINER_NAME=flexget -n 5 -- Logs begin at Fri 2022-01-28 12:23:01 EST, end at Tue 2022-02-01 12:46:37 EST. -- Feb 01 12:34:02 containment01 conmon[31764]: [172B blob data] Feb 01 12:34:03 containment01 conmon[31764]: [122B blob data] Feb 01 12:34:03 containment01 conmon[31764]: [172B blob data] Feb 01 12:34:03 containment01 conmon[31764]: [183B blob data] Feb 01 12:34:04 containment01 conmon[31764]: [122B blob data] Where'd the rest of the information go? -- Ranbir
On 2/1/22 12:48, Ranbir wrote:
On Thu, 2022-01-27 at 13:38 -0500, Daniel Walsh wrote:
It could be a bug in that version, but it would be nice if you could try on the latest version. I haven't been able to try on the latest upstream release.
podman 3.4 should be showing up in RHEL8.5 in the next week or two, don't know how quickly it will show up in Rocky. I did try the latest version available on CentOS Stream: 3.4.1-dev. I tried setting journald as the log driver and gave the log path, but no file was created.
When set to k8s-file, the file _does_ get created and written to. In addition, the log file looks exactly (minus the journald time stamps) like it does when flexget writes its own log.
On a hunch, I checked the journal for a different container that's running sabnzbd and noticed its logs were in there. Yet, I hadn't specified anything for it. Is journald the default log driver? It is going forward but not for RHEL8, because of issues on systemd and journal not being available to rootless users. Anyway, the log looked the same, although the text colour was all red, which I've seen in the journal before for errors. Can this be changed?
I switched the flexget container back to not specifying a log driver which sent its logs to the journal. But, this time the logs looked nothing like the k8s-file log file that was being written to before. It's actually missing information. See below for the comparisons.
Here's a raw log file written by flexget itself:
2022-02-01 12:34:03 INFO task tv_shows Rerunning the task in case better resolution can be achieved. 2022-02-01 12:34:03 INFO task premieres Plugin next_series_episodes has requested task to be ran again after execution has completed. Reason: Look for next season 2022-02-01 12:34:04 INFO task premieres Rerunning the task in case better resolution can be achieved.
Here's the contents of the k8s-file (not much activity before I changed it):
2022-01-31T23:35:54.992078575-05:00 stdout F 2022-01-31 23:35:54 INFO scheduler Starting scheduler 2022-01-31T23:39:35.081444166-05:00 stdout F 2022-01-31 23:39:35 INFO manager Got SIGTERM. Shutting down.
Here's the journal for the same container:
$ sudo journalctl CONTAINER_NAME=flexget -n 5 -- Logs begin at Fri 2022-01-28 12:23:01 EST, end at Tue 2022-02-01 12:46:37 EST. -- Feb 01 12:34:02 containment01 conmon[31764]: [172B blob data] Feb 01 12:34:03 containment01 conmon[31764]: [122B blob data] Feb 01 12:34:03 containment01 conmon[31764]: [172B blob data] Feb 01 12:34:03 containment01 conmon[31764]: [183B blob data] Feb 01 12:34:04 containment01 conmon[31764]: [122B blob data]
Where'd the rest of the information go?
participants (3)
-
Daniel Walsh -
Jim Melton -
Ranbir