8:59 a.m.
Detected one or more Cirrus-CI cron-triggered jobs have failed recently:
Cron build 'Periodic_release-1.15' Failed:
https://cirrus-ci.com/build/5927430822035456
Cron build 'Periodic_release-1.16' Failed:
https://cirrus-ci.com/build/6655112132165632
Cron build 'Periodic_release-1.17' Failed:
https://cirrus-ci.com/build/5988418619441152
Cron build 'Periodic_release-1.18' Failed:
https://cirrus-ci.com/build/4960200965750784
Cron build 'Periodic_release-1.19' Failed:
https://cirrus-ci.com/build/4824066273050624
# Source: check_cirrus_cron workflow on containers/buildah.
This message was generated by an automated system. Replies to the sender will bounce, be
ignored and discarded.
9:16 a.m.
New subject: Cirrus-CI cron build failures on containers/buildah
On 5/14/21 9:59 AM, Do Not Reply wrote:
Detected one or more Cirrus-CI cron-triggered jobs have failed
recently:
Cron build 'Periodic_release-1.15' Failed:
https://cirrus-ci.com/build/5927430822035456
Cron build 'Periodic_release-1.16' Failed:
https://cirrus-ci.com/build/6655112132165632
Cron build 'Periodic_release-1.17' Failed:
https://cirrus-ci.com/build/5988418619441152
Cron build 'Periodic_release-1.18' Failed:
https://cirrus-ci.com/build/4960200965750784
Cron build 'Periodic_release-1.19' Failed:
https://cirrus-ci.com/build/4824066273050624
All,
This was expected, and we will probably more until the problems are
fixed. I'm thinking we should either not run daily builds on all those
branches, or trim the set of tests down to the basic "Does Buildah
Compile" (like how we do for podman). Thoughts?
--
Chris Evich (he/him), RHCA III
Senior Quality Assurance Engineer
If it ain't broke, yain't tryin' hard 'nough.
12:33 p.m.
New subject: Cirrus-CI cron build failures on containers/buildah
I think we should do a daily full test on the most recent release branch
(besides main), so I'd keep running on 1.19 for now. I don't like the
idea of not running at least a daily test on any branch, especially
recent releases. The 1.16 branch in in RHEL 8.3.1 and we'll be
supporting that for a while yet.
So all said, full dailies on 1.19, trimmed down dailies on 1.16, 1.17
and 1.18. For 1.15 I'd still like to run basic tests on it, but if
that's failing now, I'd be fine with dropping that.
Dan/Nalin, differing opinions?
t
On 5/14/21 10:16 AM, Chris Evich wrote:
On 5/14/21 9:59 AM, Do Not Reply wrote:
> Detected one or more Cirrus-CI cron-triggered jobs have failed recently:
>
> Cron build 'Periodic_release-1.15' Failed:
> https://cirrus-ci.com/build/5927430822035456
> Cron build 'Periodic_release-1.16' Failed:
> https://cirrus-ci.com/build/6655112132165632
> Cron build 'Periodic_release-1.17' Failed:
> https://cirrus-ci.com/build/5988418619441152
> Cron build 'Periodic_release-1.18' Failed:
> https://cirrus-ci.com/build/4960200965750784
> Cron build 'Periodic_release-1.19' Failed:
> https://cirrus-ci.com/build/4824066273050624
All,
This was expected, and we will probably more until the problems are
fixed. I'm thinking we should either not run daily builds on all
those branches, or trim the set of tests down to the basic "Does
Buildah Compile" (like how we do for podman). Thoughts?
3:03 p.m.
New subject: Cirrus-CI cron build failures on containers/buildah
All,
So having looked through the results from a few days, it seems
the most commonly failing task is is the 'conformance' test, e.g.:
From 1.19 branch:
https://cirrus-ci.com/task/4834266954596352
If someone could confirm, I think this problem might be caused by always
installing and testing against the "latest" upstream 'docker'. That
task also installs a slew of other packages at runtime...which has
always been a recipe for disaster. It's a constantly shifting
environment, despite stable buildah release bits.
Would it be okay if we do as Tom suggests below, but also disable the
'conformance' task on the release branches?
On 5/14/21 1:33 PM, Tom Sweeney wrote:
I think we should do a daily full test on the most recent release
branch
(besides main), so I'd keep running on 1.19 for now. I don't like the
idea of not running at least a daily test on any branch, especially
recent releases. The 1.16 branch in in RHEL 8.3.1 and we'll be
supporting that for a while yet.
So all said, full dailies on 1.19, trimmed down dailies on 1.16, 1.17
and 1.18. For 1.15 I'd still like to run basic tests on it, but if
that's failing now, I'd be fine with dropping that.
Dan/Nalin, differing opinions?
t
On 5/14/21 10:16 AM, Chris Evich wrote:
> On 5/14/21 9:59 AM, Do Not Reply wrote:
>> Detected one or more Cirrus-CI cron-triggered jobs have failed recently:
>>
>> Cron build 'Periodic_release-1.15' Failed:
>> https://cirrus-ci.com/build/5927430822035456
>> Cron build 'Periodic_release-1.16' Failed:
>> https://cirrus-ci.com/build/6655112132165632
>> Cron build 'Periodic_release-1.17' Failed:
>> https://cirrus-ci.com/build/5988418619441152
>> Cron build 'Periodic_release-1.18' Failed:
>> https://cirrus-ci.com/build/4960200965750784
>> Cron build 'Periodic_release-1.19' Failed:
>> https://cirrus-ci.com/build/4824066273050624
>
> All,
>
> This was expected, and we will probably more until the problems are
> fixed. I'm thinking we should either not run daily builds on all
> those branches, or trim the set of tests down to the basic "Does
> Buildah Compile" (like how we do for podman). Thoughts?
>
_______________________________________________
Podman-monitor mailing list -- podman-monitor(a)lists.podman.io
To unsubscribe send an email to podman-monitor-leave(a)lists.podman.io
--
Chris Evich (he/him), RHCA III
Senior Quality Assurance Engineer
If it ain't broke, yain't tryin' hard 'nough.
8:40 a.m.
New subject: Cirrus-CI cron build failures on containers/buildah
On Tue, May 18, 2021 at 04:03:18PM -0400, Chris Evich wrote:
Would it be okay if we do as Tom suggests below, but also disable
the
'conformance' task on the release branches?
Knowing that the behavior of docker build changes subtly between
versions, is it not possible to keep testing against the same version of
dockerd on a given branch? Otherwise, we can miss breakage that comes
in via backports.
Nalin
10:21 a.m.
New subject: Cirrus-CI cron build failures on containers/buildah
On 5/19/21 9:40 AM, Nalin Dahyabhai wrote:
On Tue, May 18, 2021 at 04:03:18PM -0400, Chris Evich wrote:
> Would it be okay if we do as Tom suggests below, but also disable the
> 'conformance' task on the release branches?
Knowing that the behavior of docker build changes subtly between
versions, is it not possible to keep testing against the same version of
dockerd on a given branch? Otherwise, we can miss breakage that comes
in via backports.
Nalin,
This is possible, but it adds complexity to our setup. However, based
on your concerns, it sounds the added complexity may be necessary.
What I would propose is, instead of installing these packages from the
network at runtime, we download and cache them at VM image build time.
At runtime, for this specific test, we install the packages from cacche.
Since we never update VM images on release-branches, it should satisfy
both the testing requirements, while adding stability.
I'll add a Jira card so I don't forget to work on this.
--
Chris Evich (he/him), RHCA III
Senior Quality Assurance Engineer
If it ain't broke, yain't tryin' hard 'nough.
620
days inactive
625
days old
podman-monitor@lists.podman.io
5 comments
4 participants
participants (4)
-
Chris Evich -
Do Not Reply -
Nalin Dahyabhai -
Tom Sweeney