16:00:07 <davide> #startmeeting Hyperscale SIG
16:00:07 <centguard> Meeting started Wed Aug 31 16:00:07 2022 UTC.  The chair is davide. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:07 <centguard> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:00:07 <centguard> The meeting name has been set to 'hyperscale_sig'
16:00:45 <davide> #chair dcavalca Eighth_Doctor
16:00:45 <centguard> Current chairs: Eighth_Doctor davide dcavalca
16:00:45 <davide> morning folks
16:00:51 <anitazha> Good morning
16:02:16 <Eighth_Doctor> hey
16:02:43 <neil> o/ heya folks. first time joining the hyperscale meetings :)
16:03:33 <aekoroglu> morning
16:03:42 <DaanDeMeyer[m]> Good morning
16:03:47 <davide> welcome neil
16:03:56 <davide> let's get started
16:04:04 <davide> #topic Followups
16:04:41 <davide> we had an in-person meetup the other week! it was fun, we should do it again
16:04:45 <davide> notes are at https://hackmd.io/Gsx7ci8FRtGV3E4q_diuiA
16:05:34 <davide> I think we have tickets for most of the stuff that was discussed, but if someone wants to go over it and verify we didn't miss anything that'd be great
16:06:00 <davide> also feel free to ask if anything isn't clear, those were quick notes I jotted down during the discussion
16:06:11 <davide> we also had multiple talks at CentOS Dojo and DevConf.us
16:06:27 <davide> my SIG update: https://www.youtube.com/watch?v=aXO0eLMkCDI
16:06:36 <davide> Anita Zhang (anitazha) and Daan De Meyer  systemd in hyperscale talk: https://www.youtube.com/watch?v=PdbyYqrvlnY
16:06:59 <davide> and Conan Kudo kiwi talk: https://www.youtube.com/watch?v=RKeFR4R2IeA
16:07:18 <davide> I meant to put together some kind of conference recap to post on the blog, but got sidetracked
16:07:56 <davide> will try to get this started later in the week
16:08:59 <davide> anything else for followups?
16:10:24 <davide> alright, let's move to
16:10:24 <davide> #topic Announcements
16:11:16 <davide> Meta is starting to clean up our internal build of openssh and getting it published in hyperscale at https://git.centos.org/rpms/openssh/tree/c9s-sig-hyperscale
16:11:31 <davide> the idea is to eventually get (as much as possible of) this patchset upstreamed into openssh proper
16:12:15 <davide> we're only building this in the fb tag at the moment, as I doubt people would want to use it in the current state, but if you have any questions or see anything you find interesting lemme know and I'd be happy to connect you with the folks driving this work
16:12:49 <davide> (incidentally, this is the first time we've used the PR workflow extensively and it's working great)
16:12:57 <neil> noob question: what are some of the changes the patchsets make?
16:13:40 <davide> right now it's mostly around additional logging
16:14:33 <davide> there's a brief description of the patches in the comments in the specfile
16:14:46 <davide> https://git.centos.org/rpms/openssh/blob/c9s-sig-hyperscale/f/openssh.spec#_269
16:14:54 <neil> ah, great. Will take a look there TY
16:15:15 <Eighth_Doctor> I've published a variant of Datto's internal cloud image build descriptions (stripped of a lot of internal stuff) as part of my DevConf.US talk about how Datto uses custom cloud golden images to scale our deployments: https://github.com/datto/devconfus22-demo-golden-image-descriptions
16:15:49 <Eighth_Doctor> The descriptions won't be updated, as it's pretty much designed to be an example, but you can use it to see how to do this sort of thing yourself
16:16:09 <Eighth_Doctor> and eventually the Hyperscale SIG will be publishing cloud images of its own, modeled somewhat after these
16:16:54 <davide> those will end up in https://pagure.io/centos-sig-hyperscale/kiwi-descriptions I assume?
16:16:58 <Eighth_Doctor> yes
16:17:08 <Eighth_Doctor> there's a PR of the in-progress descriptions to that repo already
16:17:11 <davide> oh you had a PR there already, I just noticed
16:17:14 <davide> nice, thanks
16:17:33 <davide> anything else for announcements?
16:18:26 <Eighth_Doctor> nothing from me
16:19:54 <davide> k, let's move to
16:19:55 <davide> #topic Tickets
16:20:51 <davide> we found a somewhat major issue with our SIG packages when using vendor pinning: https://pagure.io/centos-sig-hyperscale/sig/issue/127
16:21:18 <davide> looks like we never bulk rebuilt packages after the vendor change in koji, so we can easily end up with a situation with mixed vendors that breaks upgrades
16:21:36 <kcwells> Sounds like that will go away once we rebuild everything, right?
16:21:44 <davide> I'm going to run a check on all of our rpms to see what's affected, we'll need to either untag or bump and rebuild them
16:22:28 <davide> yup
16:23:37 <davide> Conan Kudo: we have an update from bstinson on https://pagure.io/centos-infra/issue/696 for the kiwi plugin
16:24:19 <Eighth_Doctor> just commented a "sounds good to us"
16:24:34 <davide> do we actually need to change anything in the ansible configs for this?
16:28:06 <Eighth_Doctor> we need to define the package set for kiwi builds
16:28:20 <Eighth_Doctor> and enable the kiwi command
16:29:18 <davide> ok, we'll need to start working on a PR for that then
16:29:36 <oidoming> openshift account problem that was blocking issue #91 is solved https://pagure.io/centos-infra/issue/840
16:29:40 <davide> also on the infra side, looks like https://pagure.io/centos-infra/issue/840 was resolved
16:29:51 <oidoming> :)
16:29:53 <davide> oidoming: is https://pagure.io/centos-sig-hyperscale/containers-releng/pull-request/4 good to merge or is there more work to do there?
16:29:57 <davide> ahah excellent timing
16:30:19 <oidoming> seems to work well in openshift
16:30:37 <oidoming> we can merge
16:30:50 <kcwells> \o/
16:30:54 <kcwells> WOo
16:31:03 <davide> just did that now, thanks!
16:32:09 <davide> that's all I had here I think, anything else for tickets?
16:35:09 <davide> alright, next up
16:35:10 <davide> #topic Membership
16:35:35 <davide> I don't think we have anything here for this week?
16:37:01 <davide> #topic Misc
16:37:04 <davide> anything else we want to discuss?
16:37:08 <davide> we're making pretty good time today
16:38:27 <DaanDeMeyer[m]> Maybe not super related to hyperscale, but libbpf 1.0 got released with a bunch of deprecated API removals, I've been trying to move systemd to the new APIs but for example on centos those are not yet available in libbpf because it's quite old, even on centos 9. Anyone know why it doesn't get updated?
16:39:02 <DaanDeMeyer[m]> Or maybe libbpf gets new releases very often and it's actually the latest that was available at the time
16:39:17 <davide> Red Hat generally doesn't rebase packages within a major release train, but there are exceptions
16:39:18 <davide> we could always file a bugzilla and ask
16:39:39 <aekoroglu> even fedora is 0.8 :(
16:39:59 <davide> downward deps for libbpf don't look too horrible, so we could theoretically maintain our own updated build in the SIG
16:40:03 <davide> but I'm not sure it's worth the trouble
16:40:40 <Eighth_Doctor> it'd be worth checking how libbpf is on https://access.redhat.com/articles/rhel9-abi-compatibility
16:40:54 <Eighth_Doctor> updating Fedora has to come first, though
16:41:08 <DaanDeMeyer[m]> I asked andrii to see if he'd be willing to add the deprecated APIs back, since this is going to be quite a mess for all projects that want to support old and new libbpf otherwise
16:41:30 <Eighth_Doctor> was there a soname bump?
16:41:40 <DaanDeMeyer[m]> Yeah, and a lot of API removals
16:41:53 <DaanDeMeyer[m]> Which translates into ifdeffery for every project using libbpf if they want to support both new and old
16:42:02 <Eighth_Doctor> then there's an (ugly) path forward for CentOS by adding a compat library package
16:42:09 <Eighth_Doctor> but for Fedora, everything will need to move to the new library
16:42:36 <Eighth_Doctor> even with compat packages, RH will want only one implementation being used internally
16:44:28 <davide> Daan De Meyer: might be worth starting a thread on fedora-devel on this
16:45:33 <DaanDeMeyer[m]> I can send an email there
16:47:51 <davide> sounds good
16:48:31 <davide> btw if you didn't see https://lists.centos.org/pipermail/centos-devel/2022-August/120557.html it's worth signal boosting
16:48:56 <davide> the fixed version (-211) is on mbox now but hasn't made it to a compose yet
16:50:12 <Eighth_Doctor> there isn't a similar problem in c9s, is there?
16:50:19 <davide> c9s shouldn't be impacted
16:50:35 <Eighth_Doctor> phew
16:50:53 <neil> phew indeed :D thanks for the link davide
16:51:00 <Eighth_Doctor> that's where most of my internal focus is, so I'd really rather not something like that happen in c9s
16:52:13 <davide> I only have a small %age of hosts on c9s at the moment, but so far things are looking good there
16:52:49 <davide> oh and in case anyone here cares about perf, I was just told that https://lore.kernel.org/linux-perf-users/Yw+LCN2cX9peweWV@kernel.org/T/ affects all published versions
16:53:07 <davide> I'll probably backport this in Hyperscale and put up a MR for c9s
16:57:40 <arrfab> davide, Eighth_Doctor : so wrt kiwi, that ticket is now linked to another one about cbs.stg, so that we can see what needs to be done at the kojihub/kojid level to enable kiwi
16:57:54 <arrfab> and we'll probably need some packages to be built, so coming with a list would help
16:57:59 <arrfab> same for our roles
16:58:34 <davide> I think the packages needed should all be branched in EPEL nowadays?
16:58:46 <Eighth_Doctor> yes
16:58:49 <davide> but yeah, we should put together a list
16:59:05 <Eighth_Doctor> for kiwi builds, you need "kiwi-cli" and "kiwi-systemdeps"
16:59:27 <Eighth_Doctor> that covers the full dependency graph of packages required for any build using kiwi that I know of
16:59:51 <Eighth_Doctor> oh, and "btrfs-progs" for hyperscale kiwi builds
17:00:00 <Eighth_Doctor> and the builder host needs to have btrfs available
17:00:23 <arrfab> so we just need to rebuild these in in the infra tags
17:00:31 <arrfab> hmm
17:00:38 <davide> btrfs-progs is not in EPEL, but we do have it in hyperscale if that helps
17:00:47 <arrfab> that was not mentioned (the need for host to be running hyperscale)
17:01:00 <arrfab> as clearly bstinson told me that the goal is to migrate to imagebuilder and kiwi is temporary
17:01:16 <arrfab> having one plugin is easy, modifying more for a tmp solution would need to be reconsidered then :/
17:01:26 <davide> I don't think it needs to necessarily run our variant -- it just needs to be able to handle btrfs filesystems
17:01:39 <davide> so e.g. the kmod-btrfs from the kmods sig + btrfs-progrs would do the trick
17:01:40 <bstinson> we discussed starting with existing builders
17:01:50 <bstinson> let's do enablement there first
17:02:05 <davide> yeah I think we can take this in two stages
17:02:08 <arrfab> bstinson: but if their image need btrfs directly, that will not work ?
17:02:16 <davide> first let's get kiwi working, then we can figure out the btrfs side of things
17:02:25 <bstinson> there are non-btrfs variants that can be used
17:02:32 <arrfab> *ack*
17:03:47 <davide> thanks everyone!
17:03:47 <davide> alright, we're out of time for this week
17:03:54 <davide> #endmeeting