#archlinux32 | Logs for 2018-03-22

Back
[01:56:13] * buildmaster goes insane.
[02:06:58] * buildmaster resumes sanity.
[06:44:00] -!- titus_livius has joined #archlinux32
[06:44:45] <buildmaster> girls, my database is dirty again ...
[06:48:04] -!- buildmaster has quit [Remote host closed the connection]
[06:50:41] -!- buildmaster has joined #archlinux32
[06:54:57] * buildmaster resumes sanity.
[08:25:57] -!- deep42thought has joined #archlinux32
[08:25:58] -!- deep42thought has quit [*.net *.split]
[08:26:00] -!- gnugnugnu has quit [*.net *.split]
[08:28:43] -!- titus_livius has quit [Ping timeout: 260 seconds]
[08:29:55] -!- titus_livius has joined #archlinux32
[10:53:02] <buildmaster> girls, my database is dirty again ...
[10:58:18] * buildmaster resumes sanity.
[11:23:24] -!- mode/#archlinux32 [+o deep42thought] by ChanServ
[11:23:51] -!- mode/#archlinux32 [-o deep42thought] by ChanServ
[11:25:15] <buildmaster> krita is broken (says buildknecht).
[12:23:03] <buildmaster> girls, my database is dirty again ...
[12:28:44] * buildmaster resumes sanity.
[13:39:32] <buildmaster> haskell-hakyll is broken (says nlopc43).
[13:39:55] <deep42thought> buildmaster: and you rescheduled ... ?
[14:05:57] <buildmaster> gitlab-runner is broken (says buildknecht3).
[16:27:22] -!- deep42thought has quit [Quit: Leaving.]
[16:42:38] <buildmaster> haskell-wai-logger is broken (says buildknecht).
[16:47:39] <buildmaster> haskell-wai-extra is broken (says buildknecht3).
[17:02:20] -!- oaken-source has joined #archlinux32
[17:06:37] <tyzoid> deep42thought: https://archlinux32.org still shows #archlinux-ports
[17:06:51] <tyzoid> we should change that coming up soon
[17:06:53] <tyzoid> girls: ^
[17:24:09] <buildmaster> hopenpgp-tools is broken (says buildknecht3).
[17:44:29] <buildmaster> haskell-aws is broken (says buildknecht).
[17:47:06] <buildmaster> stack is broken (says buildknecht3).
[18:30:13] <girls> tyzoid: I changed the website
[18:30:55] -!- abaumann has joined #archlinux32
[18:49:22] -!- deep42thought has joined #archlinux32
[18:49:27] <deep42thought> Hi abaumann
[18:49:39] <deep42thought> I'm working on your cloak, but haven't got a response so far ...
[18:56:10] <deep42thought> btw: I decommissioned the state-files, so the buildmaster should be _a_lot_ quieter now
[19:01:55] <abaumann> Hi there.
[19:02:08] <abaumann> cloak? huh?
[19:02:50] <deep42thought> I want to request "archlinux32/bugwrangler/abaumann" for you
[19:03:02] <deep42thought> like I have "archlinux32/developer/deep42thought"
[19:03:17] <abaumann> ah.. now I rember having read something. :-)
[19:03:21] <abaumann> *remember
[19:03:47] <abaumann> I'm happy the buildmaster is alive and kicking again.. :-)
[19:04:52] <deep42thought> :-D
[19:05:19] <abaumann> sorry. my reaction was not appropriate when it dived..
[19:05:22] <deep42thought> https://buildmaster.archlinux32.org ios somewhat worrying, though
[19:05:57] <abaumann> BTW: can you kick librsvg and qt5-declarative to end up in stable?
[19:06:09] <deep42thought> sure
[19:06:11] <abaumann> I backported a patch from Debian for qt5-declarative..
[19:06:41] <abaumann> it seems, 2 or 3 people at least had an issue around KDE and some qt apps because of that.
[19:07:20] <abaumann> a lot of haskell stuff on that list :-(
[19:07:44] <abaumann> very good list. :-)
[19:08:03] <deep42thought> I only see librsvg which is still in testing
[19:08:11] <deep42thought> qt5-declarative is already in extra
[19:08:14] <abaumann> ah. ok.
[19:08:31] <abaumann> the failing bug in librsvg is open in Gnome's gitlab (upstream).
[19:08:40] <abaumann> so far we try to reproduce it.
[19:08:53] <abaumann> it could also be an issue with the rust compiler or/and the toolchain..
[19:13:00] <deep42thought> it turned out, that replaying the backup is no issue, but getting the database in sync with the master mirror again takes quite some time
[19:15:41] <deep42thought> librsvg is in extra, now
[19:20:55] <abaumann> ok. thanks.
[19:27:44] <tyzoid> deep42thought: Glad that the backup/restore process worked well
[19:27:53] <tyzoid> Any changes we can make to make the sync issue easier?
[19:28:00] <tyzoid> or is it acceptable as it is
[19:28:27] <deep42thought> we'd need a backup of the master mirror of the same time, too
[19:28:35] <deep42thought> this is not realistic imho
[19:28:38] <deep42thought> so: it's ok
[19:30:47] <deep42thought> we could automate the "rebuild everything on the master mirror which is not in the database" step
[19:31:04] <deep42thought> but I hope we don't need to do that too often, so it's probably not worth the effort
[20:21:14] <tyzoid> deep42thought: Well I make the archive based on my mirror... I could modify my script to force a re-sync right before the archival process
[20:21:31] <tyzoid> and then we could time the backup of the buildmaster and the archive to be closer
[20:22:09] <deep42thought> yes, I was thinking about something like that, too
[20:22:23] <deep42thought> but the question is, if we really want to play back old packages in case of emergency
[20:22:51] <deep42thought> but it's probably a good idea to make the archive at the same time as the backup, anyway
[20:23:41] <deep42thought> if we play back the packages if "only" the build master crashed, can be decided, when that actually happens
[20:32:02] <tyzoid> Theoretically we'll only ever roll-back the packages at most 24hrs
[20:32:08] <tyzoid> assuming we catch/resolve the crash same day
[20:32:44] <deep42thought> the difference between "is" and "will be" will still be <1 day - even if we recover later
[20:33:22] <deep42thought> it's more a philosophical/principal question I'm not yet decided on
[20:33:30] <tyzoid> IIRC you make the backups at 6am CEST, right?
[20:33:40] <deep42thought> um, lemme check
[20:34:10] <deep42thought> 13 6 * * * /home/deep42thought/backup.sh
[20:34:40] <deep42thought> if ! stty &>/dev/null; then
[20:34:40] <deep42thought> sleep $(($RANDOM % 3600))
[20:34:41] <deep42thought> fi
[20:34:50] <deep42thought> so it's somewhat unsharp
[20:35:47] <tyzoid> Ok, I have my backup set to run at midnight UTC (though IIRC it's currently 23:00, since Daylight Saving Time)
[20:35:57] <tyzoid> so we're about 6hrs off
[20:36:24] <deep42thought> ah, my time is not utc
[20:36:34] <deep42thought> but CET
[20:36:39] <tyzoid> Right
[20:36:44] <tyzoid> I thought I accounted for that
[20:36:56] <tyzoid> right 8 hrs off, currently
[20:36:58] <tyzoid> I did the math backwards
[20:36:59] <tyzoid> I think
[20:37:08] <tyzoid> anyway I'll bump my archive up to 7AM UTC, which should make them align closer
[20:37:25] <deep42thought> can't I just trigger the backup on the archive via borg?
[20:37:34] <tyzoid> archive doesn't trigger via borg
[20:37:50] <tyzoid> it's a copy of packages followed by a huge process of making thousands of symlinks
[20:38:17] <deep42thought> but can't you trigger that by "borg-user logs in"
[20:38:18] <deep42thought> ?
[20:38:33] <deep42thought> or I could trigger the borg backup by some signal from you ...
[20:38:36] <tyzoid> the server that hosts the borg repo isn't the same as the archive server
[20:38:45] <deep42thought> ok
[20:39:06] <deep42thought> well, there is https://buildmaster.archlinux32.org
[20:39:26] <deep42thought> which gives the epoch-date of the last borg backup
[20:40:01] <tyzoid> Does that update when borg completes?
[20:40:06] <deep42thought> yes
[20:40:28] <tyzoid> Ok, so theoretically I could watch my local last-backup file and trigger based on when that changes
[20:40:38] <deep42thought> no
[20:40:41] <deep42thought> this is on the buildmaster
[20:40:44] <deep42thought> not the master mirror
[20:40:49] <tyzoid> ah
[20:41:02] <tyzoid> well regardless, I think it'd be more robust to run the archive on it's own schedule
[20:41:23] <deep42thought> true
[20:41:25] <tyzoid> then the archive doesn't stop if the borg procedure breaks
[20:41:54] <deep42thought> but you could set some timeout: archive is created when last-backup changed or 26h no archive
[20:42:22] <deep42thought> (this sentence misses a few verbs, but I was not sure which would fit nicely)
[20:42:27] <tyzoid> Even still, I think it'd just be easier to align the schedules
[20:42:33] <deep42thought> yes
[20:43:00] <deep42thought> less effort and more robustness might be good arguments here :-)
[20:43:57] <tyzoid> only issue for me is that changing from 1900 to 0100 ET (2300 to 0500 UTC) would constitute a date change in localtime
[20:44:20] <tyzoid> I could try to get around that by using the -1day rule
[20:44:40] <deep42thought> well, you simply make two backups in 6h distance
[20:45:07] <deep42thought> .. and you should set up the change between 23 and 5 o'clock
[20:45:20] <deep42thought> (utc)
[20:45:51] <tyzoid> Yeah, but something feels a bit disingenuous having an archive for 2018-03-22 when the day has practically not even begun
[20:46:23] <tyzoid> That's why I've been doing them at end-of-day
[20:46:33] <deep42thought> matter of taste
[20:46:39] <tyzoid> Definately
[20:46:40] <deep42thought> but I could also just change the borg date
[20:46:50] <deep42thought> ... if it really bothers you :-)
[20:47:02] <tyzoid> well moving that up affects other services, as I've brought up before
[20:47:28] <deep42thought> ah, right
[20:47:42] <deep42thought> I should do the backup at a different time, but send you the data at the old time ;-P
[20:47:53] <tyzoid> lol
[20:48:13] <tyzoid> I'll move the archive time back a few hours, but keep the date the same
[20:48:29] <deep42thought> ok
[20:48:42] <tyzoid> because then it'll still be end-of-day, just at midnight in the USA instead of midnight UTC
[20:49:06] <tyzoid> And you can remove the randomization from the borg schedule
[20:49:13] <deep42thought> will do
[20:49:14] <tyzoid> that'll help us get them to line up easier
[20:49:24] <deep42thought> done
[20:49:48] <tyzoid> Is CET still UTC+1? Or do you have any daylight saving weirdness over there?
[20:50:03] <deep42thought> btw: I think, we should set up a realtime read-only database duplicate at some point
[20:50:11] <deep42thought> yes, we have!
[20:50:31] <deep42thought> but currently we're at the only true (=w/o this stupid shift)
[20:51:19] <deep42thought> the realtime db-clone would also safe us the trouble of discrepancies
[20:51:51] <tyzoid> It would, but it wouldn't fix a master-mirror going down scenario
[20:52:17] <deep42thought> for that, we have tier1 mirrors
[20:52:22] <tyzoid> though I guess the mirroring system could be considered realtime backups
[20:52:23] <tyzoid> yeah
[20:53:17] <tyzoid> So do you think we should start to clear out older isos from the isos folder?
[20:53:24] <deep42thought> umm yes
[20:53:30] <tyzoid> Currently they're not included in the archive
[20:53:33] <deep42thought> do you have permision to do so?
[20:53:39] <tyzoid> I could archive those separately if we want to keep them around
[20:53:51] <tyzoid> which might be nice if people wanted to install out of the archive, for some reason
[20:54:02] <deep42thought> your decision
[20:54:05] <tyzoid> I think I do, since I have sshfs access to add them
[20:54:09] <deep42thought> I think isos eat up quite some space
[20:54:15] <tyzoid> esp. the -dual ones
[20:54:30] <deep42thought> and they're 100% new with each release
[20:54:48] <deep42thought> whereas the packages do not change to 100% and many can be hard linked
[20:56:37] <tyzoid> Yeah.
[20:56:48] <tyzoid> Btw, ArchStrike announces they're dropping i686 support: https://archstrike.org
[20:57:27] -!- jarshvor has joined #archlinux32
[20:57:42] <deep42thought> maybe we were too unstable?
[20:57:49] * deep42thought feels guilty
[20:57:59] <deep42thought> :-D
[20:58:02] <tyzoid> No, I think they just didn't want to maintain their custom i686 packages
[20:58:50] <deep42thought> btw: I was digging throug the master mirror logs and I found two new mirrors, rsyncing from us :-)
[20:58:59] <tyzoid> hmmm, nice
[20:59:10] <tyzoid> they public mirrors? or private?
[20:59:13] <jarshvor> Hi all, first of all, thanks for great work and effort in keeping i686 as an availeable option for arch.
[20:59:30] <girls> tyzoid: public
[20:59:33] <tyzoid> :) thanks
[20:59:52] <deep42thought> jarshvor: thanks, it's being fun
[21:00:08] <tyzoid> any chance we can find an administrative contact for said mystery mirrors to see if they want to be added to the mirrorlist?
[21:00:13] <jarshvor> now for my question. I migrated from to the official mirrors around christmas on this old laptop I use occasionaly when I'm on the go.
[21:00:28] <deep42thought> tyzoid: I found, asked, got permission and added them already
[21:00:34] <tyzoid> sweet!
[21:00:45] <jarshvor> had some colourful screen corruption when starting xorg.
[21:00:55] <jarshvor> I believe it was related to this:
[21:00:56] <jarshvor> https://bbs.archlinux32.org
[21:00:59] <tyzoid> I think we've got a post on the bbs
[21:01:03] <jarshvor> yes
[21:01:06] <tyzoid> yeah, that one
[21:01:30] <jarshvor> I was getting by using the nomodesetting kernel option at boot until I saw that post
[21:02:48] <tyzoid> jarshvor: So are you still experiencing a problem?
[21:02:56] <jarshvor> after reading that post I tried tweaking my grub.cfg as user eee-on-fire suggested in that thread that it was grub related
[21:03:53] <jarshvor> I recall i treid to add a couple of dummy grub sections
[21:04:29] <deep42thought> btw: grub lets you edit the parameter on the fly if you press "e", i think
[21:04:32] <jarshvor> i must have not edited it properly but the end effect worked as desired, as grub complained slightly about some syntax quickly and jumps directly to boot the last section
[21:04:46] <jarshvor> anyway... sorry for the ramble
[21:04:50] <tyzoid> deep42thought: as does systemd-boot
[21:04:53] <jarshvor> I have updated kernel now
[21:05:14] <jarshvor> 4.15 (eee-pc mentions the bug is related to the older 'fixed' kernels)
[21:05:23] <jarshvor> I am booting fine
[21:05:42] <jarshvor> the problem I am having is if I close the laptop lid then I cant resume from hibernation afterwrds
[21:05:45] <jarshvor> not succesfully atleast
[21:05:53] <tyzoid> ah
[21:06:12] <tyzoid> yeah, hibernation is this weird thing that sometimes doesn't work right
[21:06:30] <deep42thought> related? https://bbs.archlinux32.org
[21:06:30] <tyzoid> not sure if deep42thought has any insight, but I haven't trusted hibernation in linux ever since 2014
[21:06:42] * deep42thought doesn't own a laptop
[21:06:56] <deep42thought> ... well, that's not 100% precise
[21:06:59] * tyzoid always powers off or sleeps
[21:10:18] <jarshvor> (sorry, was on phone)
[21:10:32] <deep42thought> jarshvor: you missed (almost) nothing
[21:10:53] <jarshvor> was bothers me is I'm not sure the 'grub issue/bug' is a actually resolved for me with the new kernel
[21:11:21] <jarshvor> I can try again, but im rpetty sure if I restore a default grub.cfg ill be unable to boot without screen corruption.
[21:11:38] <jarshvor> grub-mkconfig -o i mena
[21:11:53] <jarshvor> ^mena^mean
[21:12:08] <tyzoid> you can set custom parameters in the grub config by changing the autogen scripts
[21:12:28] <deep42thought> in general, it's a good idea to set the grub values in /etc/default/grub - so you can run "grub-mkconfig -o ..." w/o destroying any customization
[21:12:52] <tyzoid> /etc/grub.d/* as well
[21:13:20] <jarshvor> tyzoid: well, yeah sure, but it'd rather figure out if the bug is actually resolved with the new kernel.
[21:13:38] <jarshvor> dont read me wrong. I know you guys are trying to help. and I appreaciate greatly.
[21:13:51] <tyzoid> cp /boot/grub/grub.cfg /boot/grub/grub.cfg.bak
[21:14:00] <jarshvor> dont really mind the hacky grub right now, just the not being able to close the lid is bothering me
[21:14:01] <tyzoid> that'll make it easier to fix
[21:14:07] <deep42thought> jarshvor: well, then just add a default boot entry and keep the old ones in the file in case the default one does not work
[21:15:05] <tyzoid> jarshvor: alternatively, you can set lid close to suspend instead of hibernate, if that is acceptable to you as a workaround (assuming you don't get hibernate working soon)
[21:15:12] <jarshvor> I've actually done as you guys are mentioning last time I updated the system. had to revert. basically what I guess Im asking, is if anyone has any idea how the bug might relate to hibernation or any way to fix/work around it?
[21:16:16] <jarshvor> (sorry if im being extra verbose and not to explaining myself properly)
[21:16:33] <deep42thought> jarshvor: all good
[21:17:07] <jarshvor> tyzoid: mm thats a good suggestion. Ill check is its suspend vs hibernate
[21:20:20] -!- abaumann has quit [Quit: leaving]
[21:24:10] <deep42thought> too bad, I just got abaumann's cloak ...
[21:29:48] <tyzoid> ah, lol
[21:30:05] <tyzoid> well, he'll see it in the mornign
[21:30:07] <tyzoid> morning*
[21:42:37] <deep42thought> tyzoid: Do you have any plan how to set up the mysql duplication?
[21:48:03] <tyzoid> I think the guide I linked a while ago has instructions on how to set it up, but I haven't thought much more on where we'd put the mirror system
[21:48:26] <tyzoid> I was originally thinking of my dedi, but the disk is close to full
[21:51:43] <deep42thought> I was more thinking of a plan on what/how to test before we put this into production
[21:52:02] <tyzoid> Oh, that should be simpler
[21:52:39] <tyzoid> (1) create fake buildmaster in a vm network (natted, no net access), restoring from backups
[21:52:40] <deep42thought> e.g. I'm curious how replication affects latency
[21:53:04] <tyzoid> (2) create fake master repo in the same network
[21:53:11] <tyzoid> (3) point them at each other
[21:53:13] <tyzoid> (4) test
[21:53:25] <tyzoid> Though you'd probably swap steps 1 and 2
[21:57:24] <deep42thought> I think we could skip the replication of the master mirror and just test the database
[21:57:34] <deep42thought> there is plenty of stuff you can test with the database alone
[21:58:58] <tyzoid> In that case
[21:59:09] <tyzoid> and in the interest of coming up with more specific procedures
[21:59:27] <tyzoid> we'll probably want to clone the read-slave
[21:59:32] <tyzoid> then break the replication on the clone
[21:59:46] <tyzoid> then replicate the cloned slave over to the restore-vm
[22:00:06] <tyzoid> then flip master<->slave, so the restore-vm is the master and the cloned slave is a slave again
[22:00:31] <tyzoid> then you can run your tests
[22:02:28] <deep42thought> sry, I cannot follow - what do you mean by "read-slave"?
[22:18:30] <deep42thought> anyway, it's late here - off to bed!
[22:18:34] -!- deep42thought has quit [Quit: Leaving.]
[22:23:23] <tyzoid> Alright, good night
[22:24:08] <tyzoid> by Read-Slave, I'm referring to the read-only database that's replicating from the buildmaster's database. The buildmaster has the "master" db, and the backup server has the "slave" db.
[22:47:55] -!- oaken-source has quit [Ping timeout: 268 seconds]
[23:47:22] -!- jarshvor has quit [Quit: leaving]