#archlinux32 | Logs for 2019-07-16

Back
[00:02:55] -!- AndrevS has quit [Remote host closed the connection]
[00:29:00] <buildmaster> pentium4/haskell-configurator-ng is broken (says eurobuild6-4): https://archlinux32.org
[00:37:55] -!- samantaz has quit [Ping timeout: 246 seconds]
[00:40:10] -!- samantaz has joined #archlinux32
[01:11:27] <buildmaster> i686/haskell-ini is broken (says eurobuild6-4) - I rescheduled: haskell-hspec: https://archlinux32.org
[01:22:02] -!- isacdaavid has quit [Ping timeout: 246 seconds]
[01:43:00] -!- samantaz has quit [Read error: Connection reset by peer]
[01:46:36] <buildmaster> i686/haskell-data-serializer is broken (says eurobuild6-6) - I rescheduled: haskell-data-endian: https://archlinux32.org
[03:45:46] -!- eschwartz has quit [Ping timeout: 276 seconds]
[03:47:00] -!- eschwartz has joined #archlinux32
[04:46:51] <buildmaster> pentium4/haskell-kan-extensions are broken (says eurobuild6-6): https://archlinux32.org
[06:01:54] <buildmaster> pentium4/prometheus are broken (says nlopc46): https://archlinux32.org
[06:09:17] <buildmaster> i686/luasec is broken (says eurobuild6-5): https://archlinux32.org
[06:31:39] <buildmaster> pentium4/luasec is broken (says eurobuild6-1): https://archlinux32.org
[06:40:24] <buildmaster> i686/prometheus are broken (says buildknecht): https://archlinux32.org
[06:53:42] <buildmaster> i686/parity-ethereum is broken (says eurobuild6-5): https://archlinux32.org
[06:59:06] <buildmaster> pentium4/lrs are broken (says eurobuild6-5): https://archlinux32.org
[07:33:50] <buildmaster> i686/go is broken (says eurobuild6-4): https://archlinux32.org
[07:45:35] <buildmaster> pentium4/golang-golang-x-image is broken (says buildknecht2): https://archlinux32.org
[07:46:02] <buildmaster> i686/lrs are broken (says rechenknecht): https://archlinux32.org
[08:00:25] <buildmaster> pentium4/linux is broken (says eurobuild6-3): https://archlinux32.org
[08:02:37] <buildmaster> pentium4/linux-zen is broken (says eurobuild6-2): https://archlinux32.org
[08:04:51] <buildmaster> i686/linux is broken (says nlopc43): https://archlinux32.org
[08:05:21] <buildmaster> i686/linux-zen is broken (says eurobuild1): https://archlinux32.org
[08:07:59] -!- abaumann has joined #archlinux32
[08:07:59] <buildmaster> Hi abaumann!
[08:07:59] <buildmaster> !rq abaumann
[08:08:00] <phrik> buildmaster: <abaumann> haskell-base-compat-batteries.. ah.. Haskell comes with batteries included.. in constrast to my Arduino box..
[08:08:09] <abaumann> yes, I logged into the wrong buildmaster twice, hence the /etc/motd :-)
[08:09:32] <abaumann> I think, the format for the keys is ok.
[08:27:41] -!- abaumann has parted #archlinux32
[08:29:12] <buildmaster> dirty! girls, my database - so dirty :-(
[08:29:12] * buildmaster goes insane.
[08:49:21] -!- deep42thought has joined #archlinux32
[08:49:21] <buildmaster> Hi deep42thought!
[08:49:21] <buildmaster> !rq deep42thought
[08:49:23] <phrik> buildmaster: <deep42thought> arch32 is on the bleeding edge of arch, which is on the bleeding edge of software?
[08:50:27] <deep42thought> 3,9k packages in staging - this does not look healthy :-/
[09:03:09] * buildmaster resumes sanity.
[09:34:35] -!- bill-auger has quit [Read error: Connection reset by peer]
[09:38:34] -!- bill-auger has joined #archlinux32
[09:44:34] <buildmaster> i686/minio is broken (says eurobuild6-1): https://archlinux32.org
[09:52:08] -!- ofara_ has quit [Ping timeout: 245 seconds]
[09:53:06] -!- ofara has quit [Ping timeout: 272 seconds]
[10:00:03] -!- ofara has joined #archlinux32
[10:36:07] <deep42thought> abaumann: we have now the execution times of several slave-build-connect executions on dislpay: https://archlinux32.org
[10:41:00] -!- deep42thought has quit [Remote host closed the connection]
[10:43:05] -!- nit-picker has joined #archlinux32
[10:45:36] -!- deep42thought has joined #archlinux32
[10:45:36] <buildmaster> Hi deep42thought!
[10:45:36] <buildmaster> !rq deep42thought
[10:45:37] <phrik> buildmaster: <deep42thought> I have the impression, their main operating area is television via cable
[10:47:04] -!- nit-picker has quit [Remote host closed the connection]
[11:32:46] <buildmaster> pentium4/hasktags are broken (says nlopc46): https://archlinux32.org
[12:40:47] <buildmaster> pentium4/apparmor is broken (says nlopc46): https://archlinux32.org
[12:47:28] <buildmaster> pentium4/python-aws-xray-sdk is broken (says nlopc46): https://archlinux32.org
[12:48:10] <buildmaster> pentium4/npm is broken (says eurobuild3): https://archlinux32.org
[13:37:43] <buildmaster> i686/js60 is broken (says eurobuild6-4): https://archlinux32.org
[13:38:18] <buildmaster> pentium4/tigervnc is broken (says eurobuild6-2): https://archlinux32.org
[13:43:34] <buildmaster> pentium4/js60 is broken (says eurobuild6-3): https://archlinux32.org
[13:52:34] -!- DCyrax has joined #archlinux32
[14:36:13] -!- ofara_ has joined #archlinux32
[14:40:58] -!- abaumann has joined #archlinux32
[14:40:58] <buildmaster> Hi abaumann!
[14:40:58] <buildmaster> !rq abaumann
[14:41:00] <phrik> buildmaster: <abaumann> mmh.. my build slaves are on strike? Maybe I should pay them? ;-)
[14:41:06] <deep42thought> Hi abaumann
[14:41:12] <abaumann> hi deep42thought
[14:41:17] <abaumann> really cool graph :-)
[14:41:43] <abaumann> looks a little bit like the plot of colliding slaves ;-)
[14:41:57] <deep42thought> colliding slaves???
[14:42:22] <abaumann> large hadron collider - just for build slaves..
[14:43:18] <deep42thought> I hope, we can solve that riddle without looking into quantum chromodynamics ;-)
[14:47:11] <abaumann> :-)
[14:47:57] <abaumann> Doesn't matter we have a lot to be rebuilt, the cluster is now quite fast..
[14:48:22] <deep42thought> I'm a little concerned about the runtimes of get-assignment and return-assignment, though
[14:49:07] <abaumann> The y-axis is seconds, right?
[14:49:11] <deep42thought> yes
[14:49:32] <abaumann> yeah, as long it's a flat line, nothing to worry about.
[14:49:35] <deep42thought> sry, I live in a physics world - it's either SI or "relativistic units"
[14:49:53] <deep42thought> well, but execution time of >1min is worrysome
[14:50:14] <abaumann> yeah, true, but, maybe things where running in parallel, so it's mainly waiting for locks
[14:50:21] <deep42thought> no
[14:50:28] <deep42thought> these are only the jobs, that got the locks
[14:50:34] <abaumann> aha.
[14:50:36] <deep42thought> (exit_code=0)
[14:50:44] <deep42thought> the others exit with 1 and come back later
[14:51:26] <abaumann> I see a relative high I/O rate on the hard disks on the buildmaster..
[14:51:39] <abaumann> ..though I gave mysql Factor 10 more memory for certain areas.
[14:51:46] <deep42thought> hmm
[14:51:46] <abaumann> So, we wait for I/O
[14:52:06] <abaumann> CPU cannot be the limiting factor, currently one core basically is used for Mysql
[14:52:28] <deep42thought> one thing we should log, too is: how many trials do the mysql queries need to complete
[14:52:37] <deep42thought> (it tries upto 10 times before giving up)
[14:52:57] <deep42thought> it might be, that the actual queries are not that slow, but it has to retry them several times
[14:53:02] <deep42thought> what do you think?
[14:53:28] <abaumann> Mmh. The question really is, is I/O read or write?
[14:53:41] <deep42thought> read, I guess
[14:54:09] <deep42thought> it's mostly huge joins from binary_packages<->dependencies<->versions<->install_target_providers<->binary_packages and so on
[14:54:20] <deep42thought> and in the end a single id is retrieved
[14:54:27] <deep42thought> and a single row is being updated
[14:54:37] <deep42thought> (in `build_slaves`)
[14:54:49] <deep42thought> maybe some more rows (when adding dependencies of built packages)
[14:55:01] <deep42thought> but never more than maybe 200 rows at once or so
[14:55:43] <abaumann> mmh, we should check join memory areas in my.cnf in this case.
[14:55:59] <abaumann> vmstat 1 100
[14:56:04] <deep42thought> ?
[14:56:12] <abaumann> tells be we have 'bo' which are block-outs.
[14:56:14] <abaumann> so writes.
[14:56:24] <abaumann> so the query can not be the problem.
[14:56:39] <abaumann> are we writing a lot of rows.. I mean, besides the ssh-logs :-)
[14:58:45] <deep42thought> I can really only think of logs
[14:59:08] <deep42thought> though, there are /many/ places where logs happen
[14:59:27] <deep42thought> e.g. ssh_log, statistics, mirror_statuses, command_log, email_log
[15:00:00] <abaumann> iotop says jbd2/md2-8 and sometimes a little bit mysql
[15:00:29] <deep42thought> what's jbd2?
[15:00:39] <deep42thought> is this the worker thread for the raid?
[15:00:40] <abaumann> some daemon doing journalling, I suppose
[15:00:54] <abaumann> 3.4 GB of journald logfiles.. mmmh.
[15:00:59] <deep42thought> :-D
[15:01:48] <abaumann> I did a journalctl -fl before, and the shell basically froze.
[15:02:02] <abaumann> I have a very special opinion about journalds binary log format..
[15:02:17] <deep42thought> !grab abaumann
[15:02:17] <phrik> deep42thought: Tada!
[15:03:02] <abaumann> another cause could be too many transactions doing fsync/fdatasync to the block device
[15:03:25] <deep42thought> I have some "COMMIT"s in the queries
[15:03:30] <deep42thought> which we could theoretically drop
[15:03:31] <abaumann> there is a tradeoff between keeping the promise of ACID properties and performance.
[15:03:51] <abaumann> the problem is more, what is mysql doing currently when commiting.
[15:04:06] <abaumann> in Postgresql I know there is a fsync flag somewhere in the config.
[15:04:43] <abaumann> https://dev.mysql.com transaction long sync time or something along the lines..
[15:04:50] <abaumann> *log
[15:05:34] <abaumann> We should focus on setting some smart parameters.. this is easier than to change things in the scripts..
[15:05:40] <abaumann> ..maybe :-)
[15:07:04] <deep42thought> at least, it will break less stuff than changing something in the scripts ;-)
[15:11:27] -!- ofara has quit [Quit: ofara]
[15:13:00] -!- ofara has joined #archlinux32
[15:13:46] <deep42thought> btw: some long-running red dots are ok (return-assignment ERROR) - this can happen when something gets rescheduled
[15:16:43] <deep42thought> sync_binlog = 1
[15:16:52] <deep42thought> is probably causing much disk i/o
[15:20:06] <deep42thought> I increased that number to 32, so now we wait for 32 "binary log commit" groups
[15:20:12] <deep42thought> before actually flushing
[15:30:03] -!- ofara_ has quit [Ping timeout: 245 seconds]
[15:44:48] -!- rcf has quit [Quit: WeeChat 2.4]
[15:45:46] -!- rcf has joined #archlinux32
[16:11:42] <buildmaster> pentium4/haskell-sbv is broken (says rechenknecht): https://archlinux32.org
[16:25:50] -!- deep42thought has quit [Quit: Leaving.]
[16:57:29] -!- abaumann88 has joined #archlinux32
[16:57:37] -!- abaumann88 has parted #archlinux32
[16:59:06] -!- abaumann has quit [Remote host closed the connection]
[16:59:39] -!- abaumann has joined #archlinux32
[16:59:39] <buildmaster> Hi abaumann!
[16:59:39] <buildmaster> !rq abaumann
[16:59:41] <phrik> buildmaster: <abaumann> yeah. real servers don't grow on trees. ;-)
[17:12:29] <abaumann> 176 root 20 0 0 0 0 R 100.0 0.0 256:30.39 kswapd0
[17:12:52] <abaumann> mmh. 16 GB of RAM and swapping
[17:13:01] <abaumann> with a kswapd0 using 100% CPU?
[17:13:12] <abaumann> just from time to time, but weird.
[17:13:19] <abaumann> I'll play with the swappiness
[17:16:57] -!- rcf has quit [Quit: WeeChat 2.1]
[17:16:57] <abaumann> na. it seems ok. no clue..
[17:17:37] -!- rcf has joined #archlinux32
[17:30:17] -!- buildmaster has quit [Remote host closed the connection]
[17:31:31] -!- buildmaster has joined #archlinux32
[17:31:31] <buildmaster> !rq buildmaster
[17:31:31] <phrik> buildmaster: <buildmaster> I might be insane, but never confused ... ;-)
[17:36:25] <abaumann> strace -c -f -p 28803
[17:36:38] <abaumann> 36.19 1.908368 2 655934 655903 close
[17:36:50] <abaumann> return-assignment
[17:36:53] <abaumann> why so many closes?
[17:37:27] -!- thePiGrepper has joined #archlinux32
[17:41:38] <buildmaster> i686/subversion is broken (says eurobuild3): https://archlinux32.org
[17:41:42] <buildmaster> pentium4/python2 is broken (says eurobuild6-1): https://archlinux32.org
[17:41:55] <buildmaster> pentium4/subversion is broken (says nlopc46): https://archlinux32.org
[18:02:32] -!- ofara_ has joined #archlinux32
[18:03:34] <buildmaster> i686/python2 is broken (says eurobuild6-6): https://archlinux32.org
[18:06:05] -!- isacdaavid has joined #archlinux32
[18:19:36] <buildmaster> i686/aria2 is broken (says nlopc46): https://archlinux32.org
[18:20:22] -!- ofara_ has quit [Quit: ofara_]
[18:21:35] -!- ofara_ has joined #archlinux32
[18:30:29] -!- abaumann has quit [Quit: leaving]
[19:16:08] -!- samantaz has joined #archlinux32
[19:41:07] -!- panosdev has joined #archlinux32
[19:42:18] <panosdev> hello, I can't install Arch Linux 32 due to bad keys
[19:50:40] <bill-auger> panosdev: not sure what you mean - do you have a LiveISO? - did you run pacman -S archlinux32-keyring ?
[19:50:57] <bill-auger> pacman -Sy archlinux32-keyring
[19:51:40] -!- panosdev has quit [Ping timeout: 276 seconds]
[21:21:54] -!- isacdaavid has quit [Ping timeout: 258 seconds]
[21:30:19] <buildmaster> pentium4/yubikey-manager-qt is broken (says eurobuild3): https://archlinux32.org
[21:31:11] <buildmaster> pentium4/aria2 is broken (says eurobuild6-1): https://archlinux32.org
[21:41:41] <buildmaster> pentium4/java7-openjdk is broken (says nlopc46): https://archlinux32.org
[21:42:17] <buildmaster> i686/java7-openjdk is broken (says eurobuild6-1): https://archlinux32.org
[21:53:25] <buildmaster> pentium4/java8-openjdk is broken (says nlopc46): https://archlinux32.org
[22:12:00] <buildmaster> i686/flashplugin is broken (says eurobuild6-6): https://archlinux32.org
[22:21:41] <buildmaster> pentium4/flashplugin is broken (says eurobuild6-6): https://archlinux32.org
[22:53:55] <buildmaster> pentium4/java11-openjdk is broken (says nlopc46): https://archlinux32.org
[23:17:56] -!- Mindi has quit [Ping timeout: 252 seconds]
[23:18:40] <girls> dopsi: a new iso is online. It would be nice if you could create torrents and magnetlinks :-)
[23:19:52] -!- Mindi has joined #archlinux32
[23:52:19] -!- ofara has quit [Ping timeout: 246 seconds]
[23:52:46] -!- ofara has joined #archlinux32