#archlinux32 | Logs for 2019-08-03

Back
[00:22:57] -!- thePiGrepper has quit [Ping timeout: 245 seconds]
[00:26:04] -!- thePiGrepper has joined #archlinux32
[01:48:27] -!- eschwartz has quit [Ping timeout: 250 seconds]
[02:40:03] -!- MrBIOS has quit [Quit: MrBIOS]
[03:51:31] -!- isacdaavid has quit [Quit: Leaving.]
[04:27:48] <buildmaster> pentium4/haskell-parsers are broken (says nlopc46): https://archlinux32.org
[05:22:07] <buildmaster> i686/haskell-aeson is broken (says eurobuild6-5): https://archlinux32.org
[05:41:30] <skydrome> what determines when packages get moved from staging into testing and so forth? Havent had anything to upgrade in weeks
[07:19:49] <buildmaster> i686/haskell-hledger-lib is broken (says nlopc46): https://archlinux32.org
[08:19:43] -!- DCyrax has joined #archlinux32
[08:22:06] * buildmaster failed to execute a mysql query - can you have a look at "tmp.mysql-functions.query.2019-08-03T08:12:52.OrvVw1.stdin"?.
[09:47:26] -!- DepositePirate has quit [Remote host closed the connection]
[09:47:52] -!- DepositePirate has joined #archlinux32
[09:53:41] * buildmaster resumes sanity.
[11:19:36] -!- davor_ has joined #archlinux32
[11:22:32] -!- davor has quit [Ping timeout: 245 seconds]
[11:22:32] davor_ is now known as davor
[12:17:23] <buildmaster> pentium4/lilypond is broken (says eurobuild6-4): https://archlinux32.org
[12:32:53] <buildmaster> pentium4/epson-inkjet-printer-escpr is broken (says eurobuild6-1): https://archlinux32.org
[13:25:25] -!- abaumann has joined #archlinux32
[13:25:26] <buildmaster> Hi abaumann!
[13:25:26] <buildmaster> !rq abaumann
[13:25:27] <phrik> buildmaster: <abaumann> The next step will be to distribute code to neighbour servers or use the Amazon cluster in the build script to run tests..
[13:25:38] <abaumann> two conclusions on the performance on the buildmaster:
[13:26:04] <abaumann> get-package-updates takes 10-15 minutes, if run every 30 minutes the buildmaster is almost only updating packages, so I changed that in the user systemd timer to 1h
[13:26:20] -!- thePiGrepper has quit [Ping timeout: 272 seconds]
[13:26:32] <abaumann> second conclusion: most slaves wait in 'uploading' for return-assigments to finish, they take 2 minutes or so per package.
[13:26:54] <abaumann> especially if many small packages are built, the slaves are basically always waiting to return an already finished package
[13:28:10] <abaumann> in return-assignment I see strange stuff hapenning, wait4 without a mysql running (waiting for what?) and millions of closes with EBADF
[13:28:31] <abaumann> the problem with bash is: it's inefficient and almost impossible to debug
[13:29:27] <abaumann> So, I'm concentrating on return-assigment to see why it takes that long..
[13:29:28] -!- abaumann has quit [Client Quit]
[13:33:25] -!- abaumann has joined #archlinux32
[13:33:26] <buildmaster> Hi abaumann!
[13:33:26] <buildmaster> !rq abaumann
[13:33:27] <phrik> buildmaster: <abaumann> naa.. not testing never has bad consequences.. ;-)
[13:33:50] <abaumann> mmh. it seems it takes a long time during mirror updates (I see tons of exec screen commands passing by)
[13:33:56] <abaumann> it's not mysql on the buildmaster
[13:37:12] <abaumann> get-assignments basically produce 20 seconds of high mysqld load on the buildmaster, then they are finished
[13:38:08] <abaumann> curl http://rmatthes.de
[13:38:09] <abaumann> ?
[13:38:56] <abaumann> goes into timeouts..
[13:39:49] <abaumann> so, is this a trigger_url pointing to a slow server?
[13:41:33] <abaumann> curl -v -L --connect-timeout 10 http://rmatthes.de 504 Gateway Time-out
[13:41:41] <abaumann> nothing I can do here :-)
[13:41:42] -!- abaumann has quit [Quit: leaving]
[13:45:26] -!- abaumann has joined #archlinux32
[13:45:26] <buildmaster> Hi abaumann!
[13:45:26] <buildmaster> !rq abaumann
[13:45:27] <phrik> buildmaster: <abaumann> deep42thought: I want to test the machine, not kill it. :-)
[13:45:53] <abaumann> doing all those triggered rsyncs on my mirror is maybe not very good for the other rsync server?
[13:46:07] <abaumann> and this just because some haskell packages could have stall versions..
[13:46:41] -!- abaumann has quit [Client Quit]
[13:54:39] -!- abaumann has joined #archlinux32
[13:54:40] <buildmaster> Hi abaumann!
[13:54:40] <buildmaster> !rq abaumann
[13:54:41] <phrik> buildmaster: <abaumann> stackoverflow is more or less a collection of esotheric modern woodoo instead of a source for technical information :-)
[13:55:03] <abaumann> I have a really bad feeling about this rsync.php trigger. It seems to be plain http and has no IP restriction!
[13:56:10] -!- abaumann has quit [Client Quit]
[14:03:20] -!- abaumann has joined #archlinux32
[14:03:20] <buildmaster> Hi abaumann!
[14:03:20] <buildmaster> !rq abaumann
[14:03:21] <phrik> buildmaster: <abaumann> all slaves busy.. wait.. one slave in the north/western part of.. resists.. :-)
[14:05:22] <abaumann> SELECT MAX(`ssh_log`.`date`) FROM `ssh_log` WHERE `ssh_log`.`build_slave`=10;
[14:05:24] <abaumann> takes 10-20 seconds
[14:05:38] <abaumann> though there is an index on it, which should return an answer in under-1ms!
[14:06:43] <abaumann> SELECT MAX(date) FROM `ssh_log` WHERE `ssh_log`.`build_slave`=1;
[14:06:54] <abaumann> shoukd last longest, as it is the oldest build slave :-)
[14:07:00] <abaumann> SELECT MAX(date) FROM `ssh_log` WHERE `ssh_log`.`build_slave`=21;
[14:07:04] <abaumann> returns very fast.
[14:07:13] <abaumann> Why do we keep millions of ssh_logs in the database?
[14:10:40] <abaumann> select build_slave,count(build_slave) from ssh_log groyp by build_slave;
[14:10:43] <abaumann> syntax error
[14:10:47] <abaumann> is this a database or a joke?
[14:12:25] <abaumann> aha. using date with a where build_slave eliminates index usage
[14:12:38] <abaumann> maybe a multipart index on date AND build_slave is possible..
[14:13:31] <abaumann> create index ssh_log_build_slave_date_idx on ssh_log(build_slave,date);
[14:13:35] <abaumann> let's see :-)
[14:13:44] * abaumann is in full database optimization mood :-)
[14:15:07] <abaumann> aha, under 0.001s now.
[14:15:27] <abaumann> because, if properly indexed the size of ssh_log is no problem. :-)
[14:16:33] <abaumann> now the tricky question is: can we drop the individual indexes on build_slave and date?
[14:17:21] <abaumann> my rusy database brain says yes.
[14:18:37] <abaumann> *rusty
[14:24:43] <abaumann> the index on date is necessary for the build-slave.php script, it seems
[14:24:53] <abaumann> well, doens't matter to have two indexes there..
[14:34:41] <abaumann> mysql_determine_majority_build_slave_architecture_id seems to use the new combined key.. good.
[14:41:59] <abaumann> mirror_refresh_trigger_urls='http://rmatthes.de/sync.php'
[14:42:03] <abaumann> aha. on the buildmaster.
[14:42:27] <abaumann> let's disable it (just for a test)
[14:45:20] -!- thePiGrepper has joined #archlinux32
[14:45:23] <abaumann> ahem..
[14:45:54] <abaumann> I commented it out. It basically blocked the buildmaster quite nicely :-)
[14:48:35] <abaumann> I also increased the polling wait in the build slaves again, now that get-assignment is faster it should not pose a problem anymore..
[14:58:57] <abaumann> deep42thought: in the home of the master on the buildmaster is a small program called 'printprocs' which shows the pid, the parent pid, the age in seconds and the command being executed. Quite helpful to get some clues, what's fast and what not :-)
[15:02:30] * abaumann must leave now and let's the buildmater in peace - for now
[15:02:31] -!- abaumann has quit [Quit: leaving]
[16:48:24] -!- emordyks has joined #archlinux32
[16:48:43] -!- skydrome has quit [Ping timeout: 244 seconds]
[16:50:00] -!- emordyks has quit [Client Quit]
[18:07:32] <buildmaster> pentium4/haskell-rio-orphans are broken (says eurobuild6-2): https://archlinux32.org
[18:31:23] -!- skydrome has joined #archlinux32
[18:38:22] -!- skydrome has quit [Ping timeout: 245 seconds]
[18:56:55] -!- ofara_ has quit [Quit: ofara_]
[19:10:56] -!- ofara_ has joined #archlinux32
[19:35:43] -!- davor has quit [Ping timeout: 268 seconds]
[19:36:15] -!- davor has joined #archlinux32
[19:36:43] -!- thePiGrepper has quit [Ping timeout: 258 seconds]
[20:34:40] <buildmaster> i686/haskell-sbv is broken (says eurobuild6-2): https://archlinux32.org
[20:46:56] <buildmaster> pentium4/shellcheck is broken (says eurobuild6-2): https://archlinux32.org
[20:46:57] <buildmaster> i686/hq is broken (says nlopc46): https://archlinux32.org
[20:58:13] <buildmaster> pentium4/hq is broken (says eurobuild6-6): https://archlinux32.org
[21:09:37] <buildmaster> i686/texmacs are broken (says eurobuild6-3): https://archlinux32.org
[21:10:05] <buildmaster> pentium4/texmacs are broken (says eurobuild6-4): https://archlinux32.org
[21:46:39] -!- DCyrax has quit [Remote host closed the connection]
[23:07:44] -!- skydrome has joined #archlinux32