#archlinux-ports | Logs for 2017-04-21

Back
[00:03:43] -!- isacdaavid__ has joined #archlinux-ports
[00:07:10] -!- isacdaavid has quit [Ping timeout: 240 seconds]
[01:16:50] -!- isacdaavid__ has quit [Quit: isacdaavid__]
[01:17:13] -!- isacdaavid has joined #archlinux-ports
[01:21:35] -!- isacdaavid has quit [Ping timeout: 240 seconds]
[01:44:50] -!- isacdaavid has joined #archlinux-ports
[01:47:57] -!- yar has quit [Ping timeout: 240 seconds]
[02:29:55] -!- Faalagorn has quit [Remote host closed the connection]
[02:30:45] -!- Faalagorn has joined #archlinux-ports
[02:31:01] -!- Faalagorn has quit [Client Quit]
[03:24:01] -!- isacdaavid_ has joined #archlinux-ports
[03:26:40] -!- isacdaavid has quit [Ping timeout: 240 seconds]
[05:07:19] <phillid> What's the progress on getting repos up and running?
[05:07:30] <phillid> I have a beefy Xeon workstation that I can help out with
[05:17:47] <eschwartz> How about you run tests of the current code and see what needs fixing? :) https://github.com
[05:32:41] <phillid> 👍
[05:33:04] <phillid> Running in an i686 chroot on my workstation, but I also have real machines to test on
[05:41:14] -!- alyptik has joined #archlinux-ports
[06:17:14] -!- City-busz has quit [Remote host closed the connection]
[06:20:30] -!- City-busz has joined #archlinux-ports
[06:20:31] -!- City-busz has quit [Changing host]
[06:20:31] -!- City-busz has joined #archlinux-ports
[06:29:01] -!- deep42thought has joined #archlinux-ports
[06:36:46] <phillid> Hmm that script seems quite sensitive
[06:38:34] <phillid> For example; it bails in a messy way when no `work` dir exists with the right files in it
[06:38:59] <phillid> But surely this is a valid use case. What action should be taken on the first use like this?
[06:39:23] <deep42thought> simply touch all files and directories which are needed
[06:39:34] <deep42thought> should be "work" dir only
[06:39:36] <deep42thought> iirc
[06:39:55] <deep42thought> additionally: it's a good idea to give it a small chunk of commits first - otherwise it takes like forever
[06:40:07] <deep42thought> ah
[06:40:09] <phillid> touching didn't work
[06:40:18] <deep42thought> you also need the repos in the same places as I had them
[06:40:26] <phillid> I had to put some valid ref into each of the files that are supposed to hold refs
[06:40:28] <deep42thought> (sry, it's really just a first attempt)
[06:40:42] <phillid> No no don't apologise, I'm not dissing it
[06:40:59] <deep42thought> ah, yes, haven't pushed the messy script yet, which determines the first ref of the repo otherwise
[06:41:07] <phillid> Ah okay
[06:41:24] <phillid> I did try putting the first commit's ref into those files and encountered some errors with large renaming (?)
[06:41:27] -!- isacdaavid_ has quit [Read error: Connection reset by peer]
[06:41:37] <deep42thought> yes
[06:41:40] <phillid> Not sure where those messages were coming from, so I just got lazy and put HEAD~100 into them instead 😛
[06:42:14] <deep42thought> yes, that's a good idea to test it
[06:42:28] <phillid> After that it seems to work and track updates nicely
[06:43:26] <deep42thought> I thought of accellerating the script in case of many updates - so the first run becomes bearable
[06:44:06] <phillid> accelerating?
[06:44:53] <deep42thought> well, if you give it all commits, it needs around 30min to get all the dependencies correct and >~2h to sort them (I didn't wait that long)
[06:45:02] <phillid> oh jeepers
[06:45:40] <deep42thought> not a real surprise, since it's around 10k packages and the runtime is currently O(n^2) instead of optimal O(n log(n))
[06:46:14] <phillid> Still pretty slow for 10k packages. I guess that's just the choice of language
[06:46:41] -!- City-busz has quit [Read error: Connection reset by peer]
[06:46:48] <deep42thought> yeah, sure
[06:47:33] <deep42thought> here, this version deals correctly with missing *.repository files
[06:47:39] <phillid> Oh cool
[06:47:47] -!- City-busz has joined #archlinux-ports
[06:47:47] -!- City-busz has quit [Changing host]
[06:47:47] -!- City-busz has joined #archlinux-ports
[06:48:08] <deep42thought> as you might see, it's 3 weeks old - but I'm currently somewhat busy making high harmonics in the lab
[06:53:12] <phillid> Oh cool, that revision goes better
[06:53:31] <phillid> Still requires manually touching of build-list and deletion-list, but it does the revisions correctly
[06:55:31] <deep42thought> yes, this change is in the pipe
[06:55:56] <deep42thought> but as I mentioned: it's not a good idea to run this script on an empty directory
[06:57:16] <phillid> True
[06:57:21] <deep42thought> the logic for resolving circular dependencies needs some tunig, too: currently it simply ignores different kinds of dependencies (make/check/runtime) until the packages can be compiled - however, this seems really unclean :-/
[06:58:48] <phillid> Hmm, especially with make deps
[07:01:23] <deep42thought> so the order is build packages depending on: nothing, makedeps, runtimedeps
[07:01:31] <deep42thought> I should change the latter two
[07:03:18] <phillid> I think so
[07:07:48] <deep42thought> anyway it's ugly
[07:08:05] <deep42thought> think of two small circles connected by one long path
[07:08:29] <deep42thought> it randomly drops dependencies, so the probability to break one of the circles is small
[07:12:09] <deep42thought> or even easier: one small circle and a lot of "clean" builds depending on any of the packages in the circle
[07:14:59] <eschwartz> depends is part of makedepends, unless it is declared in the package() function in which case makepkg ignores it altogether except for writing the metadata, and you can ignore it too ;)
[07:15:30] <eschwartz> anyway, tsort is your friend
[07:16:12] <eschwartz> (assuming you don't plan to rewrite it in C for speed, or something)
[07:16:26] <deep42thought> what's tsort?
[07:16:32] <eschwartz> man 1 tsort
[07:17:31] <deep42thought> I don't get it
[07:18:07] <deep42thought> I can define the order via FILE?
[07:19:01] <phillid> Oh very cool
[07:19:04] <phillid> I had no idea about tsort
[07:19:37] <phillid> Has to be acyclic though
[07:20:22] <eschwartz> deep42thought: yep, e.g. see how aurutils uses it to compute dependency chains for AUR packages
[07:21:39] <eschwartz> the manpage would be slightly more helpful if it pointed out the format I guess. See wikipedia for a more in-depth analysis
[07:22:33] <eschwartz> Or `info tsort|less`
[07:23:29] <eschwartz> phillid: what would you prefer anyway?
[07:23:54] <phillid> tsort looks nice
[07:24:26] <phillid> If we can reliably break dependency cycles…
[07:24:33] <phillid> How does pacman and others determine this?
[07:25:08] <eschwartz> using topological sort I am pretty sure... just written in C ;)
[07:25:56] <phillid> topilogical sort cannot work on cyclic data
[07:26:07] <phillid> topological*
[07:27:07] <eschwartz> that's why occasionally you get warnings that dependency circles are being loudly ignored.
[07:28:38] <phillid> Right, so how does it break the cycle if not using topo sort, since that's impossible to define one sorted list when the input is cyclic
[07:28:45] <phillid> Does it just choose some random ordering?
[07:30:20] <phillid> Hmm it would seem so
[07:30:33] <phillid> Well that solves the issue of cyclic deps then, surely; we can just wham it through tsort?
[07:30:57] <deep42thought> maybe tsort itself gives some error, when you defined a inconsistent ordering?
[07:31:03] <phillid> It does
[07:31:06] <deep42thought> good
[07:31:12] <eschwartz> there is no good answer, so tsort does the best job it can I think
[07:31:23] <phillid> It will provide information about those members participating in the cycle on stderr
[07:31:39] <phillid> eschwartz, Yeah I was under the impression that it was impossible
[07:31:51] <phillid> I now understand it's just that there's not one single answer
[07:31:53] <phillid> Makes sense
[07:32:54] <phillid> So more or less, looking at the big picture, we use this tool to generate a list of those packages which archlinux.org have changed since we last looked, then we go ahead and rebuild those pacakges to update them, right?
[07:34:33] <eschwartz> Yeah.
[07:35:03] <deep42thought> sounds nice :-)
[07:35:19] <eschwartz> Well, we have a list and then we crawl the dependency ordering using tsort to ensure deps are ordered as early on as possible
[07:36:22] <eschwartz> and like I said, aurutils already does this for AUR packages, see aurqueue
[07:36:54] <eschwartz> In fact, you *can* actually reuse aurqueue since it just reads */.SRCINFO
[07:37:07] <eschwartz> (alad is helpful that way it's very modular)
[07:38:29] <phillid> heh
[07:38:40] <phillid> IMO it would be best to get a mirror running as soon as we can
[07:38:55] <deep42thought> mirror is no problem
[07:39:01] <deep42thought> the content is ;-)
[07:39:06] <phillid> Well of cource
[07:39:08] <phillid> course*
[07:39:42] <phillid> Once we can get this build list pegged down nicely, I have a 32-thread xeon workstation that I can throw at compiling jobs
[07:40:35] <phillid> It expedites toolchain and kernel builds nicely for me ☺
[07:41:49] <deep42thought> should we simply ignore all runtime deps for the ordering?
[07:42:36] <phillid> At first thought I think we can get away with that
[07:42:43] <eschwartz> deep42thought: Yes
[07:42:51] <eschwartz> runtime deps are ignored by makepkg too
[07:43:13] <deep42thought> eschwartz: I don't think so
[07:43:27] <eschwartz> depends are added to makedepends, but depends declared in package() are ignored and just used for generating the package metadata
[07:43:50] <phillid> depends=() is runtime deps isn't it?
[07:43:57] <phillid> or am I having a brain fart
[07:44:03] <deep42thought> yes
[07:44:05] <eschwartz> phillid: it depends where it is defined.
[07:44:14] <phillid> Defined at the root of the PKGBUILD
[07:44:14] <eschwartz> toplevel, it is both runtime and makedepends
[07:44:26] <eschwartz> scoped to a package_*() function it is runtime only
[07:44:36] <phillid> yeah
[07:44:49] <eschwartz> anyway, aurqueue handles this already, it's pretty nice :)
[07:45:20] <phillid> What about checkdepends and friends?
[07:45:27] <phillid> s/and friends//
[07:45:36] <deep42thought> ok, probably I'll have a look at aurutils at the weekend, but now I have to leave. Cu later
[07:45:38] <eschwartz> those count, unless you run --nocheck
[07:45:44] <phillid> \o deep42thought
[07:45:51] -!- deep42thought has quit [Quit: Leaving.]
[07:45:58] <phillid> eschwartz, Yeah, so we'd be concerned with them too
[07:46:39] <eschwartz> yes, at least I hope we will run the testsuites :p
[07:46:50] <phillid> heh
[07:46:56] * eschwartz laughs at ML discussion of binutils/PIE
[07:46:59] <phillid> "No, arch64 ran those for us"
[07:47:15] * phillid cringes at both
[07:47:22] <phillid> That thread is terrib;e
[07:49:35] <phillid> I unsubscribed from that arch-general list a while ago because I got sick of low-quality discussino like that
[08:03:13] <eschwartz> Oh, remind deep42thought that for .SRCINFO you'll want to use dreisner's mksrcinfo code since it is much faster than initializing all of libmakepkg and that matters when you might be running against thousands of PKGBUILDs.
[08:03:17] <eschwartz> that sed is giving me hives
[08:03:46] <phillid> How does it get away with not using libmakepkg?
[08:04:56] <eschwartz> phillid: It came first, and contains the minimal logic necessary to source the PKGBUILD and print out the variable/array keys
[08:05:09] <phillid> Oh I remember that
[08:05:26] <phillid> But why do we use one tying into libmakepkg now? What did that offer over the original?
[08:05:36] <phillid> And what will we lose by not using it?
[08:06:29] <eschwartz> unifying things into libmakepkg means for the common one-off use case people don't need a new tool.
[08:06:55] <eschwartz> also mksrcinfo doesn't handle validpgpkeys for some reason
[08:08:20] <eschwartz> makepkg will lint the PKGBUILD for errors too
[08:34:59] <phillid> Ah
[08:45:06] <eschwartz> My PKGBUILDs don't need to be linted :p
[08:47:19] -!- yar has joined #archlinux-ports
[09:58:18] <phillid> heh
[09:59:58] -!- eschwartz has quit [Ping timeout: 240 seconds]
[11:35:27] -!- Faalagorn has joined #archlinux-ports
[13:12:02] -!- eschwartz has joined #archlinux-ports
[13:38:32] -!- eschwartz has quit [Ping timeout: 255 seconds]
[13:38:39] -!- guys has quit [Ping timeout: 260 seconds]
[14:20:25] -!- guys has joined #archlinux-ports
[14:26:19] -!- eschwartz has joined #archlinux-ports
[16:29:27] -!- qguv has quit [Ping timeout: 240 seconds]
[16:31:43] -!- qguv has joined #archlinux-ports
[18:27:36] -!- yar has quit [Excess Flood]
[18:30:50] -!- yar has joined #archlinux-ports
[18:47:59] -!- isacdaavid has joined #archlinux-ports
[20:03:36] -!- isacdaavid has quit [Quit: isacdaavid]
[20:30:37] -!- deep42thought has joined #archlinux-ports
[20:54:00] -!- shentey has joined #archlinux-ports
[21:00:47] -!- shentey has quit [Quit: Konversation terminated!]
[21:02:39] -!- shentey has joined #archlinux-ports