623 Commits

Author SHA1 Message Date
Christoph Reiter
3637fea711 Update dependencies 2022-09-04 10:33:47 +02:00
Christoph Reiter
e23492ee15 also retry on 502
We just got "502 {"message": "Server Error"}" on a DELETE
2022-09-02 21:53:46 +02:00
Christoph Reiter
9f4f288d00 retry HTTP requests which return 500
We are getting "500 null" randomly recently, mabye this helps.
2022-08-27 13:51:32 +02:00
Christoph Reiter
b36a4da1da Use a temporary pacman.conf during building
Up until now we created a backup of pacman.conf and restored it after
the build was done. This can leave the environment in an undefined state
if something crashes inbetween.

Instead create a temporary pacman.conf and use that during building.
In theory pacman allows setting a custom config via "--config", but
makepkg doesn't expose this, so that's not working. Luckily makepkg
allows overriding the pacman path via the PACMAN env var, so we create
a temporary script which just forwards everything to pacman and always
sets the temporary config.
2022-08-26 15:09:05 +02:00
Christoph Reiter
5ecdbc97a7 thinko
we are in bash here, not powershell...
2022-08-21 21:52:59 +02:00
Christoph Reiter
0d4680c01f Don't use workflow inputs directly in scripts
They could inject commands that way. Instead assign them
to an env var and then use that env var in the powershell scripts.

We want to open those controls up to more people, so we need to make
sure they can only change the values and not extract tokens etc.

Fixes #60
2022-08-21 20:37:07 +02:00
Christoph Reiter
9374b1d9b4 main: run update-status at the end 2022-08-15 14:03:50 +02:00
Jeremy Drake
22ea970beb CI: uncomment clangarm64 from pacman.conf.
instead of adding from scratch.  Once the commented-out section was
added, the grep would match that and no longer run the sed to add it.

Also remove line adding clang32 section because that was added to
default pacman.conf (and was thus a no-op).
2022-08-14 22:39:52 +02:00
Christoph Reiter
45c6b89ec7 Update the build status before stopping due to timeout
In case the job stops because it has reached the time limit it would
not update the build status and just quit. Move the timeout check
a bit later to acoid that.
2022-08-08 18:50:27 +02:00
Christoph Reiter
70c6903191 CI: update python to 3.10 and setup-python to v4 2022-08-04 21:45:15 +02:00
Christoph Reiter
f33be41b0f Update dependencies 2022-08-04 21:39:19 +02:00
Christoph Reiter
5f53dab6de Enable winjitdebug to workaround python crashing issues
Why this helps, I don't know..
2022-07-24 12:31:25 +02:00
Christoph Reiter
4dbd2618fb Update deps 2022-07-21 21:54:07 +02:00
Christoph Reiter
a43bdf9479 Add a comment as to why we split up repo-add calls
This was pointed out here: 7d84a7e086 (r75916830)
2022-06-30 21:07:00 +02:00
Christoph Reiter
9360a8eebe Update dependencies 2022-06-30 21:04:54 +02:00
Christoph Reiter
7d84a7e086 Limit the amount of packages added with repo-add in one go
It errors out if there are too many (maybe memory?)
2022-06-12 18:28:48 +02:00
Christoph Reiter
ea46306e71 Add config key for limiting the max job count
We are hitting the API limit again, so reduce from 15 to 12.
This also allows self hosted runners to limit to 1 if needed.
2022-06-11 18:32:11 +02:00
Christoph Reiter
97faefb5b3 Update deps
requests-cache now provides an option to always revalidated, so use that.
Before it mostly worked by accident.
2022-05-29 10:29:38 +02:00
Christoph Reiter
745e5e2c40 Fetch pgp keys before upgrading packages
To avoid any ABI bump breaking pgp
2022-05-08 20:08:43 +02:00
Christoph Reiter
84315e8e56 cache: include the job ID in the cache key
we just want to store/replace it every time, so we need it to be as unique as possible
2022-05-04 19:53:03 +02:00
Christoph Reiter
0a302154b0 Cache the cache
This should lead to cache hits on the first calls from the spawned
jobs right after the scheduler runs.
2022-05-04 18:55:48 +02:00
Christoph Reiter
4a5355f5dc Another try at fixing the cache race
Turns out disabling the cache just disables the monkey patching,
so we have to disable when creating the session object and not when
we are using the cache.

Create a session object without cache in the main thread at first,
so that the download code can re-use it as is later on.
2022-04-30 22:19:15 +02:00
Christoph Reiter
88b49f2c6a Avoid disabling the cache in a thread pool
It isn't thread safe, so wrap the outer code instead and just
assert in download_asset() that the caching is disabled so it's
not called with caching by accident.
2022-04-30 22:07:40 +02:00
Christoph Reiter
1684dff8bc Update mypy 2022-04-30 17:28:08 +02:00
Christoph Reiter
8870b3a342 Use requests-cache for adding etag/last-modified based caching
This doesn't speed things up usually, since we still make the same amount
of requests, but it doesn't count against the rate-limit in case there
is a cache hit. Also there is a smaller chance of things going wrong,
since we don't transfer any payload.

The cache is store in a .autobuild_cache directory using a sqlite DB.
2022-04-30 17:15:18 +02:00
Christoph Reiter
258256e739 use lru_cache for Python 3.8 compat 2022-04-30 16:23:49 +02:00
Christoph Reiter
133ce88284 Cache the main github api instances
this leads to a shared session, and a bit fewer requests
2022-04-30 16:16:54 +02:00
Christoph Reiter
099438dc3f queue_website_update: just log errors instead of failing
this is optional really
2022-04-30 12:32:10 +02:00
Christoph Reiter
94d87dac25 Retry non-pygithub HTTP requests as well
There isn't an easier way to enabled retries with requests sadly.
This also shares the session between all non-pygithub requests, so
could make things a bit faster.
2022-04-30 12:27:43 +02:00
Jeremy Drake
4384e62d01 Use labels to restrict self-hosted runner selection. 2022-04-19 07:46:26 +02:00
Christoph Reiter
cb4434c72b CI: Move the clear-failed actions into its own workflow
So they can be run independently.
2022-04-18 17:29:31 +02:00
jeremyd2019
892e1a3206 break clang/libc++ cycle
before libc++ was split off from clang package, it was built after clang within the same PKGBUILD, so this order seems reasonably safe.

Also remove a couple of prior cycle breaks from before it was possible to break them manually in a run.  These packages are not related by the same source repo and release, like mingw-w64 and llvm-project are, so are less likely to consistently require a cycle break on every upstream update.
2022-04-05 09:39:39 +02:00
Christoph Reiter
5f5d895cb1 move more common inputs up 2022-04-01 20:11:18 +02:00
Christoph Reiter
e4c2d446d2 Include the input name in the description 2022-04-01 20:09:25 +02:00
Christoph Reiter
cfe519fbb0 clear-failed: Allow clearing the failed state for packages via a workflow input 2022-04-01 20:05:38 +02:00
Christoph Reiter
1b14e2ed4d cycles: skip cycles where one of the packages is already built 2022-04-01 18:36:32 +02:00
Christoph Reiter
8c060f3142 cycles: show the version change of all packages 2022-04-01 15:44:45 +02:00
jeremyd2019
c2f77181d7 work around powershell arg parsing fail
It appears that powershell doesn't properly handle an empty argument, resulting in all the subsequent arguments being shifted left by one.

So, don't specify --optional-deps argument if it is empty.
2022-03-30 23:56:20 +02:00
Christoph Reiter
cd67c3a66a show: also set optional deps 2022-03-30 17:29:59 +02:00
Christoph Reiter
5e037680d6 also add optional deps when checking if we should run 2022-03-30 17:17:04 +02:00
Christoph Reiter
5f628fb63a and now for real 2022-03-30 17:03:37 +02:00
Christoph Reiter
777bbb73af Try to make it possible to pass optional dependencies to the workflow
The idea is that in case of a cycle we explicitely break it on a case
by case basis.
2022-03-30 17:00:20 +02:00
Christoph Reiter
c1807c19a7 show the cycles also when writing the build plan
Otherwise we don't see it in CI, since the rest is skipped if there
is nothing to build.
2022-03-30 10:08:04 +02:00
Christoph Reiter
20ba53752d show: default to not fetch the build log URLs
Instead add a --details option.

It's quite slow and rarely needed, so default to off.
2022-03-30 10:00:18 +02:00
Christoph Reiter
81dd6cabad show: include a list of dependency cycles
This gives all cycles in the queue right now, ignoring the build
status of the packages.

If one part of the cycle is already built then it will not matter.
2022-03-30 09:57:02 +02:00
Christoph Reiter
5c6f39a511 break cycle 2022-03-29 20:26:19 +02:00
Christoph Reiter
e7fdb6dab2 fix yaml types 2022-03-19 10:17:22 +01:00
Christoph Reiter
d423d68901 CI: test with Python 3.10 2022-03-19 10:16:30 +01:00
Christoph Reiter
4cc7908a95 CI: update to Python 3.9 2022-03-19 10:15:12 +01:00
Christoph Reiter
0cf933cc9b CI: run every 3 hours instead of 4
it just takes 20 secs if there is nothing to do
2022-03-18 16:36:49 +01:00