17 Commits

Author SHA1 Message Date
Christoph Reiter
e2042058f1 gh: improve repo caching
We were caching based on the build type, but for most build types the repo
is the same, so cache one level below instead.
2025-03-07 12:04:27 +01:00
Christoph Reiter
1ef3f8f5f5 Use new pygithub global lazy feature
Requires 2.6.0. Means data is only fetched if it is accessed,
so fewer API calls for us (hopefully).
2025-03-07 12:04:21 +01:00
Christoph Reiter
59bb7f6f18 fetch-assets: test all downloaded files with zstd
Test them before moving them to the final location.
This makes the download fial of there is some file corruption etc.

This adds a dependency on the zstd exectuable for the fetch-assets
command.

Motivated by https://github.com/msys2/msys2-main-server/issues/42
2024-05-25 14:03:54 +02:00
Christoph Reiter
1ed7c15c97 flake8 fixes 2023-10-22 16:19:57 +02:00
Christoph Reiter
e6700d2089 Disable the new pygithub read throttling
It seems a bit excessive and doesn't take into account that
lots of our request hit the cache via etags.
2023-10-16 20:28:22 +02:00
Christoph Reiter
d1048413f8 Update pygithub to v2
It now has its own default retry logic that fits the GH API,
so no longer pass our own and assume it handles things better.

The datetimes are now timezone aware, so we no longer have to fix
them.
2023-10-16 20:21:47 +02:00
Christoph Reiter
79a45bf6c7 Require a user confirmation for manual uploads
We currently allow some users to manually upload packages (in case
they take too long for CI, or to bootstrap things).

In case of an account takeover this would allow an attacker to upload/replace
files in staging. To reduce the risk a bit ask for confirmation when downloading
the manually uploaded files.

Also add a "--noconfirm" option so we can avoid the questions in the staging
download script.

Ideally we would require users to sign their files, but this helps a bit at least.
2023-09-16 14:07:08 +02:00
Christoph Reiter
a3a5c1da40 update-status: only replace the status file if something has changed
Before uploading the status file we make a cached request for the old status
content and if there is no difference we don't upload anything.

This reduces the amount of write API calls and the amount of useless
packages.msys2.org refreshes a bit.
2023-08-16 20:40:22 +02:00
Christoph Reiter
d68ad18de2 Port to new pygithub auth API 2023-06-30 22:00:01 +02:00
Christoph Reiter
dfc132af9d download_text_asset: don't use the cache
This gets called in a threadpool and something in requests_cache
deadlocks.
2023-05-08 09:15:06 +02:00
Christoph Reiter
b51cfd02af Avoid upload_asset()
git fails to delete files we have uploaded, and I'm wondering if
upload_asset() is somehow keeping a handle open. While I can't find
anything suspicious in pygithub let's make the file handling explicit
and open/close ourselves.
2023-04-07 19:21:42 +02:00
Christoph Reiter
e3bb36afac more type annotations 2023-03-24 14:09:24 +01:00
Christoph Reiter
956ac59246 write_build_plan: remove the check for running workflows
This was required to avoid running multiple builds at the same time.
But GHA now has concurrency groups which solves the same problem,
so drop that code
2023-03-23 11:59:37 +01:00
Christoph Reiter
e2ca121180 Replace readonly with write everywhere
less confusing, at least for me
2023-03-23 11:58:17 +01:00
Christoph Reiter
b453032363 Get rid of MAIN_REPO
in most cases at least. either derive from the current
build type, or via get_current_repo() which reads the
GITHUB_REPOSITORY env var.
2023-03-23 11:58:17 +01:00
Christoph Reiter
88871c4cb0 Rename _PathLike to PathLike
it's no longer internal
2023-03-23 11:17:10 +01:00
Christoph Reiter
87f0603c87 Split the code up into separate modules
with minimal code changes
2023-03-22 09:59:05 +01:00