Befator-Inc-Firmen-Netzwerk/msys2-web
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
577 Commits 1 Branch 0 Tags
82b4e4e4eef0bbabb2dfd1256ce7e52e32da424d
Commit Graph

59 Commits

Author SHA1 Message Date
Christoph Reiter
f2ed76293f Rename pkgmeta to pkgextra 
So things are consistent
 2023-09-17 18:59:12 +02:00
Christoph Reiter
cdd119e863 Parse the pkgmeta data from the srcinfo json file 
See https://github.com/msys2/MSYS2-packages/issues/4019
 2023-09-17 18:59:12 +02:00
Christoph Reiter
b2843b55f1 Some more logging 
configure the root logger to log INFO for all loggers to stdout and
add our own logger which logs from DEBUG and up.
 2023-08-19 23:43:21 +02:00
Christoph Reiter
789c72ae75 fetch: send cache request headers when checking for changes 
Up until now it only did a HEAD request and compared the returned
cache headers to see if something changed. By also sending the
request cache headers, the other side can potentially respond from their
caching layer and not hit any actual service.

With this we can reduce the checking interval without harming any
external service.
 2023-08-15 22:04:23 +02:00
Christoph Reiter
b5a6db9025 Split the pkgmeta code into its own file 2023-07-22 17:43:47 +02:00
Christoph Reiter
515198449c Add proper pypi integration 
We now fetch the cached pypi API responses and extract the version
and the oldest date of any version artifact
 2023-07-03 19:55:50 +02:00
Christoph Reiter
7bd958cca3 Link to pypi if we know the pypi name 2023-07-02 14:45:22 +02:00
Christoph Reiter
29cb2a56e7 parse_repo: log the correct URL in case we don't load the files DB 2023-06-20 18:00:28 +02:00
Christoph Reiter
75872f30e3 fetch: don't download the db including the package files 
there really is no need to, we only need the version really.
 2023-05-21 21:05:46 +02:00
Christoph Reiter
7ca9efda81 fetch: also parse cygwin mingw64 packages 
Register them under a separate "cygwin-mingw64" name.
 2023-05-20 09:59:55 +02:00
Christoph Reiter
78a636b3c8 Always prefer Arch versions over AUR and Cygwin 
AUR contains dev versions we don't care about and cywin
sometimes uses a different versioning scheme.

Only if there is no matching Arch package consider the other ones
 2023-05-12 21:35:11 +02:00
Christoph Reiter
8db831b53c httpx: always use raise_for_status() 
otherwise we ignore 404 etc..
 2023-05-07 00:14:15 +02:00
Christoph Reiter
96c1dc7365 Make sure to keep a reference to asyncio tasks 
as suggested in the docs
 2023-04-22 20:33:31 +02:00
Christoph Reiter
514c16f963 arch: provides should only be a fallback 
fallout from the arch/aur split
 2023-04-13 00:38:43 +02:00
Christoph Reiter
5569252cee Generalize external references and handle AUR separately 
Remove all cygwin/arch specific code from the core models to make
it easier to add more references.
 2023-04-11 20:02:55 +02:00
Christoph Reiter
6b244070f8 Merge data structures for cygwin/arch infos 2023-04-11 19:33:08 +02:00
Christoph Reiter
8dcc5b99e3 Link to archlinux.org instead of www.archlinux.org 
www.archlinux.org redirects to archlinux.org now, so link to the
new location.
 2023-03-17 11:29:00 +01:00
Christoph Reiter
54f4164f02 cygwin: still skip test versions 2023-01-24 19:37:49 +01:00
Christoph Reiter
bbb3458aad cygwin: always take the newest version available 
In case a previous version obsoleted a package that might still stay
around and come first in setup.ini, so always take the newest version
in case there are multiple entries.
 2023-01-24 19:29:32 +01:00
Christoph Reiter
4a65e2d54b Be more explicit about showing virtual packages 
Up until now we would show a list of packages and providing
packages. That's confusing when you don't scroll down and not
really obvious.

Instead if it's a virtual package show a simple page listing
the providing packages.

And in case a package of that name exists, show the other packages
providing the same name inline ("Provided By")
 2023-01-06 07:59:29 +01:00
Christoph Reiter
7be7b56370 queue: also read the msys2-arm build status 
Both builders will upload the complete build status when they run,
but up until now we only read the one from the main autobuild repo.
This meant that if the arm runner was building something without
the main autobuild running the website wouldn't notice and always
show the out of date state until the next scheduled run.

This now loads both and takes the newest according to the last-modified header
send by github. To keep local caching working during development we
store the last-modified as the cache file mtime.
 2022-12-30 10:28:07 +01:00
Christoph Reiter
9c6be6c7e3 Switch to new mapping format 
This adds the following things:

* the mapping data is now sourced from the package repositories
* cygwin mapping works too (for example see "msys2-runtime")
* packages can be marked internal so they are never linked
  to arch or cygwin (for example see "base")
 2022-12-29 13:53:29 +01:00
Christoph Reiter
4d16111fb4 arch mapping: add provided packages as a fallback 
This gives us a few more matches
 2022-12-29 06:32:22 +01:00
Christoph Reiter
498a10897f Use the new repo package prefix in more places 
instead of special casing based on repo and package names
 2022-12-28 17:37:05 +01:00
Christoph Reiter
7b09145e9a Handle the new build status format 
Changed in https://github.com/msys2/msys2-autobuild/commit/307799fd271fdfeeffec30c8
 2022-12-27 16:19:21 +01:00
Christoph Reiter
e0b0ae48da Add a base group collection 
This is conceptionally similar to a pkgbase, but for groups.
Something like this doesn't really exist on the PKGBUILD level,
but end users can't really work with pkgbase with pacman either,
so for them that doesn't matter.

It still gives the end user the same structure as with packages,
there is a base thing that has multiple instances for each environment,
and searching might be a bit easier.

The UX could be improved, but the basics are there at least.

Fixes #47
 2022-12-26 12:00:45 +01:00
Christoph Reiter
b595541183 Allow two updates per 5 minutes 
Now that we only update what is needed, we can allow more of them
 2022-08-01 08:14:55 +02:00
Christoph Reiter
43fc9c48e5 Split up the source update checks 
Check for each source instead of updating everything every time
 2022-08-01 08:06:12 +02:00
Christoph Reiter
140a572948 Make the URL update check reusable 2022-08-01 07:52:09 +02:00
Christoph Reiter
f4f1d592d6 Clean up the app config 2022-08-01 07:35:06 +02:00
Christoph Reiter
1ec7244b13 Rename arch_mapping to external_mapping 
we might extend it with cygwin mappings in the future
 2022-08-01 07:16:43 +02:00
Christoph Reiter
ec9760f6d4 Extract the upstream versions at parsing time 
It was a mix of everything right now, make sure we only work
with upstream versions after parsing.
 2022-07-31 08:59:22 +02:00
Christoph Reiter
988e1b41dc cygwin: switch to a better mirror and use the compressed index 2022-07-31 08:21:07 +02:00
Christoph Reiter
6eff4ff779 Download all of the AUR metadata instead of searching via rpc 
Turns out they provide it as a json file which gets updated every 15 minutes,
which is good enough for us.
 2022-07-31 08:20:39 +02:00
Christoph Reiter
5d324dad8f srcinfo files are now compressed 2022-07-22 17:57:17 +02:00
Christoph Reiter
6ba3c8bfca fetch: wait 5 seconds after we get triggered via the api 
it seems sometimes we update things at github, but it takes some time
until our server sees the new version, so wait a bit.
 2022-01-23 10:53:15 +01:00
Christoph Reiter
e8aa409ced parse_repo: iterate over tar object 
this is ~20% faster compared to calling getmembers()
 2021-11-27 09:48:45 +01:00
Christoph Reiter
09cbac7fb0 Add support for zstandard compressed repo DBs 2021-11-26 16:13:22 +01:00
Christoph Reiter
79d4e9d202 Update python deps 
httpx now disables redirects by default
 2021-10-18 08:23:55 +02:00
Christoph Reiter
bd6fd47cc9 Use mirror.msys2.org for direct download URLs 2021-07-02 11:16:23 +02:00
Christoph Reiter
0a1435a88f Some cleanups for the update trigger 
Use an Event() so we can queue an update during an update as well.
Use lru_cache() for the singleton event.
 2021-05-16 19:54:05 +02:00
Christoph Reiter
dca64c5797 api: add an endpoint for triggering an update 
Rate limited to once every 5 minutes.
Can be called from GHA for example.
 2021-05-16 15:40:45 +02:00
Christoph Reiter
7f7bb0aee4 Move the cache dir to the top level directory 2021-05-14 07:45:15 +02:00
Christoph Reiter
379079823f Also cache all AUR calls when started with --cache 
Hash the URL so there is no limit, make AUR URLs deterministic
 2021-05-13 20:30:31 +02:00
Christoph Reiter
66c162a3e9 Fetch AUR versions in multiple chunks 
We get back errors due to too long URLs sometimes
 2021-04-30 07:41:06 +02:00
Christoph Reiter
be72c2f847 Use head requests again for fetching the response headers 
Seems like github now returns the correct HTTP headers for heade requests
as well.
 2021-02-15 18:09:09 +01:00
Christoph Reiter
bf59548484 Print exceptions to stdout 2021-01-30 21:19:32 +01:00
Christoph Reiter
00ec82e7a2 queue: add a build status column 2021-01-22 19:52:15 +01:00
Philippe Ombredanne
9fd2d89154 Fix minor typo in WARNING 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
 2020-12-19 14:55:50 +01:00
Christoph Reiter
c7ee6eb2bb Don't hide VCS packages 
Treat them like all other packages. They used to cause various problems
by hitting edge cases, but this should all be fixed now.
 2020-10-01 19:15:20 +02:00