An update from GitHub: https://github.com/orgs/community/discussions/159123#discussioncomment-13148279
The rates are here: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28
- 60 req/hour for unauthenticated users
- 5000 req/hour for authenticated - personal
- 15000 req/hour for authenticated - enterprise org
60 req/hour for unauthenticated users
That’s low enough that it may cause problems for a lot of infrastructure. Like, I’m pretty sure that the MELPA emacs package repository builds out of git, and a lot of that is on github.
Do you think any infrastructure is pulling that often while unauthenticated? It seems like an easy fix either way (in my admittedly non devops opinion)
It’s gonna be problematic in particular for organisations with larger offices. If you’ve got hundreds of devs/sysadmins under the same public IP address, those 60 requests/hour are shared between them.
Basically, I expect unauthenticated pulls to not anymore be possible at my day job, which means repos hosted on GitHub become a pain.
Ah yeah that’s right, I didn’t consider large offices. I can definitely see how that’d be a problem
Same problem for CGNAT users
Quite frankly, companies shouldn’t be pulling Willy nilly from github or npm, etc anyway. It’s trivial to set up something to cache repos or artifacts, etc. Plus it guards against being down when github is down, etc.
It’s easy to set up a cache, but what’s hard is convincing your devs to use it.
Mainly because, well, it generally works without configuring the cache in your build pipeline, as you’ll almost always need some solution for accessing the internet anyways.
But there’s other reasons, too. You need authentication or a VPN for accessing a cache like that. Authentications means you have to deal with credentials, which is a pain. VPN means it’s likely slower than downloading directly from the internet, at least while you’re working from home.
Well, and it’s also just yet another moving part in your build pipeline. If that cache is ever down or broken or inaccessible from certain build infrastructure, chances are it will get removed from affected build pipelines and those devs are unlikely to come back.
Having said that, of course, GitHub is promoting caches quite heavily here. This might make it actually worth using for the individual devs.
If I’m using Ansible or something to pull images it might get that high.
Of course the fix is to pull it once and copy the files over, but I could see this breaking prod for folks who didn’t write it that way in the first place
That’s low enough that it may cause problems for a lot of infrastructure.
Likely the point. If you need more, get an API key.
Or just make authenticated requests. I’d expect that to be well within with capabilities of anyone using MELPA, and 5000 requests per hour shouldn’t pose any difficulty considering MELPA only has about 6000 total packages.
This is my opinion on it, too. Everyone is crying about the death of Github when they’re just cutting back on unauthenticated requests to curb abuse… lol seems pretty standard practice to me.
I didn’t think of that - also for nvim you typically pull plugins from git repositories
That’ just how the cookie crumbles.
Its always blocked me from searching in firefox when I’m logged out for some reason.
Wow so surprising, never saw this coming, this is my surprised face. :-l
is authenticated like when you use a private key with git clone? stupid question i know
also this might be terrible if you subscribe to filter lists on raw github in ublock or adguard
is authenticated like when you use a private key with git clone
Yes
also this might be terrible if you subscribe to filter lists on raw github in ublock or adguard
Yes exactly why this is actually quite problematic. There’s a lot of HTTPS Git pull remotes around and random software that uses raw.githubusercontent.com to fetch data. All of that is now subject to the 60 req/hr limit and not all of it will be easy to fix.
This is specific to the GH REST API I think, not operations like doing a git clone to copy a repo to local machine, etc.
These changes will apply to operations like cloning repositories over HTTPS, anonymously interacting with our REST APIs, and downloading files from raw.githubusercontent.com.
downloading files from raw.githubusercontent.com
oh fuck, this is going to break stuff.
These changes will apply to operations like cloning repositories over HTTPS, anonymously interacting with our REST APIs, and downloading files from raw.githubusercontent.com.
Maybe charge OpenAI for scrapes instead of screwing over your actual customers.
I honestly don’t really see the problem here. This seems to mostly be targeting scrapers.
For unauthenticated users you are limited to public data only and 60 requests per hour, or 30k if you’re using Git LFS. And for authenticated users it’s 60k/hr.
What could you possibly be doing besides scraping that would hit those limits?
I hit those many times when signed out just scrolling through the code. The front end must be sending off tonnes of background requests
This doesn’t include any requests from the website itself
You might behind a shared IP with NAT or CG-NAT that shares that limit with others, or might be fetching files from raw.githubusercontent.com as part of an update system that doesn’t have access to browser credentials, or Git cloning over https:// to avoid having to unlock your SSH key every time, or cloning a Git repo with submodules that separately issue requests. An hour is a long time. Imagine if you let uBlock Origin update filter lists, then you git clone something with a few modules, and so does your coworker and now you’re blocked for an entire hour.
60 requests per hour per IP could easily be hit from say, uBlock origin updating filter lists in a household with 5-10 devices.
Probably because of AI agents. This is why we can’t have nice things.
RIP yocto builds
I have a question: why do lemmy dev keep using microsoft github?
Yeah, shoulda use https://gitflic.ru/
Github is owned by Microsoft, so don’t worry, it’s going to get worse
This going to fuck over obtanium?
Codeberg has used way stricter rate limiting since pretty much forever. Nice thought, but Codeberg will not solve this problem, like at all.
What? I have never seen a rate limiting screen on codeberg. Ever. If I click too much on github I get rate limited. It happens so frequently, I use https://sourcegraph.com/search when I have to navigate a repository’s code.
Good thing I moved all my repos from git[lab|hub] to Codeberg recently.
