This is starting to feel like Twitter. For a variety of reasons, we had a centralized place where everyone of a particular set of persuasions could connect, and this had outsize benefits for the community as a whole. That place is becoming untenable, and with the loss of goodwill and stability, the community’s started to dissipate. But there isn’t one obvious transition candidate, so the diaspora is finding itself spread across a bunch of disparate places and services. The stated need they satisfy (source control with a web interface and some technical features, like pull requests with reviews) will be fulfilled. But those emergent features like a core community and default expectation of where you can find someone will fade. And that is a very real loss for all of us.
Good. That's a gain, not a loss. We're better off without social media psyop manipulations and subscription services bolted onto our code repositories. We're better off without a giant evilcorp getting wholesale access to a large majority of the world's source code.
Personally I never engaged in the community aspect of GitHub.
In my eyes it would be a ginormous gain to our colective mental health to not have the expectation to contribute to open source projects for free, or to have "green squares" every day on a public GitHub profile.
I just use GitHub for the basics: repo hosting, reviews and actions - what are the "social media psyop manipulations"? I've never really paid much attention to the other features.
All the metrics based around how many commits you have made, how many stars your repo has, etc. Why is that info gamified? Its a dopamine addiction channel.
But now the users who only use core functions (re: repo hosting, reviews and actions) suffer when the service is down due to overreach with feature implementation.
It's good you can isolate yourself from it. But there's plenty of cases.
Open source maintainers being harassed or being demanded work through Github's issues. Or being feed AI slop to review endlessly. Or recruiters filtering resumes via Github activity graphs or repository stars.
And obviously the side shady market of all of the above (e.g. buying/selling popularity).
Like the parent comment, I can't wait for social media to be completely obliterated from code repositories.
Even if the platform itself doesn't take off, the protocol and the data ownwership (you host your repo artifacts, issues, prs, etc on your ATproto server) mean that a competitor could come in easily if you were no longer satisfied with performance, reliability, or monetization.
It's a social network with github-like social features and git hosting, but you can still self host your git and have all the features like issues, PRs and so on just like GitHub.
And it's built on ATProto so you can also have your issues PRs and so on hosted on your infra, but everything interconnects nicely with your single account, just like using GitHub.
They even added a vouching system for users, to know who to trust based on who you already trusted.
I like Fossil, but there are good reasons to use git too.
There are many git hosting options, including some for self hosting. its a lot easier to switch to a different version control host than to a different version control system.
There are plenty of alternatives for git hosting. The last thing which needs to be migrated before we get an exodus off this vibe coded monstrosity is reputation (i.e. github stars) which are inherently sticky.
I don't think GitHub stars are what makes GitHub sticky. Having a single UI / account and being able to jump between work projects, private projects and random open source projects where I want to open a quick issue or submit a PR is much more sticky.
The reason why GitHub is sticky was exactly this. It’s easy to fork, make a change, and submit a PR to another project. It’s the same interface for your code vs another person’s. It’s a simple, consistent interface to submit an issue.
You can argue that the lack of diversity in infrastructure is bad, and I wouldn’t disagree. I think we should have more projects to explore better ways of working. But, the tradeoff is that when you have a centralized system, like GitHub, you also have less friction. The learning curve is small when everything uses the same tools.
Git doesn't even need to be "hosted" anywhere. That's the point of it being distributed. What needs to be hosted somewhere are all the barnacles (like issue tracking, code reviewing, CI, and so on) that have grown out around git, that people and businesses have become dependent on.
Which is why they cleverly moved the central story of a software change from the commit to the PR. Commits migrate automatically with the repo. But migrating PR descriptions and comments takes engineering effort.
Rhetorically, why can't folks add the feature of PRs to Git? Git is fundamentally just a graph manipulation engine and data structure. It would just be another object type, in a layer which sits above the normal commit graph. Obviously there is no "just make X" in software, but this feels reasonably achievable, and would have tremendous benefit for the cost.
some investors use stars (and star growth velocity) to determine which projects they invest or don't invest in. (not saying it's good or bad - just saying it's a real thing.)
> Luckily we already have a distributed version control system that can take over github
Surprised that this wasn't just a link to Git. The biggest benefit of Git over predecessors was that it enabled decentralization. Github re-centralized the common case for the sake of convenience.
Yes but on Earth every project needs a canonical main branch. Most also need a way to manage and review changes to that branch that all project members have direct access to.
It's on the project to decide which is canonical and export a read only accessible version. Change review only requires a communication medium (the patches are mostly plain text) and that can be done in a lot of way. As for merging those changes, it's an organizational issue.
The biggest benefits of Git over predecessors were branching, 3-way merge, and history rewrites (in particular, rebase and squash). As much as Git fans wanted decentralization to be a major selling point, decentralization has never been a major selling point.
Github and other providers before proved that git itself is just a small part of managing code, there's a lot more needed for a full workflow, and it's meaningful for it all to be tightly coupled.
> "Crypto had attracted grifters from the beginning, even though its most respected figures were more interested in changing the world. As Silicon Valley investors piled in after them, they wanted to fund companies that would become essential to the crypto economy, allowing them to make vast amounts of money. Entrepreneur after entrepreneur discovered that the best way to turn a profit in a decentralized economy was to figure out ways to centralize parts of it again." (Henry Farrell, Underground Empire)
Github pricing is aimed at US companies. Small EU companies that can only afford to pay their 5-6 developers 60K euro per year rather selfhost than pay GitHub.
Where I work we're hosting our own gitlab instance and the previous company I worked for hosted their own Bitbucket. For most developers I know it's either a self hosted git platform instance or Github enterprise.
It is, but a single point of failure is itself also an unstable equilibrium, given that a lot of things have to continuously go right for its continued existence in terms of governance of all kinds (technical, moderation policies, commercial/pricing etc.)
I think that, exactly how 95% of the people that left Twitter eventually returned, we will see a similar outflux of people from Github and the vast majority of them will quietly be back in a few months.
Do you have numbers to back that statement up? I don't necessarily doubt it, but I haven't seen anything to support it.
Note that raw user numbers won't really cut it; if 1 million people left Twitter, it's possible that 500 000 of those people joined and 500 000 new people joined; the user numbers would look like everyone returned, but really only 50% did in this hypothetical.
Twitter is a shadow of what it once was. The design of the site has shifted very very hard in favour of being an algorithmic content soup rather than a social network in the traditional sense. This has juiced the engagement numbers but there's a lot less meaning to them now. (I can say all this because I am a bitter holdout who still tries to use the site to have conversations with people and form personal connections. Tragically it still offers something that Bluesky and the Fediverse do not, even if that thing is slowly but surely being smothered out.)
I'd say about a third of software engineers whose blogs I stumble upon and end up liking aren't active on twitter. Their accounts typically have nothing posted for 3-4 years, but they're active on bluesky or mastodon.
Was this reported and quantified somewhere, or does 95% reflect your experience? My experience was the opposite -- everybody I cared to follow left and never went back -- but I'd be willing to believe I'm in the 5%.
That was a political decision, this is an operational one. A lot of companies have become dependent on a working github and effectively have to down tools when it is down.
Being down for a few hours 2-3 times a month is obviously garbage from a reliability standpoint, but I think they will keep it going at just the level necessary to barely hold on to these customers.
Good. Git is a decentralized version control solution. It's built for this. The centralization around GitHub was beneficial for the initial proliferation of Git, but has long since become a net negative. The "community" you speak of was never bound to GitHub. Better alternatives have emerged and for many it's time to move to greener pastures.
I think that especially when it comes to hosting, competition is good. Everything centered on github was the bad state previously. I remember people on HN claiming they wont hire someone without github account.
Are you potentially in the US? I think the experience is very different globally.
Ever since the Twitter -> X rebranding, every time I open Twitter links, I have a very high chance of hitting a "Something went wrong" type page, with a retry button that just does nothing. It's been like this for ages. I barely open Twitter links anymore as a result.
Fuck the community, nobody needs that, especially in a version source control system. Thinking otherwise is just dweebs' eternal longing for belonging, or someone has some bullshit to sell to the unsuspecting dirty masses.
I had a lot of goodwill for GitHub but I think today is the tipping point.
Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again.
> scaling problems due to the influx of LLM assisted coding
You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-).
I am more inclined to think that it is the second, but if the reason is the first, what is particular hard to scale ? It's not like everybody is vide-coding into a single repository, so I would expect that most of the load would naturally be scalable by increasing the resources. If for some reason that is not possible, I would expect a corporation as big as Github to keep the paying customers working by gracefully denying service to the free tier, not to collapse like an house of cards.
The quality and performance of GitHub enterprise has degraded horribly this year. Their problems are self-inflicted and influx is just an excuse.
Despite minimal new features, and drastically increasing hardware resources to keep up, self-hosted GitHub enterprise went from things being instant to most requests taking many seconds. And activity on issues in one repository tends to mysteriously re-open issues with the same number in other repositories.
I think they are vibe coding and facing the consequences.
They've had an insane influx. It seems that they should more aggressively limit free tier activity or isolate paid users, this has become unbearable...
How many weeks/months can they use this excuse? They are literally are at the forefront of this emerging industry and are capturing untold value. To let their product suffer and potentially lose market share because of it is extremely foolish
Scaling can take time but they've had at least a year to prepare. It's not like any demand increase they've seen was overnight...
At most normal companies you monitor your systems and address potential bottlenecks before they reach a tipping point. And generally you want enough headroom that a sudden 2-3x increase in demand wouldn't take out the service.
Either Github's technical leadership/talent is completely out of their depth (seems unlikely), or they're not hiring enough engineers to work on these capacity issues.
That’s the reason they gave, and I believe them. But I also think there has to be more to it. If your paying users are seeing degradation caused by (likely) free tier users (for years by the way), restrict the free users.
I'm really curious how other codeforges are faring. I only use codeberg and github - codeberg seems to have less outages, especially for a much smaller organisation, but they've recently removed support for LLM projects which could be linked.
Anyone here have data on whether the likes of gitlab have been impacted in the same was as github?
I love codeberg but it also have outages and is sometimes quite slow.
But I’m more lenient with codeberg because they are a non-profit and as you say, they are banning vibe-coded project because of the cost to infrastructure. In the meantime, GitHub is encouraging vibe-coding, multiplication of the source code, and genAI discussions.
If they did, they would’ve paused new feature development a long time ago and redirected that resource towards scaling the platform. And yet they’ve been releasing tonedeaf features more than ever.
Isn't all the LLMs and AI supposed to be "solving these problems" because all the engineers are now 10x ? Not being sarcastic but I mean, this is what we were told and sold...by the likes of...Microsoft.
Well, it has been 8 years since the acquisition and they have had access to Azure infrastructure to keep up with the scale and yet, it is still struggling due to free compute thanks to Actions.
They need to start getting rid of the free tiers for GitHub Actions and actually charge appropriately or increase prices for it.
They've done the typical sillicon valley sillyness of having 10 000 engineers over-engineer a product so that there's way too many features built on top of what should be a simple tool and are now paying the price. Literally every larger US company makes this mistake.
I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.
Because any price at all will immediately cause users to shift to another platform, and GitHub's value is that it is _the_ place to put your code on the internet.
I was there before 2018, their reliability was never that good.
I still remember that they launched GitHub Actions in 2018 and back then I was thinking: They can't even make the website reliable, surely this will only cause them more problems.
I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat.
Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn't someone recently calculate that GitHub is running at 95%? For comparison the Irish Rail service which is not reliable has 80% of it's trains run on time.
This seems absurd and really challenges a lot of ideas I had about big tech and cloud infrastructure. GitHub seems to have remained the dominant player relative to GitLab etc.
That canot happen if the market becomes monopolistic, with every bigtech out there buying every startup worth a penny. The erosion of anti trust in the US created this monstrosity.
A lot of it is inertia I think. I hear people saying that they "just use so-and-so for version control" but really we all have tons of CI/CD build and test pipelines, config, business processes etc in Github (rightly or wrongly).
I think AI is going to change this. Once you've built around github you feel locked in, since moving all your CI and other actions out of there, and your workflows out of there, is labor intensive. Now you could migrate from github to another service in a week, maybe less depending on how many workflows you need to move.
I wonder if you could replace the whole github issues feature with just a directory within each git repo named "Issues/". Within that, have another couple directories, "open/", and "closed/". Within "open", have a directory per issue, e.g. "such-and-such-does-not-work/". Within each issue directory, have "issue.txt", or "issue.md", which takes the place of the page for each issue on github. You could store other things in the issue directory which "issue.md" could reference, e.g. screen shots, logs, whatever. When an issue is closed, it gets git mv'ed to "closed/". You want to see a list of open issues sorted by time? Just "ls -ltr Issues/open". Want to see open issues related to the frakfrobber? "grep -i frakfrobber Issues/open/*". Add a comment to an issue? "vim Issues/open/my-issue/issue.md; git add !$; git commit -m 'Updated my-issue';" It would move the entire issues subsystem back under control of bare "git" and make it portable. Undoubtedly, I am not the first to think of such a thing.
To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.
If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
It is interesting, how much money is being lost during this outage?
My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced out by end-of-year due to incompetence. The new executive has taken to firing employees much more eagerly than their predecessor, like some sort of Jack Welch acolyte.
Their firing has put a lot of things into perspective for me. Mostly, fuck "at-will" employment and its negative effect on the American social contract.
But also this "angry ranting" online that the original poster was referencing. Not everyone has the privilege to calmly respond to things that directly impact their livelihood.
Any engineered system where an individual can accidentally cause a 7 figure outage is poorly designed. And engineering leadership that decider to terminate an individual due to such failure (as long as there was no malicious actions) is completely clueless.
GitHub Enterprise Cloud has been chugging along with no issues. I hope your critical infrastructure isn't dependent on a free tier / service. And that you have a business continuity process in place.
I'd be curious what the statistics might actually be for people who are directly affected because their business is suffering vs. people affected because their employer's business is suffering.
most people's employers arent firing people over a few hours of github outage.
not to mention that any business which could potentially lose enough money that they would need to let go of developers from a github outage should probably already have some business continuity plans in place.
"It's just money. It's made up. Pieces of paper with pictures on it so we don't have to kill each other just to get something to eat". Jeremy Irons in movie Margin Call
If you pay developers x money / day and one of their core tools is down for n hours during the day and they spend their money on HN instead that's pretty straight forward to calculate.
“lol just use a workaround” doesn’t work in an environment with hundreds or thousands of employees coupled with audit, security, and other legal requirements to ship software.
If you're actually bleeding money, you better believe you'll get permissions for a workaround, if you know what you're doing.
It all ties back to the OP, where the issue you've might not be as bad as you think. I have been in situations where we have dropped all procedures to push a hot fix because we were actively bleeding money, and in situations where you know there is an issue, and you let it be.
If its so critical why relying on it, and not having ie some mirror or some other way to handle any sort of outage like this. its not like Microsoft is your friend or good business partner, ever.
With every single of these enterprise 'cloud' offerings you are giving (almost) complete power over your business/project to somebody else who couldn't care less about your success or failure, you are simply irrelevant for them. I see it at work too, every time critical external systems go down whole bank stops still, just because few bucks were saved yearly on some on-prem servers.
Look at it this way, you are learning some important lesson today and finding great area of improvement for resiliency from now on.
Boomer opinion but trusting third parties to be critical infrastructure, especially with no SLA in sight, will always end in tears. "The cloud" is very convenient, but its providers will never care about your infrastructure or your customers as much as you will.
Agree. But also, it's affecting everyone equally, whether they have a free personal account or are part of an enterprise account with SLA. Understanding the practical value of an SLA is an interesting problem.
Boomer here as well, but I'd add that trusting your own org for critical infra usually also ends in tears. Most everything in IT involves failure, including in well designed systems designed by great engineers.
I worked for a few years in an exceedingly well capitalised place which ran everything in their own data centers, money no object, with a truck parked somewhere, ready to go, with a smaller version of our critical infra. We had a serious business-stopping outage once every 18 months or so, every time for fringe reasons one only learns about when trying to run a large data center. Its convenient to blame the cloud and pretend that self-hosting in private sector was so, so great with six nines.
3P-maintained infrastructure is what makes civilizations work efficiently. We're not all digging our own wells, generating our own electricity, and burning or burying our own garbage.
Agreed. So is the issue really then that people were inadequately prepared with backup plans and now they're suffering the consequences?
It's not all that different from, say, an AWS region having a service impact. People would rather complain about AWS than prepare and utilize a well-tested recovery plan to shift to a standby region. Oftentimes there's no fallback plan because the business already considered it and decided it was too costly relative to the benefit, but when the incident happens, they still can't help but complain. Humans being humans.
+1 (as a millenial) ... especially given that setting up a git server for non-OSS company code isn't too much of a challenge really. also, no need to self-denigrate this reasonable opinion in preemptive obedience.
GitHub is way more than just a git repo host. It manages code reviews, merge (pull) requests, and has an entire CI/CD workflow engine in it. Replicating all that is a challenge that most orgs are not up to.
How did this become a boomer opinion? It is proved truth thousand times a day. Not that you shouldn't use third parties - but in this industry you can shrink this exposure to the minimum, and have plan B for anything else.
However, I'm also of the boomer opinion that you should get what you pay for. "Ranting online" about a service (you pay for) being unavailable is a reasonable reaction. It's not like they have a call center you can dial into for support ...
People are already saying things like "We need a plan B in case we urgently need to deploy a fix to production, and GitHub Actions is unavailable again".
But in general, it's not feasible to do everything in house.
> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).
I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.
You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.
They _are_ the LLM coding agent vendor, and are _owned_ by MS, the majority owner of OpenAI. How can you NOT consider that 20x+ scaling in your capacity roadmap projections, where you are trying to get everyone to use these agents as part of your core OKRs?
Or, maybe your #1 IT priority was moving everything to Azure instead ;)
Github being under the CoreAI division probably also doesn't help the engineers prioritize addressing infrastructure issues and makes using LLM load as an excuse feel self-inflicted. Akin to feeling sorry when a pyromaniacs house burns down...
For the amount of money enterprises are paying to GitHub, there is a reasonable (and contractual) expectation of uptime. I don't think my boss would find it to fun if I bailed work to hit the gym just because GitHub was down.
- we are getting no benefit beyond getting excuses replies to our emails
- right or wrong, but we depend heavily on GitHub actions so this has a very real impact on us
My understanding is the us.githubstatus.com is only if you are using GitHub enterprise with a custom subdomain and data residency in the US because then *some* of the infrastructure is separate from GitHub.com.
This, like all of their dashboards, is complete bullshit. As a GitHub enterprise user i am absolutely affected by the outage today, and have been affected by all of the GitHub Cloud outages.
There is no separate enterprise platform unless you buy their self hosted server product.
Just take away free Actions and it would surely solve a lot of their scale issues.
It's absurd that I have dozens of repos, many with GH Actions that run CI, test and then package and push to prod/package managers, and haven't paid GH anything.
Right. It is not an excuse and they were notorious for outages before LLM era. Also, LLM can serve Github engineers as well, so we are all playing in the same field.
I can't check the commit history of some OSS projects without hitting a rate limit if I'm not signed in. My request rate is one request per (arbitrary time interval) at that point.
Actually, I do have an important fix for a deal we are trying to close that does need to go out. And I am paying GitHub to host this. Once or twice, I can see, but GitHub's SLA is getting worse than just hosting it myself, and that's the entire reason I pay GitHub.
This has to be rage bait, this is a critical piece of infrastructure for many people. Stuff going down can lead to deployments failing and as you mentioned in many cases emergency hotfix's.
The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.
>The idea that we should be fine with this unreliability is just amazing.
the comment doesnt say you should be "fine" with the unreliability.
they are saying people shouldnt get so emotionally worked up over it. which, while i wouldn't phrase it in the way the parent did, i agree with the direction of their point.
"Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code"
"Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside."
with assumptions like above.
If it was, "hey they f*ed up but there's no point having an overly emotional reaction" that would be fine. But he seems to be justifying this. Especially with Github's record up to now of unreliability I think it's completely fair to be annoyed.
GH shouldn't have promoted LLM usage the way they have if they didn't have the infrastructure to support it. Regardless of how you feel about the code quality (which you have zero evidence of), if GH has made commitments to supporting broad LLM usage, they need to back that up with the proper hardware and without whatever fragile SDLC processes they have.
> To everyone who is angry: calm down. Github’s servers are constantly on fire
Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?
This might be naïve but wouldn't the appropriate response be to reduce access/rate limit free/new accounts in order for service to be maintained for everyone else?
I understand the need to level-head the discussion and calming everyone.
However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.
The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.
(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)
Allowing and accepting the LLM load is a willful choice they made and are making at the expense of their users, including their paying enterprise users.
They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.
Well, they’re a critical piece of infrastructure with terrible stability. I’m in the process of migrating us off GitHub now.
I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.
Or, you know, it's perfectly reasonable and natural to feel angry when a service you paid for gets worse over time.
> Unless you have a emergency hotfix (you don’t)
Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.
Except I would like to get work done at work and spend time with my family when get home. Taking a walk at work is fine during the outage, but the kids will have to eat dinner without their dad. I'm not okay with that.
Wouldn't be surprised if we lost free private repos because of the slopocalypse. Though most don't seem to be shy about sharing their slop with the world so I'm not sure if that will actually increase stability.
I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket.
So now I am not angry because some critical piece of US-american infrastructure is down all the time.
I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.
is the infrastructure of the free tier shared with that of paid users? That might be the issue. Otherwise if usage of paid users scales, then it would be fine.
Ah, so that's what the agent meant by summarizing a documentation page as "Unicorn!" I'd thought it was a very colorful hallucination (it's just GitHub's (unicorn-themed) failure page).
Gathered details on embedding request syntax from
Unicorn! · GitHub
I was querying docs to figure out how to set up RAG to query docs locally. Alas.
GitHub, Charles Schwab, Microsoft Teams, Copilot, New York State, AWS, Progressive, Azure, Microsoft Store, Cloudflare, ADP, Google Sheets, E-Trade, Fidelity... Did Mythos break out of it's container?
We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?"
No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it.
Not the same company but similar discussions here:
- It was believed that GitHub would be more reliable than our self hosted Gitlab CI runners. This has proven to be very false.
- It was believed it would be easier to write tooling. I’m not on the dev tooling team, so can’t confirm if they found it true or not, but from the outside I see a lot more “Github can’t do that” than I did “Gitlab can’t do it”
Those are both hilarious to me. In my last FTE role we had the exact opposite beliefs - we wanted to leave GitLabs increases in pricing and move to annual sub, but the reliability of self-hosted runners and the flexibility (and clarity) of the CI system was fantastic.
The hilarious part is more the absurdity and cognitive disconnect experienced when reading about serious evaluations that went in a completely opposite direction to those directly participated in.
It's a ~year old product that started on GH. So this is the first time this team has had to think through "okay, so this isn't as reliable as it was at $oldjob."
Worth being upfront with, I'm an employee, but; Buildkite.
Frontier labs and global saas heavyweights run CI through us, plus a crop of open source tooling like Bazel, vLLM, Firecracker, Rails, Ray, Sorbet and Rust.
Shopify, Uber, Airbnb, Canva stresstested the platform ten years ago for scaling issues.
You need to take a little time to design upfront and take advantage of dynamically programmable primitives. It isn't the same ease/convenience as Actions. Bit more available though.
How would I be able to run my CI via Buildkite when I still host in GitHub and their API is down? We are currently already using self-hosted runners and they also don't work because either they aren't started by GitHub or can't download the code.
ultimately, folks need to stop writing massive YAML files to define their builds. CI should schedule jobs, your build system should define them. my workflow files look like this:
steps:
name: Do the thing
run: nix build .#
at any point, if my CI is down or i need to switch or i need to run builds locally, i can do so trivially.
If you're seriously looking at Gitea/Forgejo, I run Fjord, dedicated managed Forgejo, so you don't have to operate it yourself. We can help with migration and runners too. We also run a free public Forgejo instance if you just want to kick the tires first.
It's worked great for hosting my smaller personal projects so far, and I've already received a few helpful pull requests from the community.
Tangled.org really been working to ensure all their features work from data hosted on your personal data server (it's based on ATproto) which means should their performance, reliability, or monetization ever get you down it's easy to switch to a competitor since you're not locked in.
If your app is not too large you can do CI/CD perfectly fine from your laptop. Put it in a docker container if you want to ensure it is perfectly reproducible. You still need Github as the origin repo to pull from, but the "git part" of GitHub seems to be (almost) always up and running. (We use "capistrano" for this)
> If your app is not too large you can do CI/CD perfectly fine from your laptop.
Not if you want to make sure it compiles and works on Windows (which you don't have) or macOS (which you don't have and can't emulate). That's something github gives for free, and for me, it's the only reason to stay on github for small/tiny free software projects (at least as a secondary host).
I remember in 2024 passing over codeberg as the host for some of my personal projects because of the uptime. I don’t think Codeberg’s uptime has gotten any better but despite that it’s getting a lead over GitHub now.
blacksmith.sh just raised tens of millions on the core premise that self-hosting CI/CD is hard [0].
they've had multiple outages in the last week, so i guess they are proving themselves correct.
in practice, giving ssh/sudo to AI (codex) and telling it to setup ephemeral self-hosted runners on your own host VM (qemu), takes about 30 minutes and works extremely well. it bangs out all the shell scripts. moved my own business over to it. zero problems.
you might even find it is kind of neat too. you can customize your VM with software pre-installed. speeds up your builds quite a bit.
When you self host, what happens with your compute costs? CI can be super expensive depending upon your test pipeline. Do you end up spending more than you would with GitHub actions for example?
Just get some x86 and ARM runners off netcup, manage Woodpecker CI + Woodpecker Agent. Generally pretty hands-off once you get it set up. That plus a cheap VPS running Forgejo with postgres off-site backups, you are looking at a small fraction of the cost compared to per-seat GitHub pricing for orgs + Action Minutes.
He is likely referring to the CI runners. Gitea has their own version of Actions runner. It is very much usable, though currently inferior to GitHub Actions (not always a 1:1 replacement, even though they aim for it to be):
https://docs.gitea.com/usage/actions/
Cost as always depends on your usage, but it will likely be less than your current cost of GitHub Actions.
> We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available
Well it's 100% down for me, so does this mean 20% of all users are fully down?
It means it doesn't count towards their uptime at all, I guess, because all their uptime stats are very high for a service that is down more than anything else I use
I'm self-hosting Gitea and Woodpecker CI for least 6 years, alongside my own docker and pypi registries. Adding x86 and ARM runners, it costs me circa $30/month to run the whole thing, and I am reasonably sure that I could make the whole thing scale at a lower cost per user than what Github charges.
I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github. Given that I am fully employed, I'd only be able to do this on the weekends. Could I get a show of hands to get an idea of potential customers?
I work at a startup with ~20-30 devs. We're talking about a project like this. I'm not sure weekends would work for an actual company, though maybe it depends. I think there would be an appetite for this. From my POV - the main areas of interest are how you can roll this out to a team gracefully without breaking everyone's workflows for multiple days. Good luck
The most natural way to roll this out would be to have a "blue/green" system, where both systems are running and the weekends would be the time to take the steps towards the migration. First set up the servers, integrate Auth/SSO (depending on the current company set up), etc. That could be done probably in a single weekend.
Once this is done, the team would have two possible git remotes to work with. Given that gitea also allows mirroring with Github, the team would be able to even switch to Gitea as the "main" git remote and for PR reviews even if the CI part is still on Github.
The harder part would be migrating the CI jobs. But this also can be done in parallel and no one needs to cut out their Github actions immediately. Start with the simplest of the stages, e.g, the automated test suite, and once you get the whole pipeline from Woodpecker on an staging environment, you'd could take another weekend to test the whole workflow. Once the first project is successfully migrated, it becomes a matter of repeating the process.
Woodpecker is still a bit finicky and debugging it can be a bit annoying, but given that it's all based on docker it is a lot easier to debug a pipeline locally.
Funny enough, I received a marketing email from GitHub during the outage with the following subject: "What if every pull request you started actually shipped"
Huh, they usually do? I'm not a full-time developer though so I'm surely biased towards doing the obvious quick fixes or already-discussed things, but discussing how to solve a problem before throwing something at the wall seems like the normal thing to do for me
Or is their point about abandoned projects where the dev never clicks release after merging, or even never merges?
Yeah I saw the humor of that :) Just curious about the thing they're trying to even sell, it doesn't sound like something they can offer even if they had a good SLA
The PSA is that you should always have a mirror—even if just a bare repo folks can pull from during downtime.
…& this is ignoring all the moral reason for avoiding US-based, publicly-traded, proprietary software for host largely free software. They also deny access to users due to US sanctions, privacy-conscious users won’t want to create accounts/agree to ToS, & rubbing shoulders with other corpos your project can be taken down if it upsets the corpos don’t like it (see youtube-dl, Switch emulators, etc.).
Sure, but it's not just the reoo. It's PR's, code review and you've got a bunch of actions then it's difficult (not impossible) to deal with those being down.
A self hosted version of GitHub that is a precise mirror of an actual GitHub repository. That is, PRs from GitHub appear on the selfhosted instance and my reviews from the self hosted instance appear on GitHub. That way I can work on my private instance when GitHub is down (or slow) and I can be assured my work still appears to everyone online.
Perhaps relevant tho. I would imagine if they could simply throw more hardware at it they would. And I wouldn't imagine inability to throw hardware at it would be due to software scaling issues. So perhaps it is simply lack of available hardware?
(Umpteenth time:) GitHub is not about just publishing source code; with git, it's trivial out of the box.
GitHub's value is in the PR process, in CI/CD, releases, the issue tracker, and an identity system propping all of that. This is the thing to replace, and it's quite a bit harder.
Seriously. It seems like there’s an issue every other week. It would historically be normal to provide post mortems, and maybe I’ve missed them but seems like there are almost weekly issues and no follow up into the core reasons why.
Might not help your deploy process if webhooks are degrading, but at least it can keep your remote in sync with local changes, if you need to deploy from local.
Copy/paste of a comment I posted a couple months ago:
I am not trying to make any excuses for GitHub, however I recall one tweet that made my eyes bug-out:
> Yup, platform activity is surging. There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
> GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.
That's fine, but rate limiting is a solved problem at this point. They need to rate-limit free accounts and be realistic about their limits instead of letting the system fail.
Free (and non-free) accounts are already rate limited. The likely problem is that individual accounts/repos rarely hit those limits, instead the explosive demand is due to a massive growth in the number of small, individual projects being creatd.
And most of that growth is just garbage. I'm not even anti-AI, but just look at the quality of projects on GitHub these days. The site is full of spam, it's not a place for professionals anymore.
Their infrastructure is buckling under increased demand from AI generated code.
Imagine the software development lifecycle is a river blocked by several dams- necessary processes that also slow the water’s volume and speed. Writing the code was a very big dam upstream, and successive dams didn’t need to handle a lot of water. With that dam busted wide open, the torrent of code is inundating downstream processes. GitHub happens to host many of them.
Earlier this year the GitHub COO shared a post on Twitter with explosive numbers they were facing. I cannot seem to find it at the moment. (Edit: shared in other replies to this thread.)
I mean, that’s what they say but it doesn’t pass the sniff test. Which other major service struggles so much with basic scaling? They’re not serving streaming video, they’re not a real-time system. Any problems scaling should have been fixed years ago. Especially with all of Microsoft’s resources behind them.
I would argue they are. You can't cache a git repository and send people what it looked like a minute ago, because someone might have pushed 2 seconds ago.
I think the most charitable theory is that they're seeing rapidly growing usage (more users churning out higher volumes of code, via LLMs), and are struggling to scale to meet demand.
Alternative theories include "azure sucks" and "they're vibecoding everything now".
Github's, COO Kyle Daigle, claimed that the number of commits has increased 14x since last year. So shear scale has something to do with it. I don't know of too much software already operating a planetary scale that could easily absorb a 14x bump in a year.
Microsoft simply can not make software anymore. I worked at S&P Global and when you work at a global-mega corp, you quickly see why they lose the ability to create anything and just acquire companies to grow.
What is the best Github alternative that has something like Github Actions? What is the easiest to switch to? I do not want to switch but it is having too much of an impact on me.
It is time to give up on Github. This is just too much.
Forgejo is probably worth a look if Actions compatibility and an easy migration are priorities. I run Fjord, a dedicated managed Forgejo, so you don't have to operate it yourself. We can help with the migration and runners too.
We also have a free public Forgejo instance if you want to try it first.
I’m hosting my own https://tangled.org knot and spindle, and also hosting and building my own Frontend browser for https://forgejo.org looking forward to open sourcing it in the future for others to enjoy.
For people who want a similar experience today, I’d recommend trying Codeberg (An official instance of Forgejo) though. However, keep in mind they voted in July to ban mostly AI generated repositories. You can read about that here: https://blog.codeberg.org/protecting-our-floss-commons-from-...
Yeah, Codeberg banning AI generated code is weird. AI generated code is the future. It is like during the days of assembly banning high level languages.
GitLab is probably the most reasonable alternative. I migrated and it took me some weeks to adapt to the new CI, but for my use case it's basically equivalent in features.
I started with the cloud version, then moved to self-hosted because it wasn't a big effort for our team. We also reduced our monthly invoice by about 50% after the move.
I am building an MIT-licensed compatible alternative to GitHub Actions at github.com/automata-ci/automata. Very early in development but it already runs some of our internal workflows.
I really wish I could move to a different provider, but GitHub is just so damn practical. Downtimes such as this should at least be compensated, providing an actual financial incentive to improve their services, instead of slapping AI on yet another product.
I can imagine the massive growth they're in right now, old assumptions are breaking down, many things need to be rethought.
That's why personally i don't hate them with all of this saga. But they need to work faster in terms of product decisions and technology.
All of my code repos are on gitea (personal and professional). The instance is not exposed to the internet (wireguard access). It's fantastic honestly, simple and just works.
How did Github become so famous for outages? It is almost like a routine. I haven't used other code repo hosting / CI/CD pipelines platform for last decade, so maybe it's just the nature of this business, I don't know.
it seems to be coinciding with teams and other microsoft services having issues would be some indication, but some other non microsoft products are also having a bit of a spike too.
NB: I am not OP but just providing some context as to why they might think this.
GitHub is down days of that week that end with day these days. If I were a paying customer I'd be cancelling soon. GitHub is sort of foundational at this point and when its down nearly every day of the week it cannot be foundational.
So glad I moved my OSS org mostly over to a self-hosted Forgejo + Woodpecker CI. I get that we are operating on a completely different scale but I swear the GitHub outages are far too regular than they ought to be.
What is really interesting is that no insider engineer has leaked anything. What we got was essentially "LLM" hitting GitHub really hard. But there is no retro or maybe i missed it.
Never quite understood why large enterprises use github as core infrastructure.
Sure use it to collab externally, use it if you can't justify running your own infrastructure, but why are large companies choosing to place code externally - which is one secret leak away from compromise?
Ideally something like IPNS to have a mutable ref based on (repo, pubkey, branch) to resolve a commit hash, a DHT for who has what commit hashes, and then just p2p transfers. You could still have a github or gitlab for issues and CI, but code (and maybe issues) could all be p2p like git is already designed for. Your forge could also act as a peer for code cloning, of course.
The complexity is in locating what is already a distributed database item (git is fundamentally already designed that way), just using a protocol meant for that. Basically just support `git clone` with e.g. IPFS URIs.
I think the clear reason is even with their unreliability, the cost of migrating off of GitHub for _most_ places is not worth it, and so companies don't / won't (yet)
This bugs me so much. Management decides what is reasonable cost of service and the employees have to deal with all bullshit coming from that product. And you can't really escape it. Maybe the real moat was enterprise deals we made along the way.
The GitHub failure unicorn image always brings me back to the good old days of trying out new Android ROMs. That unicorn used to be the logo of the Android Open Kang Project, which used to be my favorite ROM.
You know, I think Fossil with its integrated issue management right in the repo is actually a perfect match for the LLM era. Maybe today is the day I migrate to using a Fossil forge with GitHub becoming increasing untenable.
If someone needs lightweight self hosted git server with ci pipelines via general programming languages - consider DSCI - http://deadsimpleci.sparrowhub.io
You would think for a company that has close to a monopoly of developer mind share wrt git that they'd maybe put in the bair minimum to get several 9s.
If you are required to use a Microsoft tool is Azure DevOps now a better choice? Our Azure hosted instance rarely (if ever) has had an outage that effects our ability to work.
As someone who maintained a product that was, until recently, on Azure DevOps, stay away. Like, really really far away. There's nothing positive I can say about it.
That's not necessarily a bad thing for stability, but long-term I would not be surprised if it gets EOL-ed, or radically shrunk to just a basic Git mirroring service.
Discussion about it going ELO has been around since 2020-ish. But that is a worry i agree.
I get the feeling there is a long tail of old-school corps (migrated from the TFS days) that will keep the lights on. And for some orgs you just need "good enough" CI/CD pipelines that are reliable.
Maybe I'm a weirdo, but i find the kanban/sprint board pretty usable as well.
Githubs a great advert as to why you shouldn't use Azure. What an utter mess its become since they took over. Their argument that its scaling issues due to increased traffic just makes them and Azure look incompetent.
As I said 4 days ago GitHub is in chronic operational decline which is killing trust in hosting repositories [0]
You can use as many AI agents as you want (it won’t solve the problem), but with that many outages ever since getting bought by Microsoft, it is more than enough time to give them to change. They don’t care and there is no CEO of GitHub to help either.
A self hosted instance would have much better uptime than GitHub and GitHub Actions at this point as I predicted 6 years ago. [1]
It ain't the cause, but is definitely an enabler, because foreign talent on work visas effectively depends on their job much more so than their domestic counterparts and are thus very unlikely to push back against unreasonable demands or workload.
No but it demonstrates a disinterest in building software products and a bigger interest in playing business cost economics and securing their moat. Part of their strategy to force as much money into AI-hype as possible.
I have worked with some "american born engineers" that were some of the least competent people I've ever seen, while getting paid 2x or 3x of some "foreign contractors" on the same team
Yeah, that was unnecessary. Incompetent are born and work everywhere. In this case, Microsoft's problem with GitHub is trying to shove the square peg that is GitHub in to the round hole that is Azure.
Unfortunately, my company uses Outlook and Teams. Both are the worst piece of software I've ever used. They're just trash.
The UX is so bad. Things don't work as expected. Even basic things like composing an email, copy and pasting, formatting text are always a chore.
Try setting up a Teams integration vs a Slack integration. Gmail is absolutely light years ahead in UX. I really miss working at a company that uses Slack + Google Suites.
I'm not a huge fan of Team, but I do have to give them credit for making the experience better over the years- a few years ago, search didn't even work!
I'm not sure I understand the significance of them having almost no American born engineers. Do we think it would be meaningfully different if they were? I tend to believe that Americans can be just as incompetent as the next guy!
In my eyes it would be a ginormous gain to our colective mental health to not have the expectation to contribute to open source projects for free, or to have "green squares" every day on a public GitHub profile.
Open source maintainers being harassed or being demanded work through Github's issues. Or being feed AI slop to review endlessly. Or recruiters filtering resumes via Github activity graphs or repository stars.
And obviously the side shady market of all of the above (e.g. buying/selling popularity).
Like the parent comment, I can't wait for social media to be completely obliterated from code repositories.
Even if the platform itself doesn't take off, the protocol and the data ownwership (you host your repo artifacts, issues, prs, etc on your ATproto server) mean that a competitor could come in easily if you were no longer satisfied with performance, reliability, or monetization.
It's a social network with github-like social features and git hosting, but you can still self host your git and have all the features like issues, PRs and so on just like GitHub.
And it's built on ATProto so you can also have your issues PRs and so on hosted on your infra, but everything interconnects nicely with your single account, just like using GitHub.
They even added a vouching system for users, to know who to trust based on who you already trusted.
This is the way. Stop centralizing everything.
I suppose people can use git too if they ever figure out how to send emails from a dedicated client.
There are many git hosting options, including some for self hosting. its a lot easier to switch to a different version control host than to a different version control system.
You can argue that the lack of diversity in infrastructure is bad, and I wouldn’t disagree. I think we should have more projects to explore better ways of working. But, the tradeoff is that when you have a centralized system, like GitHub, you also have less friction. The learning curve is small when everything uses the same tools.
If I build something that satisfies me and say 100 users (who all star me) does that really say anything. Are those 100 stars even "real" any more?
All the old problems of the web are new again, and still remain unsolved.
some investors use stars (and star growth velocity) to determine which projects they invest or don't invest in. (not saying it's good or bad - just saying it's a real thing.)
Surprised that this wasn't just a link to Git. The biggest benefit of Git over predecessors was that it enabled decentralization. Github re-centralized the common case for the sake of convenience.
https://forgejo.org/2023-01-10-answering-forgejo-federation-...
Seems like it's far from production-ready and there hasn't been much activity this year?
EVERY other client I've ever worked with used their own gitforge or at least bitbucket...
Hell, even I do it.
Note that raw user numbers won't really cut it; if 1 million people left Twitter, it's possible that 500 000 of those people joined and 500 000 new people joined; the user numbers would look like everyone returned, but really only 50% did in this hypothetical.
On Twitter, it's mostly the networking effect, you want to be where peoples are, and unless a large chunk move at once, you are stuck there.
On Github, it's the technical cost of migrating, the git part is trivial, but tickets, PR history and CI/Integration are another can of worms.
Projects which moved away, moved away for good, at most only leaving a synchronized mirror.
Github is actually down and I can't push code right now, it's blocking my business which is unacceptable. It's a broken tool.
This splintering should happen more often.
Ever since the Twitter -> X rebranding, every time I open Twitter links, I have a very high chance of hitting a "Something went wrong" type page, with a retry button that just does nothing. It's been like this for ages. I barely open Twitter links anymore as a result.
Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again.
The hope is dead.
But maybe I’m wrong?
You mean people pushing LLM assisted coding, or github's own coders pushing LLM assisted code to production ? :-).
I am more inclined to think that it is the second, but if the reason is the first, what is particular hard to scale ? It's not like everybody is vide-coding into a single repository, so I would expect that most of the load would naturally be scalable by increasing the resources. If for some reason that is not possible, I would expect a corporation as big as Github to keep the paying customers working by gracefully denying service to the free tier, not to collapse like an house of cards.
Despite minimal new features, and drastically increasing hardware resources to keep up, self-hosted GitHub enterprise went from things being instant to most requests taking many seconds. And activity on issues in one repository tends to mysteriously re-open issues with the same number in other repositories.
I think they are vibe coding and facing the consequences.
At most normal companies you monitor your systems and address potential bottlenecks before they reach a tipping point. And generally you want enough headroom that a sudden 2-3x increase in demand wouldn't take out the service.
Either Github's technical leadership/talent is completely out of their depth (seems unlikely), or they're not hiring enough engineers to work on these capacity issues.
Anyone here have data on whether the likes of gitlab have been impacted in the same was as github?
But I’m more lenient with codeberg because they are a non-profit and as you say, they are banning vibe-coded project because of the cost to infrastructure. In the meantime, GitHub is encouraging vibe-coding, multiplication of the source code, and genAI discussions.
The original source was Kyle Daigle, GH COO.
- https://x.com/kdaigle/status/2040164759836778878
- https://xcancel.com/kdaigle/status/2040164759836778878
They need to start getting rid of the free tiers for GitHub Actions and actually charge appropriately or increase prices for it.
I guess they're quite happy with that TBF, it doesn't seem like they're that interested in being a git forge anyway.
https://imgur.com/b0Xj3Z6
Seems presumptuous to assume a root case while the incident is still ongoing, no?
If they make it harder to push slop on github, people might treat it as an incentive to generate less slop. Less tokens sold. Bad. Very bad.
The github paying customers are not going to move away.
Plus, it's probably genuinely hard to scale. And they can't throw compute at the problem - all compute is already going to the LLMs.
I still remember that they launched GitHub Actions in 2018 and back then I was thinking: They can't even make the website reliable, surely this will only cause them more problems.
https://news.ycombinator.com/item?id=48086786
> Error: Response status code does not indicate success: 429 (Too Many Requests).
> Warning: Back off 19.714 seconds before retry.
> Warning: Failed to download action 'https://codeload.github.com/'.
> Error: Response status code does not indicate success: 502 (Bad Gateway).
> Warning: Back off 22.228 seconds before retry.
> Error: Response status code does not indicate success: 429 (Too Many Requests).
> Error: Failed to download archive 'https://codeload.github.com/' after 3 attempts.
Nice - even their own action runners are getting rate limited atm
Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn't someone recently calculate that GitHub is running at 95%? For comparison the Irish Rail service which is not reliable has 80% of it's trains run on time.
This seems absurd and really challenges a lot of ideas I had about big tech and cloud infrastructure. GitHub seems to have remained the dominant player relative to GitLab etc.
We don't care. We don't have to. We're the phone company.
https://youtu.be/iBDUTaKWVdU
Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.
If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced out by end-of-year due to incompetence. The new executive has taken to firing employees much more eagerly than their predecessor, like some sort of Jack Welch acolyte.
Their firing has put a lot of things into perspective for me. Mostly, fuck "at-will" employment and its negative effect on the American social contract.
But also this "angry ranting" online that the original poster was referencing. Not everyone has the privilege to calmly respond to things that directly impact their livelihood.
not to mention that any business which could potentially lose enough money that they would need to let go of developers from a github outage should probably already have some business continuity plans in place.
It all ties back to the OP, where the issue you've might not be as bad as you think. I have been in situations where we have dropped all procedures to push a hot fix because we were actively bleeding money, and in situations where you know there is an issue, and you let it be.
With every single of these enterprise 'cloud' offerings you are giving (almost) complete power over your business/project to somebody else who couldn't care less about your success or failure, you are simply irrelevant for them. I see it at work too, every time critical external systems go down whole bank stops still, just because few bucks were saved yearly on some on-prem servers.
Look at it this way, you are learning some important lesson today and finding great area of improvement for resiliency from now on.
I worked for a few years in an exceedingly well capitalised place which ran everything in their own data centers, money no object, with a truck parked somewhere, ready to go, with a smaller version of our critical infra. We had a serious business-stopping outage once every 18 months or so, every time for fringe reasons one only learns about when trying to run a large data center. Its convenient to blame the cloud and pretend that self-hosting in private sector was so, so great with six nines.
It's not all that different from, say, an AWS region having a service impact. People would rather complain about AWS than prepare and utilize a well-tested recovery plan to shift to a standby region. Oftentimes there's no fallback plan because the business already considered it and decided it was too costly relative to the benefit, but when the incident happens, they still can't help but complain. Humans being humans.
However, I'm also of the boomer opinion that you should get what you pay for. "Ranting online" about a service (you pay for) being unavailable is a reasonable reaction. It's not like they have a call center you can dial into for support ...
But in general, it's not feasible to do everything in house.
And I don't think that GitHub is devoid of SLA: https://github.com/customer-terms/github-online-services-sla
The issue is that they're not achieving two nines uptime in practice.
I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory).
I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the downtime right now, as I'll make a cup of coffee and do something else. I will say though, I did have a hotfix a week or so ago during the Actions outage, which really was a pain.
You're right that getting angry and ranting isn't the right reaction here, but I do no give them the LLM load excuse. I don't give them an out for having awful uptime during work hours for a product we pay quite a bit for as an org.
[0] https://damrnelson.github.io/github-historical-uptime/
Or, maybe your #1 IT priority was moving everything to Azure instead ;)
Largest shareholder (27%), not majority.
This is the outcome of violating single responsibility principle in business.
- we are getting no benefit beyond getting excuses replies to our emails - right or wrong, but we depend heavily on GitHub actions so this has a very real impact on us
There is no separate enterprise platform unless you buy their self hosted server product.
Which means they have a pricing problem. Rate limit non-paying users.
It's absurd that I have dozens of repos, many with GH Actions that run CI, test and then package and push to prod/package managers, and haven't paid GH anything.
The idea that we should be fine with this unreliability is just amazing. It's not a mental health issue to have problems when important infra fails.
the comment doesnt say you should be "fine" with the unreliability.
they are saying people shouldnt get so emotionally worked up over it. which, while i wouldn't phrase it in the way the parent did, i agree with the direction of their point.
"Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code"
"Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside." with assumptions like above.
If it was, "hey they f*ed up but there's no point having an overly emotional reaction" that would be fine. But he seems to be justifying this. Especially with Github's record up to now of unreliability I think it's completely fair to be annoyed.
Which they encouraged by pushing Copilot down everyone's throat
No qualifiers necessary. But try arguing with a lawnmower...
Is this rage bait? Isn't Google, Amazon, and all the other services in the world similarly impacted by LLM's? Isn't Claude, OpenAI, etc.? Why can they handle the load but not Github?
Github and their parent company are active participants in pushing LLM-driven coding.
However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.
The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.
(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)
I find it hard to believe that github APIs don't have rate limits to handle traffic spikes.
They could easily tighten things up in that regard, and make a choice that is right for their main users at the expense of The MS corpo mandate/mission. It is a choice to do otherwise.
I’m not a Meta fan but it’s interesting that they manage to keep their systems up with an order of magnitude more traffic. GitHub’s uptime is inexcusable.
> Unless you have a emergency hotfix (you don’t)
Oh, we do. Given the sheer number of users, it's almost guaranteed someone is on fire ever time GitHub is down. Statistics is a very charming branch of reality.
hey bud your bias is showing
This should not be an excuse since they probably can use AI to fix it /partially sarcasm
My mind immediately went to this classic xkcd: https://xkcd.com/303/
I have little time in my day to spend with my daughter. But I also have responsibilities. And one client has decided to bet on github. It's the only client I've ever worked with who has their code on github. EVERY other client has hosted their own gitforge or used bitbucket.
So now I am not angry because some critical piece of US-american infrastructure is down all the time.
I am angry because instead of spending time with my daughter I have to work on this later, because there are due dates and "Well, fucking GitHub was down" ain't gonna cut it.
github enterprise is fully operational at the moment
https://eu.githubstatus.com/posts/dashboard
We cannot pull any actions images (or how it is called) for example.
So with the "elapsed time" out of the way (now over an hour, incidentally), how about "how long will it be down?"
The question is which features do we want of Github but functioning in a distributed way?
Identity? Stars? Issues?
https://downdetector.com/
No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it.
What's everyone else using? Self hosted Gitlab? Gitea?
Did you have the conversation "do we need to move from $whatever CI/CD to GitHub Actions?" beforehand? What were the arguments in favour?
- It was believed that GitHub would be more reliable than our self hosted Gitlab CI runners. This has proven to be very false.
- It was believed it would be easier to write tooling. I’m not on the dev tooling team, so can’t confirm if they found it true or not, but from the outside I see a lot more “Github can’t do that” than I did “Gitlab can’t do it”
Why is it hilarious to believe that the industry leader might be better at something than doing it yourself?
Frontier labs and global saas heavyweights run CI through us, plus a crop of open source tooling like Bazel, vLLM, Firecracker, Rails, Ray, Sorbet and Rust.
Shopify, Uber, Airbnb, Canva stresstested the platform ten years ago for scaling issues.
You need to take a little time to design upfront and take advantage of dynamically programmable primitives. It isn't the same ease/convenience as Actions. Bit more available though.
tangled.org!
Tangled.org really been working to ensure all their features work from data hosted on your personal data server (it's based on ATproto) which means should their performance, reliability, or monetization ever get you down it's easy to switch to a competitor since you're not locked in.
Not if you want to make sure it compiles and works on Windows (which you don't have) or macOS (which you don't have and can't emulate). That's something github gives for free, and for me, it's the only reason to stay on github for small/tiny free software projects (at least as a secondary host).
I mean, that plus better uptime.
See: https://blog.codeberg.org/protecting-our-floss-commons-from-...
they've had multiple outages in the last week, so i guess they are proving themselves correct.
in practice, giving ssh/sudo to AI (codex) and telling it to setup ephemeral self-hosted runners on your own host VM (qemu), takes about 30 minutes and works extremely well. it bangs out all the shell scripts. moved my own business over to it. zero problems.
you might even find it is kind of neat too. you can customize your VM with software pre-installed. speeds up your builds quite a bit.
[0] https://www.blacksmith.sh/blog/blacksmith-seed-announcement
https://status.blacksmith.sh/cmsrg61yg09vd0lo9s1eagibm
Cost as always depends on your usage, but it will likely be less than your current cost of GitHub Actions.
I think it's fine if we begin to move back towards distributed software supply chain infrastructure.
Well it's 100% down for me, so does this mean 20% of all users are fully down?
> Posted 1 minute ago. Aug 17, 2026 - 13:58 UTC
It also took them 20 minutes to realize that peoples requests to /pull/xx are not working :D
I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github. Given that I am fully employed, I'd only be able to do this on the weekends. Could I get a show of hands to get an idea of potential customers?
Once this is done, the team would have two possible git remotes to work with. Given that gitea also allows mirroring with Github, the team would be able to even switch to Gitea as the "main" git remote and for PR reviews even if the CI part is still on Github.
The harder part would be migrating the CI jobs. But this also can be done in parallel and no one needs to cut out their Github actions immediately. Start with the simplest of the stages, e.g, the automated test suite, and once you get the whole pipeline from Woodpecker on an staging environment, you'd could take another weekend to test the whole workflow. Once the first project is successfully migrated, it becomes a matter of repeating the process.
Woodpecker is still a bit finicky and debugging it can be a bit annoying, but given that it's all based on docker it is a lot easier to debug a pipeline locally.
Or is their point about abandoned projects where the dev never clicks release after merging, or even never merges?
…& this is ignoring all the moral reason for avoiding US-based, publicly-traded, proprietary software for host largely free software. They also deny access to users due to US sanctions, privacy-conscious users won’t want to create accounts/agree to ToS, & rubbing shoulders with other corpos your project can be taken down if it upsets the corpos don’t like it (see youtube-dl, Switch emulators, etc.).
You have a free bank holiday, go look at green trees, listen to birds, find somebody who's wrong on HN etc.
The only poor souls here are some github devops team (team of copilots?) and those using alternative services.
A self hosted version of GitHub that is a precise mirror of an actual GitHub repository. That is, PRs from GitHub appear on the selfhosted instance and my reviews from the self hosted instance appear on GitHub. That way I can work on my private instance when GitHub is down (or slow) and I can be assured my work still appears to everyone online.
Does this exist?
"Microsoft’s data-center shortages to persist longer than expected - Bloomberg"
https://ca.finance.yahoo.com/news/microsoft-data-center-shor...
Perhaps relevant tho. I would imagine if they could simply throw more hardware at it they would. And I wouldn't imagine inability to throw hardware at it would be due to software scaling issues. So perhaps it is simply lack of available hardware?
GitHub's value is in the PR process, in CI/CD, releases, the issue tracker, and an identity system propping all of that. This is the thing to replace, and it's quite a bit harder.
Seriously. It seems like there’s an issue every other week. It would historically be normal to provide post mortems, and maybe I’ve missed them but seems like there are almost weekly issues and no follow up into the core reasons why.
I slowly but surely lowered my expectations.
I am not trying to make any excuses for GitHub, however I recall one tweet that made my eyes bug-out:
> Yup, platform activity is surging. There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
> GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.
https://x.com/kdaigle/status/2040164759836778878
https://xcancel.com/kdaigle/status/2040164759836778878
---
edit: see a counterpoint I posted down-thread.
Manager: Why is the site down!?
Engineer: Because it can’t handle this level of traffic. We can rate limit to the level we can handle to keep the service running.
Manager: Okay, let’s do that!
Engineer: Okay. Rate limiting at N req/min.
Manager: I’m getting complaints we’re denying service to customers!? We can’t rate limit our users if they want to use it. Increase the rate limits.
I’ve had this conversation more than once in my career.
Imagine the software development lifecycle is a river blocked by several dams- necessary processes that also slow the water’s volume and speed. Writing the code was a very big dam upstream, and successive dams didn’t need to handle a lot of water. With that dam busted wide open, the torrent of code is inundating downstream processes. GitHub happens to host many of them.
Earlier this year the GitHub COO shared a post on Twitter with explosive numbers they were facing. I cannot seem to find it at the moment. (Edit: shared in other replies to this thread.)
I would argue they are. You can't cache a git repository and send people what it looked like a minute ago, because someone might have pushed 2 seconds ago.
Alternative theories include "azure sucks" and "they're vibecoding everything now".
Many people suspect it's the year-long migrating to Azure and all their products instead of the things they have been using.
Whatever the actual increase in productivity may be, the commits, clones, pushes, action runs, etc. are skyrocketing.
It is time to give up on Github. This is just too much.
We also have a free public Forgejo instance if you want to try it first.
For people who want a similar experience today, I’d recommend trying Codeberg (An official instance of Forgejo) though. However, keep in mind they voted in July to ban mostly AI generated repositories. You can read about that here: https://blog.codeberg.org/protecting-our-floss-commons-from-...
I started with the cloud version, then moved to self-hosted because it wasn't a big effort for our team. We also reduced our monthly invoice by about 50% after the move.
* Forgejo
Those are my favorites. Both you can self-host for your company, including CI/CD.
Interestingly the API still works for creating an issue (but webhooks weren't fired).
Did Anthropic or OpenAI fail to sandbox their models again?
https://downdetector.com/
NB: I am not OP but just providing some context as to why they might think this.
https://docs.gitlab.com/install/docker/installation/#install...
Sure use it to collab externally, use it if you can't justify running your own infrastructure, but why are large companies choosing to place code externally - which is one secret leak away from compromise?
"Yeah we're unable to offer services, but so is [X BIG CORP] and [Y BIG CORP]. It's an industry outage, nothing we can do!"
Or let me ask a more useful question: if we were to move, then to where? Ideally somewhere new (not Gitlab)
Otherwise there's no "we", Git is decentralized. Everyone can use whatever they want.
I suspect even poor uptime for many services is a cosmetic problem for most users.
I wonder how it made it onto GitHub.
Anyone else using Fossil?
2) Issues showed Not found.
Anyway, GitHub Actions for building new release has progress. Good enough.
[1] https://github.com/ljtn/epiq
Thankfully this is encouraging me to stop working overtime for now and enjoy the rest of my night
https://azure.microsoft.com/en-us/updates?filters=%5B%22Azur...
That's not necessarily a bad thing for stability, but long-term I would not be surprised if it gets EOL-ed, or radically shrunk to just a basic Git mirroring service.
Discussion about it going ELO has been around since 2020-ish. But that is a worry i agree.
I get the feeling there is a long tail of old-school corps (migrated from the TFS days) that will keep the lights on. And for some orgs you just need "good enough" CI/CD pipelines that are reliable.
Maybe I'm a weirdo, but i find the kanban/sprint board pretty usable as well.
Seems like a simple solution.
You can use as many AI agents as you want (it won’t solve the problem), but with that many outages ever since getting bought by Microsoft, it is more than enough time to give them to change. They don’t care and there is no CEO of GitHub to help either.
A self hosted instance would have much better uptime than GitHub and GitHub Actions at this point as I predicted 6 years ago. [1]
[0] https://news.ycombinator.com/item?id=49276571
[1] https://news.ycombinator.com/item?id=22867803
Yes, github now has issues. Github is always having issues.
It's just a constant pain in the ass.
Wherever I have anything to say I prefer ANY other gitforge over this piece of shit.
The UX is so bad. Things don't work as expected. Even basic things like composing an email, copy and pasting, formatting text are always a chore.
Try setting up a Teams integration vs a Slack integration. Gmail is absolutely light years ahead in UX. I really miss working at a company that uses Slack + Google Suites.
- Super poor formatting when trying to do more complex posts such as headings and bullet points in replies
- Copy and pasting images often don't work
- When you upload a video, sometimes the last 1-2 seconds get cut for no reason at all
- Keyboard shortcuts are suboptimal. For example, cmd+ shift + v which should clear formatting does simply does not work on Teams/Outlook.
- Grouping chats is far less intuitive than Teams channels
First of all, I highly doubt it. Second, how would you even know that?
Was that a sharp turn, or was it related in some way?
Like..where does one 'get help' for being racist? Honest question, I don't mean to sound snarky...
Man. This site is filled with weird racism.
If Capitalists could, and are doing, they'd make alabama be engineers if they could afford it.
It's so weird watching the tech work force go ham into the "tey terk ow jobz" rather than look at capitalism.