Board Update: On Last Night's Issues at 11:00PM
Hey all,
I wanted to let you all know what went on last night with the boards.
At 10:50PM last night I was notified by our PRTG Network Monitor that the backend API had stopped responding during the game. I immediately reached out to Will Stewart to check if we were getting connectivity reports and emails from subscribers.
At 11:00PM we were serving 1.28 Million requests per 15 minutes, or 85,333 Requests per Minute, 1422 Requests Per Second.
I love our community, but we are not that big. Normally we have 1000-2000 active users during games.
Looking at the data I have at Cloudflare, it looks like there were 1.49 million requests by bots last night between 10:45 and 11:15PM. The servers were responding again by 11:30PM according to PRTG.
I have a set of IP addresses that were each doing 3.5k requests during this time period. I'm going through the process of compiling them and banning them as well as submitting them to the authorities. I will also be cross referencing them with our user database and see if anyone matches up.
Nobody's information has been compromised, the only thing that "happened" was that we received over a million fake requests during the roughly 20 minute period.
Thank you all for being patient with us. I'll be putting extra security in place(Cloudflare offers extra DDoS protection, but we have most of those features disabled as this has never happened before).
Feel free to ask me any questions, or if you have information about the incident please email me at [email protected]
8 Replies
H
hok1eboy
3y
(In response to this post by John Donna Choi)
P
Polito1
3y
(In response to this post by John Donna Choi)
3y
(In response to this post by John Donna Choi)
S
Stech
3y
Just to prove they can? Because they don't like TSL? Because they don't like Virginia Tech? Any clue as to why?
(In response to this post by John Donna Choi)
3y
(In response to this post by John Donna Choi)
3y
Even if the bots don't have a password, they still can request for the page. We respond "you don't have access to this" and redirect them to login.
Each message board home has its own HTML cache. What I'm guessing happened is that the site got overwhelmed with bot requests and couldn't generate the page cache.
** Edited by John Donna Choi at 10/28/2022, 5:15:24 PM
(In response to this post by TomTurkey)
It was the Russians?!? Chinese?!? Both?!? We bin hacked!!!
(In response to this post by John Donna Choi)