HwKeeper
Auto-translated
As a result of analyzing the logs over 4 days, it was possible to identify a potential cause of such a heavy load.
A huge share of traffic and requests to the site is made by bots of various kinds, with classic search engines like Google and Yandex causing little harm, while bots like GPTBot, Amazonbot, AwarioBot, MJ12bot pose a direct threat.
Over these 4 days, the bots collectively made about 800 thousand requests out of a total of 1.9 million and consumed about 29 GB of traffic out of 56 GB.
I added a script that should restrict bot access to the site when a certain threshold of requests per minute is exceeded, and completely blocked some of them.
I think these measures should ultimately stabilize the site's operation.
A huge share of traffic and requests to the site is made by bots of various kinds, with classic search engines like Google and Yandex causing little harm, while bots like GPTBot, Amazonbot, AwarioBot, MJ12bot pose a direct threat.
Over these 4 days, the bots collectively made about 800 thousand requests out of a total of 1.9 million and consumed about 29 GB of traffic out of 56 GB.
I added a script that should restrict bot access to the site when a certain threshold of requests per minute is exceeded, and completely blocked some of them.
I think these measures should ultimately stabilize the site's operation.