April 3Apr 3 36 minutes ago, parisienne said:Uploads works, yes, but still no latest posts...Like activity? screenshot plz
April 3Apr 3 28 minutes ago, maddog107 said:Like activity? screenshot plzIt’s working again, but for example I made a bunch of posts during that 17 hour gap, and they don’t show up.
April 3Apr 3 2 hours ago, maddog107 said:Like activity? screenshot plz2 hours ago, Matt! said:It’s working again, but for example I made a bunch of posts during that 17 hour gap, and they don’t show up.I have the same problem.Posts between 10 minutes ago and 19 hours ago. These posts are visible but not listed.
April 4Apr 4 1 hour ago, parisienne said:I have the same problem.Posts between 10 minutes ago and 19 hours ago. These posts are visible but not listed.4 hours ago, Matt! said:It’s working again, but for example I made a bunch of posts during that 17 hour gap, and they don’t show up.Not sure how the forum handles that, it may not come back as elasticsearch is sort of ephermeral ish type storage. I would have to rebuild the whole index which takes many days to get back that small gap in the activities. Sorry.
April 4Apr 4 6 minutes ago, maddog107 said:Not sure how the forum handles that, it may not come back as elasticsearch is sort of ephermeral ish type storage. I would have to rebuild the whole index which takes many days to get back that small gap in the activities. Sorry.No worries, that makes sense. I figured it was worth mentioning just in case, but definitely not worth rebuilding the whole index over a small gap.
April 4Apr 4 9 hours ago, maddog107 said:Not sure how the forum handles that, it may not come back as elasticsearch is sort of ephermeral ish type storage. I would have to rebuild the whole index which takes many days to get back that small gap in the activities. Sorry.Will the pictures posted in that small gap stay online in the threads? Even if they're not visiable in the activity search? Thanks for your answer!
April 4Apr 4 9 hours ago, maddog107 said:Not sure how the forum handles that, it may not come back as elasticsearch is sort of ephermeral ish type storage. I would have to rebuild the whole index which takes many days to get back that small gap in the activities. Sorry.That's fine. Thanks
April 4Apr 4 6 hours ago, Modelupdater said:Will the pictures posted in that small gap stay online in the threads? Even if they're not visiable in the activity search? Thanks for your answer!Yes, that is stored in the DB. So as long as you actually saw it post and didnt throw and upload error it’s guaranteed to be there.
April 5Apr 5 5 minutes ago, maddog107 said:I knew it. It's stable again, though. Happy Easter!EDIT....mildly degraded performance Edited April 5Apr 5 by Matt!
April 5Apr 5 Degraded performance and activity feed is acting weird; I only posted the last two photos.
April 5Apr 5 28 minutes ago, Matt! said:Degraded performance and activity feed is acting weird; I only posted the last two photos.Load still climbing at 36.21. The embed pipeline is NOT running, so it's not contributing. Here's the diagnosis: It's bot traffic — distributed Vietnamese botnet What's happening: - php-fpm socket is completely exhausted — every slot is busy, new requests get "Resource temporarily unavailable" - Hundreds of unique Vietnamese IPs (113.16x.x.x, 14.1xx.x.x, 123.xx.x.x — VNPT/Viettel residential ranges) are scraping /main/topic/ pages with deep pagination - Mixed in with some DigitalOcean VPS IPs (146.190, 24.144, 167.99, 134.209) - Three major spikes today: 08:10 (worst — 15K errors/3min), 13:55, and now - All requests have referrer https://www.bellazon.com/ — faked - DB is fine (all queries completing in 0s), this is purely request volume overwhelming php-fpm Not the problem: - MariaDB queries are fast, no locks - Embed pipeline is not running - No slow query backlog Immediate options: 1. Cloudflare WAF rule — block/challenge Vietnam country code (VN) on topic pages, or rate-limit per IP 2. Nginx rate limiting — add limit_req_zone on the real IP for /main/topic/ endpoints 3. Block the DigitalOcean ASN at Cloudflare (AS14061) — no legitimate users come from DO VPS IPsI dont know why the bots are going crazy these days, we shut them down. Lets see where tehy pop up next.
April 5Apr 5 19 minutes ago, maddog107 said:Load still climbing at 36.21. The embed pipeline is NOT running, so it's not contributing. Here's the diagnosis: It's bot traffic — distributed Vietnamese botnet What's happening: - php-fpm socket is completely exhausted — every slot is busy, new requests get "Resource temporarily unavailable" - Hundreds of unique Vietnamese IPs (113.16x.x.x, 14.1xx.x.x, 123.xx.x.x — VNPT/Viettel residential ranges) are scraping /main/topic/ pages with deep pagination - Mixed in with some DigitalOcean VPS IPs (146.190, 24.144, 167.99, 134.209) - Three major spikes today: 08:10 (worst — 15K errors/3min), 13:55, and now - All requests have referrer https://www.bellazon.com/ — faked - DB is fine (all queries completing in 0s), this is purely request volume overwhelming php-fpm Not the problem: - MariaDB queries are fast, no locks - Embed pipeline is not running - No slow query backlog Immediate options: 1. Cloudflare WAF rule — block/challenge Vietnam country code (VN) on topic pages, or rate-limit per IP 2. Nginx rate limiting — add limit_req_zone on the real IP for /main/topic/ endpoints 3. Block the DigitalOcean ASN at Cloudflare (AS14061) — no legitimate users come from DO VPS IPsI dont know why the bots are going crazy these days, we shut them down. Lets see where tehy pop up next.Start with Cloudflare WAF/rate-limiting on /main/topic/, and block the DigitalOcean ASN, since the goal is to stop requests before they exhaust php-fpm?EDIT: It’s early morning in Vietnam, so I’m not sure how many legit users would be affected by a full VN block. Edited April 5Apr 5 by Matt!
April 5Apr 5 38 minutes ago, maddog107 said:Load still climbing at 36.21. The embed pipeline is NOT running, so it's not contributing. Here's the diagnosis: It's bot traffic — distributed Vietnamese botnet What's happening: - php-fpm socket is completely exhausted — every slot is busy, new requests get "Resource temporarily unavailable" - Hundreds of unique Vietnamese IPs (113.16x.x.x, 14.1xx.x.x, 123.xx.x.x — VNPT/Viettel residential ranges) are scraping /main/topic/ pages with deep pagination - Mixed in with some DigitalOcean VPS IPs (146.190, 24.144, 167.99, 134.209) - Three major spikes today: 08:10 (worst — 15K errors/3min), 13:55, and now - All requests have referrer https://www.bellazon.com/ — faked - DB is fine (all queries completing in 0s), this is purely request volume overwhelming php-fpm Not the problem: - MariaDB queries are fast, no locks - Embed pipeline is not running - No slow query backlog Immediate options: 1. Cloudflare WAF rule — block/challenge Vietnam country code (VN) on topic pages, or rate-limit per IP 2. Nginx rate limiting — add limit_req_zone on the real IP for /main/topic/ endpoints 3. Block the DigitalOcean ASN at Cloudflare (AS14061) — no legitimate users come from DO VPS IPsI dont know why the bots are going crazy these days, we shut them down. Lets see where tehy pop up next.Which option did you go with? I guess just forcing a Cloudflare Captcha/Challenge should do the trick, no?
April 5Apr 5 4 minutes ago, anon2222 said:Which option did you go with? I guess just forcing a Cloudflare Captcha/Challenge should do the trick, no?Yea pretty much, a JS check from time to time.
April 5Apr 5 10 minutes ago, anon2222 said:Which option did you go with? I guess just forcing a Cloudflare Captcha/Challenge should do the trick, no?1 minute ago, maddog107 said:Yea pretty much, a JS check from time to time. Edited April 5Apr 5 by Matt!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.