This forum topic will contain announcements from the team about any issues that are occuring for the OzBargain website and/or related services.
Site Issue Announcements
moocher on 14/04/2014 - 20:12
Last edited 15/04/2014 - 09:09
Last edited 15/04/2014 - 09:09
Comments
Probably. Maybe @Muzeeb didn't greet them cordially enough and now they demand retribution?
FFS it's probably true. FFS
OzB is broken on android desktop mode but not mobile mode. I guess limited shopping for today sigh….
This is due to the Cloudflare challenge that's appearing, don't think the app can handle it
I'm with Optus and Cloudfare won't let me onto the site. Turn on VPN and no problems.
Another DDOS attack is in progress, DDOS mitigation will be switched on again. During this time,
- you might encounter Cloudflare security checks (full page interstitial) on your browser. This should take no more than a few seconds to complete.
- the Ozbargain API will be unavailable and the mobile app will also not work.
Update
Please read this if you have problems with Cloudflare security checks or challenges looping on your browser.
Noting that I cannot access Ozbargain on mobile due to an infinite Cloudflare loop. I'm not yet sure if this is an issue on my side or not.
Details: Samsung Galaxy A32, Android 13, no custom software of interest. Steps tried: Internet cache cleared, VPNs turned off, restarted phone. This happens on both cell and wifi networks, in incognito mode, and in multiple browsers (Chrome and Edge).
I can access the site fine on a computer.
For whichever odd reason, turning OFF 'Desktop Site' fixes my issue. Magic.
Took me a while to figure this out. Hopefully the attack will stop and we can get it back.
Seems like clockwork - another DDoS happening?
cant access site just then. dammnit was price comparing
having beef with ozbargain is crazy
Yes it is. The attacker(s) basically tried different attack factors so we now basically need to put up difference defences.
Feel for you guys - best of luck.
Hopefully they lose interest soon
Is the attack directly on ozb or is it on cloudflare and affecting ozb secondary?
They all pass through CloudFlare. However CloudFlare does not block all requests, so we'll have to put in more rules on OzBargain to filter out invalid requests.
Merged from General Site Stability (Cloudflare Logo Popups)
Hey, I was just wondering if everything is okay. Over the past week or so I've noticed the site down on occasional requests and every time I log in via desktop or mobile I get a Cloudflare challenge before the site loads (which it doesn't always on Mobile).
Is there some infra migration going on? Has anyone else noticed this?That is Cloudflare attack mode coming into play I believe. It is usually switched on to mitigate DDOS attacks.
It is usually switched on to mitigate DDOS attacks.
I don't use DDOS anymore, just Windows.
jv, see a Dr. You seem to be developing a stutter.
Pollbludger security certificate expired
Scotty is preparing for war.
He is lighting the flares.
Live threads are dead!
Armageddon!!! 😲
The end is nigh. Repent!!!
Receiving big read "Sorry, you have been blocked You are unable to access ozbargain.com.au" when trying to hide expired deals on filter.
Sorry, you have been blocked
That's normal…
Sorry but you do sound like an AWS sales here. In response to your "recommendations"
we have IP level rate limiter for almost a decade, but we all know that it's useless against DDoS when the requests come from thousands of different IP addresses. Even CloudFlare WAF's "blocking known bot nets" only blocks less than 1/10 of our total fraudulent traffic.
as much as I want to complain about CloudFlare's DDoS protection performance, it's still far better than AWS Shield / Shield Advanced. AWS would cost a lot more as well.
Sorry but you do sound like an AWS sales here
Agree, LOL
we have IP level rate limiter for almost a decade, but we all know that it's useless against DDoS when the requests come from thousands of different IP addresses
Why useless, too many IPs for your solution to keep track of? Honest question. Why it fails now?
Rate limiter works by restricting a fix number of requests per time frame. For example, an IP address cannot send more than 10 requests per second. The web server will send back 429 if such condition meets.
However a bot net can just send 5 requests per second per IP address, but sent from 1,000 IP addresses across 50 countries — that's 5k rps coming your way but won't trip your IP based rate limiter.
Even a single nginx instance would have no problem tracking thousands of IP addresses in its built-in rate limiter, but that's not the solution for DDoS mitigation.
@scotty: You can maybe add page rules to only allow traffic from certain ASN's (i.e. Australian residential ISP's) with the existing rate limiting? Although some malicious requests would still persist - surely that would block out the majority of malicious requests to the point whereby the attack becomes ineffective.
@marcus84: How many Australian residential ASNs are out there? There are also many legitimate VPN users here. Limiting you certain country or ASN basically won't work because there will be too many false positives.
Currently we are deploying automated CF under attack mode (if loadavg gets too high, turn on managed challenge mode on CloudFlare) but still have to manually deploy WAF rules depending on the attack. So there's always going to be a lag between attack & mitigation. People also seem to get very annoyed by CF's managed challenge page.
However a bot net can just send 5 requests per second per IP address
I see, if a bot slows down to below the rate limiter's RPS limit per IP, he gets through … Hmmm…
- Is there public trackers of active botnets and its IP lists?
- The attacker is likely using same user-agent and some other common data, headers maybe … Can some solution aggregate on requests, see the anomaly (95% of requests have this user-agent and window-size) and act on it?I know knothing :)
Is there public trackers of active botnets and its IP lists?
Yes
@scotty: Its very interesting how Cloudflare can't seem to deal with it properly.
I understand the bot net can send 5 requests per second spread out across IP's but shouldn't Cloudflare have enough data from all the websites they protect as to which IP Address's are rogue and simply ban them permanently?
Why should we care about sync flood when all HTTP requests are routing through CloudFlare? We just need to take care of L7 attacks — just the ones not yet filtered by CloudFlare.
AWS Shield sucks as a service to actually mitigate DDoS attacks - it's way too conservative and just doesn't block the majority of attacks.
It's more so used an insurance policy since they cover the cost differential in autoscaling to handle an attack if you're using an AWS service that supports auto-scaling (i.e. ELB).
AWS Shield sucks as a service to actually mitigate DDoS attacks - it's way too conservative and just doesn't block attacks.
Thank you, valuable knowledge here. Did you experienced it yourself or have a trusted source? It is a big statement, would be nice to have details.
Combination of experience and also what co-workers have also experienced. Pretty much any attacker who has put in a little bit of effort will get through AWS Shield due to how conservative it is. Shield Advanced is also really expensive for smaller-medium sized websites.
Cloudflare is ultimately as good as it gets to mitigate against DDoS attacks - but some level of autoscaling is required because it doesn't block everything. The thing with autoscaling is - you need to be sure that everything that can potentially be overwhelmed through a DDoS attack can scale adequately - otherwise all the autoscaling in the world won't help if you have a single point of failure (e.g. DB).
@[Deactivated]: Why do you think that with a forum size of OzBargain we haven't implemented any sort of RAM based caching?
memcache has been used on all level of caching on OzBargain — queues, blocks of HTML or even entire page for guest users. We also use redis (another RAM based DB) to buffer write intensive operations. Both have been in OzBargain for more than 10 years.
How long have these DDOS attacks been in progress for now? It seems like it's been months.
The site has become borderline unusable for the past week, especially during peak periods and it's still happening today.
I can't remember OzBargain's availability being this severely disrupted in the past and I've been a lurker since 2008, member since 2009. Any idea why the site has become the target of some seriously determined threat actors right now?
I would suggest a banner notification being displayed at top of the page alerting users to the disrupted service/availability might be useful (especially when the DDOS flood peaks) as to the untrained eye it just looks like your site is severely broken currently.
How long have these DDOS attacks been in progress for now?
For the recent DDoS, the first one was detected last Saturday (13 Jan).
Any idea why the site has become the target of some seriously determined threat actors right now?
No idea. Lots of people got offended by OzBargain (disgruntled merchants, members, etc) but it could just be bored school kids.
I would suggest a banner notification at the top of the page
I'll consider that whenever we turned on CloudFlare's under attack mode, however I don't want to give the attacker any satisfaction that the attack is "working" either (as many requests got blocked before reaching our servers).
Hi Scotty
Could you unblock one of my ip?
I have slow internet, i clicked on new forum and new comments at the same time continuosly cos nothing is loading, then got blocked.For the recent DDoS, the first one was detected last Saturday (13 Jan).
It's quite obvious the same threat actors have been DDOS'ing your site intermittently for the past week now but just going by the responses in this thread from further back in 2023, I would wager they've been at it for longer than that.
Sorry "last Saturday" should have been 6 Jan. They are also probably different actors since none were as persistent as this one — it has been more than 10 days and we still got DDoS'ed.
Episode today — CloudFlare detected the DDoS at around 3:45PM. However there were too much incoming traffic (up to 200Mbps) and the upstream null routed our IP address. We have to get a new IP and reconfigure CloudFlare, and now we are back to business.
They are also probably different actors since none were as persistent as this one
Other than the script kiddie on FB who was trying to make ransom demands, have you received any sort of communication from any potential threat actor/cyber criminal?
No suspicious signs of account compromises occurring around this time?
Site security intact and no evidence of any breaches or unusual activity?However there were too much incoming traffic (up to 200Mbps) and the upstream null routed our IP address.
Whoever they are, they've definitely got access to a sizeable botnet.
It just seems weird to pour so much effort into trying to take down an Australian bargain-hunting site unless this is someone with a personal axe to grind against the site itself or some user/staff member.
That actually seems like a fairly plausible explanation just given the tendency of this site to get people hysterically enraged, which happens pretty routinely now.
@Gnostikos: Nope. If you ask me whether there are people with axe to grind, there are lots. However they have been hitting Whirlpool at the same time, so I guess as Simon has suggested — these could just be school kids getting bored during holidays.
@scotty: I love how you make these DDoS attacks sound like the modern day equivalent of school kids TPing houses over the holidays.
these could just be school kids getting bored during holidays.
School kids with access to this much bandwidth for DDOS'ing?
I don't think so. Given you had some Russian kid messaging your FB page, it's almost certainly originating from overseas.
@Gnostikos: Not sure if you’ve seen the YouTube documentaries but it’s very easy for anyone (including kids) to run a DDOS attack. They can pay people to do it or purchase the proxies themselves for as low as $0.50 each.
@WoodYouLikeSomeCash: I'm just going by statistical probability. The majority of DDOS attacks worldwide are carried out by cybercriminals, nation states and organised threat actors (hacker collectives, hacktivists, etc), not schoolkids.
@Gnostikos: Simon and Whirlpool have seen many DDoS over the lifetime of Whirlpool. Pretty sure he can tell the difference between kids and real bad actors.
He does say though that Whirlpool is "unusually ruthless in rejecting malicious traffic":
https://forums.whirlpool.net.au/thread/9l10w105#r20Not sure how "ruthless" Ozbargain is though in filtering, though may not be as ruthless given the multiple downtimes from DDoS.
@Windthunder: I think they’re more resilient because the site is not restricted to users with an account. You have to have an account to access any part of whirlpool and must go through the login page first. Whereas with OzBargain you can DDOS everything without an account.
Obviously OzBargain restricting page access to accounts only would make the DDOS’ers think they’ve won and have notable negative impacts on the site that would need to be taken into consideration.
@Gnostikos: I don't know why you've been downvoted for that.
Scotty said on the previous page there were ~4.8m requests on the homepage in an hour, which equates to 13333 requests per second. If a server is sending 5 requests per second (also mentioned by Scotty above) that comes to 2666.67 servers required for that attack. If they cost 50c each that costs $1333.33.
I don't know about others but I wouldn't be spending that money as a high school kid working a part time job (or even now as an adult on a salary) to conduct a silly DDoS attack to annoy users of a website for an hour.
I would say the most logical explanation is that this is simply a part of the wider cybercrime issues we've been seeing in Australia recently.
@Ghost47: People don't do DDoS from their own servers. They rent time slots or bot nets from hacked computers, phones, routers, network printers etc. Much much cheaper than you have outlined.
@scotty: Nobody's saying they're using their own servers. Schoolkids having the money to throw around for renting bot nets just to DDOS OzBargain and Whirlpool for months on end really doesn't compute nor does the idea that they have the technical ability to compromise vast amounts of computers to form their own botnets capable of sending up to 4.8 million requests an hour.
You already received an obvious (though poorly-constructed) ransom demand from a Russian script kiddie; that's probably where most of this is originating from or if not Russia, other known cybercrime safe-havens and common sources of DDOS attacks like China, the US, Moldova, Romania, India, Brazil, Nigeria, etc.
I'm amazed at how many people really believe that modern-day cybercrime is the domain of bored schoolkids and nerdy, lone wolf hackers. That hasn't been true for at least 10 years now.
@Gnostikos: Lol, the negs on this comment thread are hilarious.
Who is getting this butthurt over the attribution of the recent DDOS attacks against OzBargain? Do the attackers have accounts on here or has someone not touched grass for a while?
@scotty: I see, I was going off the 50c cost for proxies mentioned above (assuming that proxies meant servers that would send the requests).
IVI, it's been so long!
Who is/are your suspect/s?
JV
😲
It’s probably that guy that owns a dodgy milkbar trying to block the Amazon Warehouse thread so he can get all the nearly expired foods to sell at huge markups…
Here we go again. OzBargain has been hit by DDoS since around 10:45AM AEDT, probably one of the biggest DDoS we had so far. In 2 hours we've blocked around 194M requests and quite a few DDoS requests still didn't get filtered properly. Right now we are stilling blocking around 30k requests per second. CloudFlare's Under Attack Mode is a bit useless and we pretty much relying on custom WAF rules to prevent those requests from hitting our servers.
I guess that explains why the notifications are broken.
One of the DDoS attempt at ~12PM caused an OOM event that killed the background queue process.
Sorry but I can't help but laugh, though I'm sure it's annoying for you.
OzBargain being a little slow at times or minimal downtime is not an issue. I don't feel it's impacting our bargain hunting / community, so they aren't causing any major disturbances.
I could understand it being more of an issue if this stopped sales or OzBargain was a listed company, but if their goal was to get disgruntled users, their plan isn't working ;)
Do the attacks incur additional costs for you when Cloudflare's Under Attack Mode isn't as effective? What type of costs are we talking?
Not sure. We have been on their US$200/month Business Plan but we've chew through 2.7TB of data over the last 12 hours. They have been trying to sell me their "enterprise plan" though.
Edit: another wave of DDoS since 4:50PM AEDT and almost blocked 600M requests so far, peaked at 70k requests per second.
I have no issue accessing the site via a direct link to a deal and then hitting up different sections but the OzBargain home page is a constant "checking if site connection is secure…connection is secure" loop.
Good luck battling those turds.
@dirtybird: We've added extra JS challenges on pages that get attacked. However I am not sure why the CloudFlare security challenge doesn't work for you. I've tested on Chrome/119 or 120, latest Firefox, with or without uBlock Origin, and the browsers have no problem getting through security challenges.
Is this also impacting ozbargain.com? I notice that @Leho's no-delay telegram bot links to .com rather than .com.au. I'm not sure if there was just plain rewrites/redirects in place for that previously, but now it's well, gone
It was just parked at shared hosting. Maybe it was attacked as well & the host turned off the hosting.
Ahh, makes sense.
Is it possible to let logged-in users store a cookie which is accepted by Cloudflare for longer than 20 minutes, or whatever the current short timeout is? It seems like I frequently get re-prompted to verify my humanity. I haven't become less human in the last 20 minutes.
Challenge is only on the home page & new deals page, and the session will last 30 minutes.
Do we assume the attacker is not following this thread?
Because knowing that the challenge is only on home page and new deals page, I would link the attack to randomly generated number between 100,000 and 829,043 and postfix it to https://www.ozbargain.com.au/node/ to bypass challenge.
I have absolutely 0 knowledge in cyber security or ddos attack but it sounds like the logical step for them?
@CodeXD: We'll see whether those show up in the logs :) There was an attack from earlier this month that does something like that as any cache-bursting code would pretty much throw the servers into spiral of death.
I guess we can all take a bit of break from bargain hunting if that happens.
I use Cloudflare as my DNS, is there a reason for the last few days i always get a "checking if the site connection is secure" message when trying to browse? it DOES let me into the site, but just curious
Ozb don't like you. Deal with it. 🙃
/s
yep, pretty annoying when refreshing the page.
same, CloudFlare DNS, my home static IP address is even rego'd with cloudflare as a subdomain. seems a bit aggressive.
The Cloudflare Challenge has nothing to do with your DNS, IP address or Cloudflare account status.
Check the large running thread above - it's due to a DDOS attack against OzB and @Scotty implementing mitigation defences to attempt to keep the site operational whilst under attack.
I keep getting checked by cloud flare, even when just going back or after a refresh, sometimes I get blocked from accessing ozb.
Both mobile and desktop
Same has been happening to me. The good thing is that it doesn't ask for a captcha, just performs a check that takes a few seconds.
That's true, I'd rage quit if it asked me for a captcha.
DDoS'ed again this morning (1 Feb 2024).
- Traffic flooded in at around 11:15AM AEDT
- CloudFlare Under Attack Mode automatically kicks in at around 11:17AM
- Some custom WAF rules get deployed at 11:23AM
Currently blocking around 4-5 million requests per minute. 3rd day of heavy DDoS, and am hoping the attacker will get bored and leave soon.
Any idea why people are attacking?
No idea. No ransom note. However we suspect it's someone on OzBargain. I'm sure we've made plenty of enemies because of the way we moderate (see OzBargain page on ProductReview), but there's no evidence that the DDoS is from one of the banned accounts either.
Oh wow. There are some real cranky bums out there.
What are they trying to achieve? Just take the site down temporarily to be a nuisance?