Twinkl.com.au is not accessible for members due to technical difficulties. Therefore twinkl.com has free ultimate access for everyone until midnight. Great teaching and learning resources for teachers and parents. Just need to create an account for free.
F
writing script too as we speak, but with a massive pool of dc proxies ;) - no limits
Noice. If we can divide and conquer, maybe that would be faster since I'm not sure the endpoint will still be publicly available. Can you start crawling from 3000+?
Btw I find DC IP addresses way easier to get blocked since any WAF worth their salts already have egress traffics from popular public cloud providers blacklisted
sure, lemme finish my script then il switch to rotating resi - oxylabs my beloved <3
@PopCounty: I changed the strategy already, it's no longer an incremental thing.
I looked at the site's sitemap (so if you look at this https://www.twinkl.com.au/sitemap/www.twinkl.com.au/sitemap_…, the page has smaller resource XML group such as https://www.twinkl.com.au/sitemap/www.twinkl.com.au/resource…). These sitemaps are deliberately placed there so that Twinkl has better SEO Ranking on Google - in this case it's helpful to us.
Then each of the resource page links to a normal page, and that leads to a download button.
So my bot is no longer doing just naively +1, it is way smarter.
I rewrote it to use Puppeteer now (so it mimicks the behaviour of a Google Chrome user in front of a computer).
@Leho: nvm ive realised posting the resources is probably a copyright notice death wish
selenium written script is posted here (https://gist.github.com/Lehoooo/25096c03806888ddfdade82ac32c…) - works a lot better than requesting it directly as no auth, no cookies, no proxies needed. probably pretty buggy as I wrote in like 10 mins lmao
requires firefox, geckodriver, selenium, python3
pm if you need help :)
Thanks guys. Would be great to see the resources shared here.
Pretty sure Scotty has eyes on you…. So thus many hiring managers. Congrats if you find some (better) fancy job from some bargainer. Good luck mate !!!
Is there anything much on here for senior maths? The various famous mathematicians posters are nice (downloaded a few packs and might put them up in my classroom) but nothing actually content-related that is appropriate for my students.
No, not really much for anything past Year 10. Might have to get creative on the searches though.
While my bot is still running, Here is an update on the first drop so that anyone waiting can get something :).
So my bot/script managed to snatch 2,021 resources so far, and it's currently sitting at around 12.93GB (so… whoever said they got the whole site for 8.4GB is probably not true). And I don't think I'm nearly one tenth of the site yet. That's not counting that many experiences are interactive so that cannot be saved offline.
The good thing is I managed to download all the Twinkl original eBooks (they are pretty nice), a lot of Australian related content (since I directed the bot to grab the sitemap from the Australian / Australian curriculum) search keywords. The downside is that since I only download the files, you might have a bit of a hard time categorising which files belong to which class, but the file names will give you some hints.
I will keep the bot running until, well, the free download period or somehow Twinkl kicked me out, and I promise I will do a drop at the end.
P/s: Fun fact, all the tools today I used were made possible from all the deals I got from Ozbargain:
Thanks nkahoang, it's appreciated!
Incredible, really appreciate your work and want to give back nkahoang, and the update is great.. Do you think you'll have to begin rotating through WA based IP's past 9pmWST/12amEST?
The files appear to work without logging in, so may continue working
Thanks for the update, mate. I'm sure the OzB community is thanking you wholeheartedly for the content you are grabbing for us. I also learned quite a bit from your explanations and used ChatGPT to create a script based on your instructions. I got some content, but kept getting kicked out - I guess I'll leave that to the experts then. :)
selenium ;)
Could you write a blog how you manage to do this? So that peasant like us can learn to be like you one day?
Ps: in the blog you can use fictitious website
Thanks
Are there any Year 11/12 resources?? Site is a mess!
Search by specific topic. Also expand to UK content. Itll still be limited.
I am not sure what is the problem but I can not download the file after 5GB download it just stop and says Failed - Network error, if I click on resume it says Failed - Forbidden
Yeah my download just failed after a few gig. Thanks for the effort though
thanks. got the entire zip! :D
I think @nkahoang Google drive hit the daily limit. It will be helpful if you can share your drive link as well. Thanks.
yep. will do tonight/this weekend and share.
left for work early this morning
Thanks!!
no issues with downloading this batch of files.
Can you share your thoughts on how to download as batch files ..?
unzipping it is going to take awhile.
hi allergic2work, that link seems to be the same as nkahoang's and not a different one.
So it looks like my bot can continue past the cut-off date, but I have stopped it for now as I don't want to abuse their good will of opening it up for everyone to download.
Some learnings:
In the end the automation script downloaded more than 5,500 assets, totalling more than 26.37GB (and this is zipped). Twinkl has around 900,000+ assets in total, so that any offline saving attempt would be really challenging. The download rate is around 1000 assets an hour, give or take, on my internet bandwidth + VPNs.
The method that I takes would mean it must run for ~40 days, and in total, it would exceed 4.5TB to save everything.
Furthermore since it doesn't have the category, the files are not easy to look through the 900,000+ files either.
The more I look into this, the more I admire the effort and how resourceful this website is, so for ~ $13/mth to access - i think it's a good deal.
From the technical side of it for anyone interested, any attempt for an easy loop of aria2c, selenium or puppeteer script, or any off the shelf website offline saver will probably fail:
- The WAF is very clear; once it detects a "pattern" (even if you try to emulate opening a page, clicking on button), it will ban you.
- It requires a combination of frequently changing IP addresses, creating new browser context (clearing your cookies and swapping user-agent), and introducing heuristic waits/timed-out.
It certainly was a fun exercise, though.
Appreciate your work and time mate! By the sounds of it a subscription does seem more practical while at the same time also supporting them!
wow, thanks for the background and the insights!
Thanks again for your effort. Totally understand and I'm considering subscribing to their service if the resources are genuinely good.
Btw any chance you could upload and share the remaining resources that you have downloaded?
Thanks for the awesome effort mate! But does this mean there will be a 2nd drop?
Hi all. I’m the original poster 👋.
Any chance someone downloaded an Easter activity booklet. I forgot to look for Easter resources and mainly focused on literacy and numeracy.
I tried the Google drive link but too many people are trying to access it at once. Thanks
You might just have initiated pathway for someone’s being fired. And at the same time a roadmap to strengthen their processes.
OzB is really a wonderful site to stress test and do security assessment.
Sorry, you can't view or download this file at this time.
Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.
ahmmm can anyone help pls =)
I got transfer quota exceeded warning. Would you be able to upload to my dropbox if I sent you a link.
@navarch: You should be able to import it into your Mega account, the warning is just that, a warning. It will resume the sync by itself later. It may take a few days but it will be there later.
@x x: Import just means file added into your own cloud storage for you to keep and download at anytime
@x x: I don't know about the importing, but you can get the download still from that link, it'll prompt you to pay due to free quota limit reached (about half way) and give a time of sixish hours till it will resume, it does it automatically, so just leave that web page going overnight and it should be ready for you by morning
To allergic2work for providing another avenue to nkahoangs great work, thank you!
Currently downloading the package that was uploaded to Mega. Thinking about sharing it via torrent, so no download limits or quota problems. Let me know if anyone's keen and i'll set it up (just have to put up with my crappy NBN speeds until people start seeding)
Heres a magnet link to the .zip file originally uploaded by nkahoang.
To download, you need a torrent client already installed (uTorrent, qBittorent), copy/paste or click (not sure how OZB will format the link) the magnet link into your web browser, it will want to open up your torrent client, click yes/accept/open and click download within your torrent client. Please seed once downloaded as my internet is crap. Cheers
magnet:?xt=urn:btih:c84d997cc861c19e665c84c656c7fcd4fb96dbe9&xt=urn:btmh:12209a5374336cd6c8eb996825a441c817dc79f479482d8e1c964ae13dc1a9c33ae0&dn=twinkl_ozbargain_first-drop-2023-02-23%202.zip
**This is for educational purposes only. I will be taking this down after a few days in respect to Twinkl. If you like the files contained in the .zip, please show your support by subscribing to Twinkl.
Cheers mate. Currently stuck at 'magnetized transfer - retrieving metadata (85.7%)' with 5 peers. But will stick to it.
No worries mate, will be leaving the pc on for a few days to seed. Hopefully you're able to get it man!
thanks, downloading now and will seed when i get it all
@Konecs Can we have the 2nd part, please?
Thank you so much!
Please drop the 2nd batch of files
Thanks guys, downloaded the 1st drop, now waiting for the 2nd drop please.
hello @batrarobin anyway to share this? my torrent just not getting any seed…thanks
No worries, will look into creating another torrent shortly.
are you able to please make another torrent?
same here
What a legend!! Waiting on 2nd drop of files.
is anyone able to provide me a link to the download?
@allergic2work are you able to provide a link?
This deal is on again! Can anyone pass me a download link to access the resources offline? Thanks
Your are a legend mate ….! Waiting for the link … hopefully 🤞