Woolworths PriceHipster/Price History - help finding finding Woolies price history - or help web scrapping Woolies using API

Related Stores

Woolworths

• 

  singingwolf on 03/07/2021 - 16:42

  +1

  10 PRINT $7.95
  20 GOTO 10
  30 RUN

  • 

    MiniMoo on 03/07/2021 - 19:12

    +3

    Syntax error in 10

    • 

      afoveht on 03/07/2021 - 19:30

      The program never ran and line 30 is another syntax error.

• 

  c64 on 03/07/2021 - 19:56

  they probably fingerprint your client, in which case you'll need to use selenium or something similar

  • 

    jazoom on 04/07/2021 - 08:32

    That won't help. They block IP addresses too. Just from browsing around a lot at home you will get blocked at your home. Just be patient. They'll stop being stupid eventually. I've seen this many times before with many different retailers.

    • 

      c64 on 04/07/2021 - 10:00

      having a dynamic ip address has some advantages :)

• 

  FFx0 on 03/07/2021 - 20:37

  +1

  You can just use their API (238473) was from the URL known as the stockcode:
  https://www.woolworths.com.au/apis/ui/product/detail/238473

• 

  jakka on 04/07/2021 - 10:12

  Yes, you can use their API. Ihave created a Google Apps Script function to pull the price and various other pieces of information from their API, feel freet o use the below.

  function WOWData(Article, Attribute){
    var url = "https://www.woolworths.com.au/apis/ui/products/" + Article;
    var result = UrlFetchApp.fetch(url).getContentText();
    var JSONResponse = JSON.parse(result);
    if (JSONResponse.pageCount===0) {
      return "";
    }
    else {
      switch (Attribute) {
        case "Barcode":
        return JSONResponse[0].barcode; 
        break;
        case "Price":
        return JSONResponse[0].Price; 
        break;
        case "InstorePrice:":
        return JSONResponse[0].InstorePrice; 
        break;
        case "Name":
        return JSONResponse[0].Name; 
        break;
        case "IsNew":
        return JSONResponse[0].IsNew; 
        break;
        case "IsOnSpecial":
        return JSONResponse[0].IsOnSpecial; 
        break;
        case "InstoreIsOnSpecial":
        return JSONResponse[0].InstoreIsOnSpecial; 
        break;
        case "IsEdrSpecial":
        return JSONResponse[0].IsEdrSpecial; 
        break;
        case "SavingsAmount":
        return JSONResponse[0].SavingsAmount; 
        break;
        case "WasPrice":
        return JSONResponse[0].WasPrice; 
        break;
        case "IsInStock":
        return JSONResponse[0].IsInStock; 
        break;
        case "Brand":
        return JSONResponse[0].Brand; 
        break;
        case "IsInFamily":
        return JSONResponse[0].IsInFamily; 
        break;
        case "tgawarnings":
        return JSONResponse[0].AdditionalAttributes.tgawarnings; 
        break;
        case "tgawarning":
        return JSONResponse[0].AdditionalAttributes.tgawarning; 
        break;
        case "vendorarticleid":
        return JSONResponse[0].AdditionalAttributes.vendorarticleid; 
        break;
        case "vendorcostprice":
        return JSONResponse[0].AdditionalAttributes.vendorcostprice; 
        break;
        case "Promotion":
        return JSONResponse[0].HeaderTag.Content; 
        break;
      }
    }
  }
  

• 

  rebates on 03/08/2021 - 13:09

  Hi Jeremy, were you ever able to get this working? Maybe we can work on it together; I have basic Python skills for scraping if that helps.

  Edit: PS. This is a longshot, but does anybody call you "JB"?

  • 

    wittyusername on 10/10/2021 - 10:28

    Hi mate,

    I know this is an old thread, but would you happen to have any resources to help me learn how to cobble something together to price scrape some camera sites?

    • 

      rebates on 11/10/2021 - 06:47

      Hi, do you write in Python? Or alternatively you could try Octoparse with the free trial.

      • 

        wittyusername on 11/10/2021 - 08:26

        Cheers man, I am giving octoparse a crack. Seems good so far!

• 

  muppet on 21/08/2021 - 14:54

  I wrote a code to check for my favourite items at woolies every day. My code used to call their apis/ui/product/detail to retrieve the json data, but they have implemented security checks recently. Upon closer examination, the json data is also embedded in the html source, e.g. https://www.woolworths.com.au/shop/productdetails/439768/kbs-prawn-gyoza

  It would contain this at the end of the html source:

  ....:0,&q;InstoreSavingsAmount&q;:0,&q;WasPrice&q;:22.7,&q;InstoreWasPrice&q;:22.7,&q;QuantityInTrolley&q;:0,.......

  So I would first strip off the &q;, then just look for ,FieldName:(.*), and extract the value using regex.

  • 

    c64 on 21/08/2021 - 20:33

    replace

    &q; -> "
    &s; -> '
    &l; -> <
    &g; -> >
    &a; -> &

    then use a json library to parse the resulting json string

    • 

      muppet on 22/08/2021 - 07:44

      Thanks! Do you know what those things are called? They're not standard html entities. I would've preferred to clean them up using a library rather than manually.

      • 

        c64 on 22/08/2021 - 13:23

        they are custom html entities that woolworths has decided to use

  • 

    wittyusername on 10/10/2021 - 10:29

    Hi mate,

    I know this is an old thread, but would you happen to have any resources to help me learn how to cobble something together similar to what you've done? It'd be for camera sites who I don't think protect their prices as vigorously as woollies seem to.

    Cheers.

    • 

      muppet on 19/10/2021 - 19:51

      basically you first need to figure out where the data is located. I used the browser's developer mode and look at the html source and also look at what other requests it makes, like on this screenshot: https://pasteboard.co/4P2IRcMLijB5.png

      Then just write code in your favourite language to make the same GET / POST http request. Usually the data is in json format.

• 

  Apologiesfor on 15/10/2021 - 19:58

  Hi JeremyDealFiend,

  I've been working on a tool to access the Woolworths API using the R programming language. If you're interested, it's over here

  • 

    JeremyDealFiend on 15/10/2021 - 20:15

    Thank you very much