I spent the past year creating Price Hipster, a product search engine and price watching website for the OzBargain community!

Update 13/11/15: Thanks everyone for the support and feedback! I realise there are heaps of comments here and no one wants to read through them all. If you have a suggestion, it has probably already been raised. I'm currently working on a big update and I'll be sure to let you all know when I release it. I'll be keen to get further feedback and suggestions then. :-)

Hi fellow OzBargainers!

I've spent the past year creating a powerful tool for you all.

https://www.pricehipster.com.au/

These are the reasons I created Price Hipster:

  1. Searching for products across multiple stores is painful. Sometimes you don't even know which stores to check for a product. Wouldn't it be great to conveniently search all stores from one place?

  2. Even if you want to find a product in one particular store, most store websites are very bad at searching their own products, and they're slow at it.

  3. There is no convenient way to find out the previous prices of products, making it difficult to hold vendors accountable for false claims of discounts. Some products vary wildly in price. Some change price nearly every day. For example:

https://www.pricehipster.com.au/product?id=NJNmxsZ3k5n

I really appreciate those OzBargainers who have taken the time to use internet archives to dig up previous prices, but it's less than ideal, and hopefully will now be a thing of the past!

Price Hipster currently includes 5 websites in its index. That will expand in the near future.

Questions for you all:

  1. Which stores should be added next?

  2. OzBargainers in Western Australia, how is the search latency for you? The search servers are in Sydney. I live in Brisbane and the search is pretty much instantaneous here.

If you find any issues with the website or have any suggestions, I'd love to hear them!

Related Stores

Price Hipster
Price Hipster

Comments

  • How does your site compare to http://www.lasoo.com.au/home.html

    • +55 votes
      1. Much faster search.

      2. Cleaner.

      3. Price history data.

      4. Daily email of biggest discounts (I don't know if Lasoo does this. Maybe they do).

      5. Many more features incoming, like price drop notifications.

      6. Indexes ALL products, not just those in catalogues.

    • It seems that jazoom's website is much more relevant for us as the consumer. This is extremely useful for price cycles and local stores that we can go to. I'd much rather buy a phone that's 20% off at a local store than somewhere online with no physical store.

      Yes, please expand, and two, can you add a date scale on the x axis (currently blank). Furthermore, seems like there's a bug, eg. Samsung Galaxy Note Edge - Charcoal Black; it doesn't show when and how far under $1000 it was as the y-axis isn't properly scaled. This is like camelcamelcamel except for much more than just amazon.

      If there was an option to plus this, I'd definitely do it.

      •  

        Hey, I'm guessing you're using Firefox? There's a bug with the graph at the moment. It's not supposed to be missing the bottom bit.

      •  

        The chart should be fixed now.

  • Looks good, will give it a go. How are you monetising (or planning to)? Advertising, PPC, Donations, Subscriptions?

    Good luck with the venture!

    • +9 votes

      Thanks!

      I'm not terribly concerned with monetisation for the time being, but the plan is to allow some vendors to advertise genuine deals among the initial products that load.

  • I love the design and speed but 5 stores is extremely small and currently useless to be useful.

    Which stores should be added next?

    Answer: every single one you can find.

  • +19 votes

    Great website! I've bookmarked it :)

    Just a suggestion, maybe have an advanced search option which allows sort by price and also an option to enter a price range eg. $500-$600

    • +3 votes

      Yup, that's already half-implemented. ;-)

      Thanks for the support!

      • Also, exclude certain keywords, sort by price/price per unit (including shipping or not including shipping if pickups allowed and in the same state/within x km), and do a generic product search (eg: for example with Computer Memory, filter with - DDR3, 1.5v, non-ECC, 1600mhz PC-12800, 240-pin DIMM)

    • +1 vote

      price-range and store filters are now implemented :-) More info here.

  • Heh, I was tossing this very idea/concept for a site around in my head - so terrible idea mate, please stop it. Joking, all the best and if you do it right great idea ;)

    Also catchy name but it might get too mainstream and people will hear about it :D

  • It's a great idea! I like it a lot

  • Just out of interest, how do you make this sort of tool?

    Presumably these places don't provide an API to accommodate price lookup, so I assume that you'd have to find some way to automatically parse pages and then extract the correct fields?

    (I'm not about to make a competing one, just interested in roughly how it's done)

    •  

      It's roughly done the same way Google or any other search engine gets their data. It's known as crawling or web scraping.

      • Just curious I know lasoo have agreements in place with the stores. Is web scraping off these sites a breach of their TOS or is it legal?

        • +13 votes

          Price Hipster is a search engine. If it was illegal then Google would not operate in Australia.

          If stores don't want their data indexed I'll take them out, but a store would be pretty dumb to request less visibility and less traffic. I don't think shareholders would appreciate that move.

        • +6 votes

          @jazoom:
          I took one of your 5 indexed sites (JB Hi-Fi) and skim-read their Terms of Use which directly contradicts your position (and is what I think @blighst was getting at):

          Copyright
          This website and its contents are subject to copyright. The copyright is owned by JB Hi-Fi, or in the case of some material, third parties.
          You may view this website and its contents using your web browser and electronically copy and print hard copies of parts of this website and its contents for personal, non-commercial use. Except as permitted under the Copyright Act 1968 (Cth) or similar statutes that apply in your location, any other use (including any adaptation, reproduction, broadcast, decompiling, disassembling, downloading, copying, storage, distribution, transmission, printing, display, publication or creation of derivative works) of any part of this website is prohibited.

          So perhaps not illegal but you are at least in breach of JB's stated "Terms of Use" and I'd be very surprised if it's not the same or similar for all of the sites you've indexed thus far.

          You might get away with this if you really are doing this "for the greater good" in that you will (probably) fly under the big corporation's radar. But if you ever do monetise the site (especially if it gains significant popularity) I reckon you'd best investigate the legality of your site and proposed business model.

        • @jazoom: Yeah but with a robots.txt file webmasters can choose not to be scraped. As @ckc said JB Hi-Fi does not permit this so probably something to look into.

        • +3 votes

          @ckc: Yeah, I wouldn't want to make a profitable search engine. That didn't seem to work out too well for Google.

          I'm not selling anything. All I'm doing is sending people their way.

        • @jazoom: Anyways nice website - what's the backend written in?

        • +3 votes

          @blighst:

          This is their robots.txt:

          User-agent: *
          Disallow: /Static/*
          Disallow: /Products/Product-Detail/AddedToCart/
          Disallow: /My-Account/
          Disallow: /Search/results/
          Disallow: /Stores/Store-Finder/?Initialparams*
          Disallow: /aspxerrorpath=
          Disallow: /productid=
          Disallow: /ReturnUrl=
          Disallow: /refferer=
          Disallow: /id=
          Disallow: /keywords=
          Disallow: /initalparams=
          sitemap: https://www.jbhifi.com.au/sitemap.xml

          Where does it say they don't want their products indexed?

          In fact, they provide a sitemap to all their products to aide indexing.

        • @ckc:

          Yeah nah. If JB were to sue (which they wouldnt) they would need to establish a loss they're suffered.

          I don't see how sending more customers JB's way would cause a measurable loss, probably more sales!

        • +1 vote

          @zeggie:
          Sue? Unlikely.

          Cease and Desist? More likely (in my non-lawyer opinion).

          My assumption is that the bigger stores aren't interested in a race-to-the-bottom with ever-slimming margins against online retailers with truly cut-throat operations. They want to control their own messaging, and heavily discount when it suits (e.g. used volume to secure greatly reduced wholesale) and still make a healthy profit on their "regular" items.

          I could be wrong

          And OP: I don't reckon they are "against indexing" per se. I just reckon you are (probably) in violation of their terms. Read it again and explain to me how your use is "personal" and doesn't involve "copying", "transmission", "distribution" and/or "displaying" of their proprietary data?

          But I'm reasonably sure we could go a few more rounds and not agree on this point :-)

          I do have a suggestion: why not ask them?
          If you are confident you aren't in breach then get in contact (email or letter) and get in writing they are on board. Then I'll be able to sleep at night knowing you have covered your arse!

        • @zeggie: Measurable loss perhaps from the increased web traffic they experienced from the scraping. Going out on a limb but could hold up.

        • +1 vote

          @jazoom:
          Google make (most of) their money from advertising.

          You already said in this comment that you were considering monteising by advertising.

          That means you are planning on selling advertising. Or have I missed something?

        •  

          @blighst: It costs peanuts to serve even large traffic nowadays.

        • @ckc:

          Ceast and desist? Yeah nah.

          Wouldnt Ozbargain with 10000x more traffic be a target first?

          Your non-lawyer thinking missed the important factor. What are the potential rewards? A website run by one guy? Not much there.

          JBs outsourced legal probably charge $250 just to send a C&D.

        • @zeggie: Yeah I'd be more worried about actively blocking his crawlers then legal action.

        • @zeggie:
          I looked at the headers, and all the images are hotlinked, which might annoy webmasters, who still pay for traffic. If he kept local copies, he might have copyright issues.

        • @blighst: I once wrote an opinion for a client on the legality of web scraping (a few years ago now). To summarise about 20 pages of legalese, we were of the opinion that it was actionable as a trespass to chatels - ie the trespass was on the servers that they were accessing.

          There has even been a few successful suits in the US on this exact topics (one was regarding Ebay and an auction sniping service). So it's possible.

          Ultimately, before a law suit there would be a C&D. As long as the OP has a policy of removing stores that request it he should be ok. Some might but I doubt most will care.

  • There has been several price tracking sites mentioned here in ozb. I have never heard of any them again recently so I assume they were all gone now. Anyone knows what happened?

    • Upkeep and monetization is generally not as expected

      • +19 votes

        100% this.

        AUS Retailers don't see it the way the US Retailers do. We were threatened several times by big retailers. They don't see that the people you're referring to their site as highly-relevant traffic (they're going to the retailers site to look at an item that you've shown them is good value).

        In fact, whilst running one of these sites, I had to maintain an active proxy list to get passed lifetime IP bans placed by the big retailers of the country.

        The amount of traffic I was pulling through in a night for the 15 or so retailers I was tracking was in the 10's of GBs. We had to come up with an algorithm that only checked items that changed frequently, and that still only brought it down to 4-5 GB in a night. And this took a long time, at 100% CPU and about 32GB of RAM completely maxed out, we're talking 10-12 hours to get all the information necessary.

        In the end we sold all of our algorithms and site database to a company. We ditched the user database as per our privacy policy and moved away completely from price tracking, there is just no money in it if retailers aren't willing to play ball by offering a price feed.

        Good luck out there mate, you can choose to take on board some of this or find a nicer way to implement it all.

        • +2 votes

          Thanks for the info. I really appreciate you sharing your experience!

        • You mind sharing which company you sold the algorithms to?

        • @blighst:

          Unfortunately I'm not in a position to share that. It wasn't a large amount and didn't recoup anywhere near what we spent on the site ;)

        • Some retailers offer a price feed for Google to get their items listed on Google Shopping.

          Find this feed URL and you have all of the products they list in one handy XML feed…

      • Not to mention the cease and desist letters from the big retailers I'm sure we never hear about.

    • There was a really good one with price tracking feature
      https://www.poachit.com/
      http://www.wired.com/2014/08/poachit/

      but seems to be closed too now :(

  • Good idea, nice website - great work!

    Just with the price history graphs, maybe make it so that when you hover over a date it will give you an exact price? (Very) slightly inconvenient going across and having to approximate the price it was on a date.

  • hey jazoom,

    I'm really interested in the dev stack you're using. Feel free to send me a PM. Are you using a cache mechanism to get the fast search happening?

    • +1 vote

      My stack is actually pretty lean. I've spent a lot of effort to minimise dependencies, do things efficiently and keep everything as lean as possible (the whole site loads with only 125kb down the wire). I'll send you a PM with more info.

    • The implementation looks like:

      • algolia (for search)
      • react (interface and model)
      • cloudflare w/ rocketloader for caching/inlining assets such as JS

      Further the product images are actually stored on the content provider (aka woolworths).

      So really the main work is writing crawlers that parse and build an algolia index which includes historical pricing data.

      A great example of using SaaS.

      • +1 vote

        Actually, I inline my own schtuff. Lucky, since I had to abandon Rocket Loader due to it causing issues. You'll be pleased to know that the site loads just as quickly without it.

        • Hi Zazoom,

          I am really impressed with quick search results from this awesome site.
          I have little curiosity to find the way you implemented
          1. When user search for a product, are you directly hitting the sites for search or fetching form pricehipster DB?
          2. How frequently are you using crawling to query data and fetching into DB?
          3. Did you used node.js to implement the crawling or any third party tools used?
          4. I believe, you were maintaining database to store all the information daily which will help for trend graphs( for price comparison)

          Sorry for too many questions.. but your response will really helpful for me.

        • @vasus:

          Your post comes across like you are someone from one of the sites been crawled and want to find out more info so you can stop it.

        •  

          @matt_will_fix_it: I can't imagine anyone wanting to stop it since I provide quite a lot of traffic to them. I'd think it's more likely someone wanting to do something similar themselves.

        • +1 vote

          @vasus: Hi Vasus,

          1. Searches go directly to my database, otherwise the searches would be slow and the results poor.

          2. Many times per day.

          3. Yes. I designed my own crawler in NodeJS.

          4. True.

        • @jazoom: Thanks a lot Jazoom..

  • I'm in WA,

    It's awesome, the speed, the structure.

    I'm impressed.