Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.

Forum rulesBefore asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the Google search box (at the top of each forum page) to see if a similar problem or question has already been addressed. This will search the entire contents of the forums as well as the iMacros Wiki.
3. We can respond much faster to your posts if you include the following information:

Would it be possible to extract pricing data from comparison sites like Google flights?

Searching for good deals on flight comparison websites is arduous to say the least.

Each month I want to be able to run automated searches for multiple destination across Europe for a combination of dates and extract all of that information into Excel to be processed to find the cheapest days and times to fly.

millenialme wrote:Would it be possible to extract pricing data from comparison sites like Google flights?

Searching for good deals on flight comparison websites is arduous to say the least.

Each month I want to be able to run automated searches for multiple destination across Europe for a combination of dates and extract all of that information into Excel to be processed to find the cheapest days and times to fly.

Can it be done?

Yep of course, iMacros is for Web-Automation and Data Extraction...!

Why do you think we have a Sub-Forum dedicated to "Data Extraction and Web Screen Scraping"...!?, that you managed to find very correctly to open your Thread...!

- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...

millenialme wrote:Would it be possible to extract pricing data from comparison sites like Google flights?

Searching for good deals on flight comparison websites is arduous to say the least.

Each month I want to be able to run automated searches for multiple destination across Europe for a combination of dates and extract all of that information into Excel to be processed to find the cheapest days and times to fly.

Can it be done?

Yep of course, iMacros is for Web-Automation and Data Extraction...!

Why do you think we have a Sub-Forum dedicated to "Data Extraction and Web Screen Scraping"...!?, that you managed to find very correctly to open your Thread...!

Brilliant!

I'm very new and lack a lot of knowledge when it comes to this. I experienced some iMacro use in a previous job and saw the potential for it to be used in this way. I'm in no way an expert in stuff like this, quite the contrary.

Is there anyone here capable of scripting something like this? Or do any of you know a freelancing type site which is reliable for posting jobs such as this?

I'm very new and lack a lot of knowledge when it comes to this. I experienced some iMacro use in a previous job and saw the potential for it to be used in this way. I'm in no way an expert in stuff like this, quite the contrary.

Is there anyone here capable of scripting something like this? Or do any of you know a freelancing type site which is reliable for posting jobs such as this?

Yep, it sounds like a "useful" Use of iMacros indeed, and a "Fun" Project..., but you should give it a try yourself, it's not very-very complicated, and it's Fun to make your own Script...

You can post (open a new Thread) otherwise/anyway in the 'iMacros Consultants' Sub-Forum and you'll probably get a few Developers interested... (And a few have already posted their Contact Info in that Sub-Forum...)
(And mention a way to contact you as PM is disabled on the Forum...)

(I'd be willing to do it if you don't get any Reactions or don't find anybody after, say..., 1 week or so..., but...,, hum..., Scripts I make for other Users usually end up as a "never-ending" Project in my Exp as I'm a bit too much of a "Perfectionist" myself I guess, and in my Exp, Users always come up with extra Requirements halfway, which requires a Rewrite/Rethinking of the Script, so you'd need to formulate upfront exactly what you want and what your Req's are... (And it's much "easier" to make a Script for yourself than for sbd else, ah-ah...!))

OK, good luck anyway...!

- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...

But, hum... (lots of "hum", hum-hum...!, I think it usually means I'm "thinking" while typing...), I like your "Project", as I find it "useful", and a perfect Use of iMacros, or even a perfect Usecase for iMacros...

So if you are not already working on your Script yourself and still want to "get it done", my "Creative Idea" would "misuse" a little bit your Project for some little "Project" of mine, but that would get you your Script for say, half the Price you had in mind, and you would even get "the Best of all Worlds", ah-ah...!
=> You can send me a Mail @ [My_Pseudo_on_the_Forum]@gmail.com and I can explain the Details...
Hum, and you can start yourself in your Mail with what Price you had in mind and wording your Req's (with concrete Examples and URL(s)) for your Project of course...

- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...

Ah OK, I notice you opened a Thread in the 'Consultants' Sub-Forum about 1/2h after my previous Post, so you are not going to try to make the Script yourself, and I guess you are not really interested by my "Creative Idea", ah-ah...! OK, fair enough...

millenialme wrote:I am looking to develop a script for extracting price data from flight comparison sites such as Google flights.

I love to travel and have spent hours in the past trying to find good deals for flights to various countries. Like a lot of people I have signed up to Jack's flight club Scotts cheap flight etc but I'd like to be able to target the places I actually want to travel to.

After working a job recently I was introduced to iMacros and saw the potential for this application. However, my very limited knowledge of writing scripts has made it very difficult to get this to work. That's why I'm looking for some help!

The intension is to run the script a few times each week to try and find the best deals for flights from UK airports to various countries around the world.
An example would involve running a search for:

(You could have posted a mini-Update and that Link yourself, @OP, minus-Point for me to get "motivated" for your Project... )

Hum..., no URL(s) mentioned from which Site(s) you usually use and want to extract your Data from... (I had asked for those...!)
iMacros is very dependent on the HTML Structure of a Site/Page, it makes a "huge" Difference if you were using some Site able to check several Air Companies, or if you were checking several Air Companies one by one from their own Web-Site... (Every Web-Site needs a separate Script...)

The Req's are not very "precise"... (Sorry, was me trying to be "diplomatic" a wee bit, I meant "completely vague", ah-ah...!)
=> Duration of your Stay not mentioned...?, Return Flight as well...?, + within the 6 weeks Date-span mentioned or are the Dates only for Departure...?, Input Destination as a fixed List of 1 or a few specific Countries/Cities or for all Countries/Cities in the World...?
You should provide as well the fixed List of Departure Airports in your Order of Preference... etc...

Advice of the Day:
Every 5 min you don't spend on defining your Req's precisely will cost you at least $100 with any Professional Developer, in back&fro Communication trying to clarify, or they will "take stg for granted" and will implement it how they "interpreted" your vague Formulation which in 80% of the cases won't be what you expected, Developer spent a few hours implementing the 1st Version, not good, need a 2nd Version, Developer a bit pissed off because of wasted time, not paid, Motivation goes down, and you'll end up with some "half-Product"...!

- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...

The break down of dates for flights would be:
Long weekends (Friday to Monday)
1 week trips
2 weeks trip

e.g. an example search may involve:Search 1(NAVIGATE TO) Departing (INPUT) LHR(NAVIGATE TO) Arriving (INPUT) LAX(NAVIGATE TO) Departing date (INPUT) 31/05/18(NAVIGATE TO) Return date (INPUT) 14/05/18
Then to extract data for the price of the cheapest flight and possibly also the airline an time of flight.

This information would then be used to collate the best deals for a number of destinations. E.g. three 'trip duration' variations would run for each, long weekend, 1 week and 2 weeks. 5 different destinations would be checked during 1 session. For example one week I may focus on North America and want prices for LA, Florida, NYC, Seattle and Boston. These searches would be made for a certain time window, e.g. various dates between May 31st and June 31st.