Discussions and Tech Support related to website data extraction, screen scraping and data mining using iMacros.

Forum rulesBefore asking a question or reporting an issue:
1. Please review the list of FAQ's.
2. Use the search box (at the top of each forum page) to see if a similar problem or question has already been addressed.
3. Try searching the iMacros Wiki - it contains the complete iMacros reference as well as plenty of samples and tutorials.
4. We can respond much faster to your posts if you include the following information:CLICK HERE FOR IMPORTANT INFORMATION TO INCLUDE IN YOUR POST

Our ISP provides an online web site, where you can download a CSV of usage data.
I am attempting to write a tool that extracts this data for each of our customers only a regular basis, and injects it into a database so they can easily retrieve a graphical representation

There are two views of data - summary and detailed.
When I login to the website manually, the process is URL for login screen -> TAG enter details -> click A 'Detailed' -> click A 'Download Usage' -> click BTN 'Download'
The CSV that downloads shows detailed usage data.

I have built this behaviour into iMacros, using recording and some manual adjustments for additional waiting.
But the downloaded CSV is always only the 'summary' data - not the detailed.

I have confirmed by watching the script that iMacros does trigger the click on the A tag 'Detailed', which in theory is what 'switches' the behaviour of the Download button to download detailed data.
But the downloaded CSV is only ever summary view. I can even perform the same behaviour side-by-side, in a Chrome window manually on the right and iMacros 12 built-in browser on the left, and the output from iMacros is summary whereas the output from Chrome is detailed. Interestingly though - when I navigate the page using the iMacros built-in browser manually, it correctly registers that I've clicked on 'Detailed' view and downloads the full detailed CSV.

The website is very AJAX-heavy, to load in controls and data (hence having to build in plenty of additional WAIT commands).
My thinking is that the built-in iMacros browser is not handling some form of local cache/cookie/session storage command from the website, whereas Chrome et al does.
As an example, in the Chrome developer when switching back forth between 'summary' and 'detailed' view (before downloading the CSV) this error is logged;
VM127:3 [Deprecation] 'window.webkitStorageInfo' is deprecated. Please use 'navigator.webkitTemporaryStorage' or 'navigator.webkitPersistentStorage' instead.

If I'm right, and the website is storing the 'current' view (detailed) in some kind of local storage, is there any way to replicate this behaviour in iMacros?

I've checked the 'Application' tab in Chrome developer console, and can't see any obvious setting stored in Local Session Storage or Cookies, just a SESSION_ID which doesn't help.
I've also tried to run my iMacros script in the main 3 browsers;
- IE 11 worked to load the web page, but no fields were ever filled in on the login page
- Chrome & Firefox were completely unresponsive when I click 'Play in -> ...' in the Script Editor

Only the built-in iMacros browser works, and it comes very close to what's needed by just refuses to 'remember' that it's clicked 'Detailed View' before it downloads
Note that I've tried clicking both the 'Detailed' and 'Download' buttons as both TAGs and EVENTs

I'm using iMacros 12 on Windows 10
If anyone has any suggestions relating to tricks for handling AJAX-heavy pages, ideas about how to force iMacros to 'fully register' a click on an AJAX page, or tips for getting iMacros to work with Chrome on Windows 10 I'd appreciate it

Our ISP provides an online web site, where you can download a CSV of usage data.
I am attempting to write a tool that extracts this data for each of our customers only a regular basis, and injects it into a database so they can easily retrieve a graphical representation

There are two views of data - summary and detailed.
When I login to the website manually, the process is URL for login screen -> TAG enter details -> click A 'Detailed' -> click A 'Download Usage' -> click BTN 'Download'
The CSV that downloads shows detailed usage data.

I have built this behaviour into iMacros, using recording and some manual adjustments for additional waiting.
But the downloaded CSV is always only the 'summary' data - not the detailed.

I have confirmed by watching the script that iMacros does trigger the click on the A tag 'Detailed', which in theory is what 'switches' the behaviour of the Download button to download detailed data.
But the downloaded CSV is only ever summary view. I can even perform the same behaviour side-by-side, in a Chrome window manually on the right and iMacros 12 built-in browser on the left, and the output from iMacros is summary whereas the output from Chrome is detailed. Interestingly though - when I navigate the page using the iMacros built-in browser manually, it correctly registers that I've clicked on 'Detailed' view and downloads the full detailed CSV.

The website is very AJAX-heavy, to load in controls and data (hence having to build in plenty of additional WAIT commands).
My thinking is that the built-in iMacros browser is not handling some form of local cache/cookie/session storage command from the website, whereas Chrome et al does.
As an example, in the Chrome developer when switching back forth between 'summary' and 'detailed' view (before downloading the CSV) this error is logged;

If I'm right, and the website is storing the 'current' view (detailed) in some kind of local storage, is there any way to replicate this behaviour in iMacros?

I've checked the 'Application' tab in Chrome developer console, and can't see any obvious setting stored in Local Session Storage or Cookies, just a SESSION_ID which doesn't help.
I've also tried to run my iMacros script in the main 3 browsers;
- IE 11 worked to load the web page, but no fields were ever filled in on the login page
- Chrome & Firefox were completely unresponsive when I click 'Play in -> ...' in the Script Editor

Only the built-in iMacros browser works, and it comes very close to what's needed by just refuses to 'remember' that it's clicked 'Detailed View' before it downloads
Note that I've tried clicking both the 'Detailed' and 'Download' buttons as both TAGs and EVENTs

I'm using iMacros 12 on Windows 10
If anyone has any suggestions relating to tricks for handling AJAX-heavy pages, ideas about how to force iMacros to 'fully register' a click on an AJAX page, or tips for getting iMacros to work with Chrome on Windows 10 I'd appreciate it

Regards,
Sam

Hum, difficult to give some "educated" Advice without seeing/playing (with) the Page or seeing your Script, but I can think of a few "things"...:
With Ajaxy Pages, the 'EVENT' Mode is usually the way to go indeed, and it should work for the Click on the 'Detailed Report' Option, add maybe some mini-'WAIT' Statement if the Report needs to be populated/generated...

iMB12 has a new Editor (I haven't seen it yet), then about "Chrome & Firefox were completely unresponsive when I click 'Play in -> ...' in the Script Editor...", you can try playing your Macro directly from the iMacros Add-on for FF/CR in their corresponding Browser itself...
Current FCI's would be:
- iMacros for FF v8.9.7/v9.0.3 + FF54/55. (v8.9.7 still works in FF v55.0.3.)
- iMacros for CR v8.4.4 + CR60.

If the Report is indeed first cached by the Browser, you can try "catching" it with a 'TAG' + 'CONTENT=EVENT:SAVEITEM'.

And if the Data for your Report is presented in the Browser, you can always build your own Extraction at the 'TABLE' Level or Field by Field (with 'EXTRACT' + 'SAVEAS') instead of relying on the Download Functionality offered by the Site...

Last edited by chivracq on Sat Sep 09, 2017 1:52 pm, edited 1 time in total.

- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...

Thanks for your reply. Judging by the "last post" column on this board, you seem to be the guru around here

I had the iMacros extension for Chrome installed, but it was completely unresponsive.
But after reading your reply I tried uninstalling & reinstalling iMacros - that made the Chrome extension work.
After installing the 'Chrome File Access Setup' add-on I was able to get iMacros running under Chrome

I was required to modify the script a little bit (Tabs & Timeouts) to make it work in Chrome, but I can confirm that in Chrome the 'full' file is downloaded. In other words, the browser has correctly registered that I've clicked on 'Detailed Usage' before clicking 'Download'. So I can confirm that is a difference in behaviour between the iMacros browser and Chrome

Unfortunately, when running under Chrome the script fails to loop properly. I created a short test script that just performed;
- Access page
- Logon

That runs successfully 5 times in a row. But when I add all of the extra instructions to click 'Detailed Usage' and then 'Download Usage', after directing back to the logon page and CLEAR'ing the browser it will not re-login.
Still, that's a different issue that I can tackle separately. My initial question about the difference in behaviours has been answered. I'll try FireFox now

Thanks for your reply. Judging by the "last post" column on this board, you seem to be the guru around here

I had the iMacros extension for Chrome installed, but it was completely unresponsive.
But after reading your reply I tried uninstalling & reinstalling iMacros - that made the Chrome extension work.
After installing the 'Chrome File Access Setup' add-on I was able to get iMacros running under Chrome

I was required to modify the script a little bit (Tabs & Timeouts) to make it work in Chrome, but I can confirm that in Chrome the 'full' file is downloaded. In other words, the browser has correctly registered that I've clicked on 'Detailed Usage' before clicking 'Download'. So I can confirm that is a difference in behaviour between the iMacros browser and Chrome

Unfortunately, when running under Chrome the script fails to loop properly. I created a short test script that just performed;
- Access page
- Logon

That runs successfully 5 times in a row. But when I add all of the extra instructions to click 'Detailed Usage' and then 'Download Usage', after directing back to the logon page and CLEAR'ing the browser it will not re-login.
Still, that's a different issue that I can tackle separately. My initial question about the difference in behaviours has been answered. I'll try FireFox now

Thanks again

Regards,
Sam Miller

Boah...!, "Guru" is maybe a "big" Word, ah-ah..., but yep, I'm usually "Creative" with finding Solutions to Pb's especially if I encountered them myself, but unfortunately (for you, ah-ah...!), I don't do any Downloads in my Scripts, so I only know that Functionality from Threads on the Forum and sometimes a little bit of Testing for a specific Thread, if at least the URL and full Script are posted, which is not the case for your Thread, ah-ah...! (But I usually only use iMacros for FF anyway (v8.8.2 for my own "Prod" Env., and v8.9.7 for the Forum), I don't like other Browsers, or only when a Pb is really-very specific to a specific Browser/iMacros Version...)

But OK, semi-good News about CR and iMacros for CR (v8.4.4 then)...
Hum, about the Looping, well a Download needs a bit of time, first your Script still needs to be running before the Download gets caught and can start, but if you are looping your Script for several Downloads, most Browsers only allow 2 parallel Downloads from the same Site at the same time (might be configurable, I've never investigated, I know I've tuned my Browser (Pale Moon v26.3.3, forked on FF (=FF47)) for the Number of parallel Connections). And depending on the Site and how it's been designed, Developers LOVE Cookies nowadays, your Connection-Cookie might still be "locked" until the Download has finished, which might prevent some "too-quick" Reconnect...

But OK, any News of your Testing on FF...?
(Mention your FCI then again..., there are 2 main Versions for FF currently, which behave completely differently, advised Version (by your favorite "Guru"...!) is v8.9.7 for FF, which still works on FF v55.0.3; v9.0.3 for FF is a bit too buggy and limited, a bit useless for "serious" Users...)

- (F)CIM = (Full) Config Info Missing: iMacros + Browser + OS with all 3 Versions...
- I usually don't even read the Question if that (required) Info is not mentioned...
- Script & URL usually help a lot for a more "educated" Help...