Hello,
I've tried for a few hours here to get various forms of the US amazon scraper working. I've tried the release version. I've tried downloading from the SVN. I've tried countless searches on google, and xbmc forums and had no luck. I turned on debugging and got these results when I attempted to scrape a movie:

Very very strange.
I downloaded plex on my mac, and used it to do a scape on amazon us, and it worked. So I opened the package contents, copied the amazonus.xml file from my mac to my xbmc installation on my ubuntu machine (karmic koala 9.10), and it fails.

Very very strange.
I downloaded plex on my mac, and used it to do a scape on amazon us, and it worked. So I opened the package contents, copied the amazonus.xml file from my mac to my xbmc installation on my ubuntu machine (karmic koala 9.10), and it fails.

I am really confused now:confused2:.

Any advice?

Thanks in advance.....

Damn. I am the person who got or had got the current version working. By the way, XBMC does still include a copy of the Amazon scrapers as standard. It is/was identical to the Plex version since I sent copies to both teams.

I have just been working on it with Plex to fix some lesser problems which I had just finished, it now seems to work reasonably well with Plex. However I just tried it in XBMC on a Mac and see the same problem as you - it does not find any movies at all, not even if you edit the movie title to make it more likely to do so.

Unfortunately the part of a scraper to do with finding and processing a list of movies is the bit I find hardest to understand - most of my fixes involve getting information for an individual movie assuming the prior bit to find the movie has already worked.

By the way, the same problem seems to apply to The Amazon.co.uk scraper.

I wonder if it is something as basic as the UserAgent that XBMC is presenting to Amazon? And that Amazon is now rejecting connections from some types? It could be Plex sends a different UserAgent.

If anyone else is willing and able to look in to fixing the part to get a working list of movies, I can then provide my changes for fixing User Votes, Movie Ratings, Plot, etc.

I have done some testing and it appears that despite XBMC sending the same searchURL as before and as Plex (for Mac) does with the same scraper, it seems Amazon is returning totally a totally different format results page, so different that my scrapers fail to understand it.

I could in theory write a totally different version for XBMC but it is hard enough work supporting one set let alone two. Clearly XBMC in-conjunction with Amazon is producing different results, probably as I suggested due to the different User-Agent being used. My tests at this with a web-browser and changing the User-Agent seems to confirm this.

I tried changing the URL in the scraper to 'spoof' the User-Agent but I must be getting something wrong as it still did not work. Can a more knowledgeable person have a look and suggest the correct format?

Updated ticket as requested. I included my (failed) attempt to add a User-Agent spoof.

Note: it is not the User-Agent I tried that was wrong, it is that XBMC and Amazon are not treating it as a User-Agent. This is almost certainly due to me getting the syntax wrong. Hence my plea for help.

--------
Good news!
I have got the User-Agent over-ride working. I am now testing corrections to scrape some fields that have been broken by Amazon.

Was trying to use the above fix that was posted for XBMC for Ubuntu (The version released for Lucid) and it seems I still can't scrape anything from Amazon US. Does anyone know how to fix this, or what the problem might be? Did they change the formatting again?

UPDATE:

If I use the scraper files from this link it seems to work slightly better:

When I use those scraper files, I get results back but only very little and with simple keywords. For example, if I want to find the movie "Anna in Kung Fu Land", Amazon.com shows it just fine. In XMBC, nothing shows up. If I type in just "Kung Fu" I get a few hits like "Kung Fu: The Legend Continues", etc, but not "Anna in Kung Fu Land".

Same thing happens with movies like "Butterfly and Sword". in Amazon.com the movie shows up fine. In XBMC, nothing shows up, and if I use the simpler terms like "Butterfly" or "sword" I get a few related hits like "The Butterfly Effect" and such, but no exact hit for "Butterfly and Sword"

Perhaps if someone who has the Amazon.com scraper working and experience with scraper development could try those search terms and see what's wrong with them? Or perhaps I'm not using the latest version of the Amazon.com scraper?

Was trying to use the above fix that was posted for XBMC for Ubuntu (The version released for Lucid) and it seems I still can't scrape anything from Amazon US. Does anyone know how to fix this, or what the problem might be? Did they change the formatting again?

UPDATE:

If I use the scraper files from this link it seems to work slightly better:

When I use those scraper files, I get results back but only very little and with simple keywords. For example, if I want to find the movie "Anna in Kung Fu Land", Amazon.com shows it just fine. In XBMC, nothing shows up. If I type in just "Kung Fu" I get a few hits like "Kung Fu: The Legend Continues", etc, but not "Anna in Kung Fu Land".

Same thing happens with movies like "Butterfly and Sword". in Amazon.com the movie shows up fine. In XBMC, nothing shows up, and if I use the simpler terms like "Butterfly" or "sword" I get a few related hits like "The Butterfly Effect" and such, but no exact hit for "Butterfly and Sword"

Perhaps if someone who has the Amazon.com scraper working and experience with scraper development could try those search terms and see what's wrong with them? Or perhaps I'm not using the latest version of the Amazon.com scraper?

The version included with XBMC 9.11 is way out of date. Amazon also keep changing their html and unfortunately breaking these scrapers. I am currently quite busy myself but now I know there are issues I will try and get round to having a look and updating it (again). Either keep watching this thread, or PM me to see how things are going in a few days.

olympia Wrote:Could you please help me to understand what amazon.com gives over imdb.com?

I've found all these movie in IMDb. Is there anything special which only available on amazon.com and not IMDb?

I just try to understand to usecase.

Ah, well Amazon.com tends to have more complete information since it's a commercial website. That is, for international titles which are more obscure, they tend to document the item faster and more accurately with their own product scans and information.

For Chinese/HK movies (which is my usecase), you'll usually find cover art and information as the title brakes for international release almost immediately. Other scrapers like IMDb will usually have information as well, but that's (I think) all user submitted and there is no financial motivation for them to get cover art, movie information, etc, unless someone submits them.

olympia Wrote:Could you please help me to understand what amazon.com gives over imdb.com?

I've found all these movie in IMDb. Is there anything special which only available on amazon.com and not IMDb?

I just try to understand to usecase.

For movies etc. IMDB is indeed normally the best choice, however Amazon is a better choice for non-movie titles, e.g. DVDs of live performance Comedy shows, music concerts, documentaries, some Children's stuff, and some other esoteric DVDs. A lot of these would not count as TV shows so TheTVDB.com does not help either.

I did not get a chance to work on it this weekend, but I will try to do so soon.