Scraping products from Walmart with PHP, Guzzle, Crawler and Doctrine

You know web scraping is a useful technique of data extraction from websites, especially if there is no any API or there is a ton of data that can’t be got in another way. The most simple and well-known way is to use CURL. But it is easy only in the aspect of using third-party components and widespread because of its universality. Our goal is to scrape data mostly simply for a programmer.
We are going to do this in 4 steps with Guzzle, Symfony DOM Crawler Component and Doctrine DBAL packages. As for an example of the site this one http://www.walmart.com will be taken.
We want to get all categories and goods from its catalogue and receive a CSV file with the goods at the end.

1. Install libraries

The easiest way to install all libraries with their requirements is to do this with Composer. We consider you have it installed.

1.1. Guzzle

The first thing we need for scraping is an HTTP Client. We choose Guzzle.
So let’s open command line and start with the command:

As you can see one additional package was installed: symfony/event-dispatcher. It doesn’t need your special attention but it is a required dependency of Guzzle.

1.2. Symfony DOM Crawler Component

Also we need something that will help us to scrape necessary data quickly and easy. Here we’ll use Symfony DOM Crawler Component.
Write the next command into the command line:

MS DOS

1

composerrequiresymfony/dom-crawler

That’s how it will looks like if everything is OK:

MS DOS

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

C:\OpenServer\domains\walmart.loc>composerrequiresymfony/css-selector

Usingversion^3.0forsymfony/dom-crawler

./composer.jsonhasbeenupdated

Loadingcomposerrepositorieswithpackageinformation

Updatingdependencies(includingrequire-dev)

-Installingsymfony/polyfill-mbstring(v1.1.0)

Downloading:100%

-Installingsymfony/dom-crawler(v.3..3)

Downloading:100%

symfony/dom-crawlersuggestsinstallingsymfony/css-selector()

Writinglockfile

Generatingautoloadfiles

This package downloads additional package too, but also it suggests to install symfony/css-selector. It is very useful while scraping so we’ll accept this suggestion and write the following command:

MS DOS

1

composerrequiresymfony/css-selector

You can see an example of the successful installation process below:

MS DOS

1

2

3

4

5

6

7

8

9

10

11

C:\OpenServer\domains\walmart.loc>composerrequiresymfony/dom-crawler

Usingversion^3.0forsymfony/dom-crawler

./composer.jsonhasbeenupdated

Loadingcomposerrepositorieswithpackageinformation

Updatingdependencies(includingrequire-dev)

-Installingsymfony/css-selector(v3.0.1)

Downloading:100%

Writinglockfile

Generatingautoloadfiles

1.3. Doctrine

After getting data we’ll need to save it to the DB. That’s why some database abstraction layer will come in handy, for example, Doctrine.
The way of its installation is the same as at the previous step:

MS DOS

1

composerrequiredoctrine/dbal:~2.5.4

Here is a screenshot:

MS DOS

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

C:\OpenServer\domains\walmart.loc>composerrequiredoctrine/dbal:~254

Usingversion^3.0forsymfony/dom-crawler

./composer.jsonhasbeenupdated

Loadingcomposerrepositorieswithpackageinformation

Updatingdependencies(includingrequire-dev)

-Installingdoctrine/lexer(v1.0.1)

Loadingfromcache

-Installingdoctrine/annotations(v1.2.7)

Loadingfromcache

-Installingdoctrine/cache(v1.6.0)

Downloading:100%

-Installingdoctrine/inflector(v1.1.0)

Downloading:100%

-Installingdoctrine/common(v2.6.1)

Downloading:100%

-Installingdoctrine/dbal(v2.5.4)

Downloading:100%

doctrine/dbalsuggestsinstallingsymfony/console

(ForhelpfulconsolecommandssuchasSQLexecutionandimportoffiles.)

Writinglockfile

Generatingautoloadfiles

At the end of this part let’s look to our project’s directory:

Here we see that all libraries are in the vendor directory and also composer.json and composer.lock files were created. So we can do the next step.

2. Load HTML code of the website page

We create file, for example, scraper.php in the root of project’s directory. Firstly, we need to include file vendor/autoload.php to make everything work:

PHP

1

2

3

<?php

require__DIR__.'/vendor/autoload.php';

?>

2.1. Include Guzzle

Now we are ready to say that Guzzle client will be used. Also we need to think about some exceptions that may be thrown by this client.

1

2

useGuzzle\Http\Client;

useGuzzle\Http\Exception\ClientErrorResponseException;

2.2. Create request

First of all let’s define some variables: URL of the site and URI of its page we want to scrape. Look at the http://www.walmart.com. In the menu there is a link to the page with all departments http://www.walmart.com/all-departments.
It is the best page for scraping categories, so we’ll use just it.

PHP

1

2

$url='http://www.walmart.com';

$uri='/all-departments';

Also it is necessary to define User-Agent header.
If not, the following error will be produced:

Let’s copy data from the browser. We will use Chrome default user-agent header.

PHP

1

2

3

4

5

$userAgent='Mozilla/5.0 (Windows NT 10.0)'

.' AppleWebKit/537.36 (KHTML, like Gecko)'

.' Chrome/48.0.2564.97'

.' Safari/537.36';

$headers=array('User-Agent'=>$userAgent);

To make a request we need to create object of the HTTP Client and use its method get().

PHP

1

2

$client=newClient($url);

$request=$client->get($uri,$headers);

2.3. Get response

When request is made we can get response from the http://www.wallmart.com. We will warp the request code block in try/catch to properly handle connection issues. Option true is required because we don’t want to echo page at the browser, we want to get it as a string.

PHP

1

2

3

4

5

6

7

8

try{

$response=$request->send();

$body=$response->getBody(true);

}catch(ClientErrorResponseException$e){

$responseBody=$e->getResponse()

->getBody(true);

echo$responseBody;

}

Now we have what to scrape so let’s move to the next part.

3. Scrape the page

3.1. Include Crawler

At the top of the script we’ll say we are going to use DOM Crawler Component and CSS Selector.

PHP

1

2

useSymfony\Component\DomCrawler\Crawler;

useSymfony\Component\CssSelector;

3.2. Get HTML block with categories

Now let’s continue our try block. To scrape the received page’s body we should create an object of Crawler and put there the body variable.
It is time to see what part of page we have to extract to get the categories’ titles. We inspect it using Chrome Dev Tools and understand that we need the first div class='all-depts-links' (see screenshot below). It will be our filter.

Now we extract every child node of this div and push it to the array within the method each() and an anonymous function. After that we don’t need this Crawler object anymore so we’ll remove it.

PHP

1

2

3

4

5

6

7

8

$crawler=newCrawler($body);

$filter='.all-depts-links';

$catsHTML=$crawler

->filter($filter)

->each(function(Crawler$node){

return$node->html();

});

unset($crawler);

If we dump this array we’ll see 13 elements which are strings with the HTML inside.
Looking at it we get to know that we have three-level tree of categories: categories, their subcategories and sub-subcategories.

3.3 Get categories titles and subcategories HTML

Categories’ titles are located in the headings, so CSS selector for them is '.all-depts-links-heading > a'. Each subcategory locates in the separate li node, that’s why filter for them is 'ul'.
To get only titles from the categories’ headings is possibly using method text(). To get subcategories’ HTML code we’ll apply method html().

PHP

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

$cats=$subCatsHTML=array();

$catsFilter='.all-depts-links-heading > a';

$subCatsFilter='ul';

foreach($catsHTMLas$index=>$catHTML){

$crawler=newCrawler($catHTML);

$cats[]=$crawler

->filter($catsFilter)

->text();

$subCatsHTML[$index]=array();

$subCatsHTML[$index]=$crawler

->filter($subCatsFilter)

->each(function(Crawler$node){

return$node->html();

});

unset($crawler);

}

And now we have array with the categories title’s and the HTML code of subcategories.

3.4. Get subcategories’ and their sub-subcategories’ data

At this step we should receive an array with the whole data inside: categories titles’ and subcategories at the first level, subcategories’ data and their sub-subcategories at the second level, and sub-subcategories data at the third level.

From the screenshot above we know that CSS filter for the subcategories is 'li > a.all-depts-links-dept', and for the sub-subcategories it is 'li > a.all-depts-links-category'.

Let’s think about data that we need to get from subcategories. Of course, the first field is a title. But also as we are going to get products from this subcategory, we need to know how to get to its page. So the second field will be a link, namely an URI for the subcategory’s page. The same we need for sub-subcategories.
The code for this section will look like this:

PHP

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

$subSubCats=array();

$subCatsFilter='li > a.all-depts-links-dept';

$subSubCatsFilter='li > a.all-depts-links-category';

foreach($subCatsHTMLas$catIndex=>$subCatHTML){

$subSubCats[$catIndex]['cat']=$cats[$catIndex];

foreach($subCatHTMLas$subCatIndex=>$subSubCatHTML){

$crawler=newCrawler($subSubCatHTML);

$node=$crawler->filter($subCatsFilter);

$tempSubCat=array(

'href'=>$node->attr('href'),

'title'=>$node->text()

);

$tempSubSubCats=$crawler

->filter($subSubCatsFilter)

->each(function(Crawler$node){

returnarray(

'href'=>$node->attr('href'),

'title'=>$node->text()

);

});

unset($crawler);

$subSubCats[$catIndex]['subCats'][$subCatIndex]=array(

'subCat'=>$tempSubCat,

'subSubCats'=>$tempSubSubCats

);

}

}

3.5. Get goods

In the Walmart catalogue some of the subcategories don’t have sub-subcategories. That’s why our task is to get goods:

from the subcategories without sub-subcategories;

from the sub-subcategories.

All goods are located inside the list with classed 'tile-list tile-list-grid'. Every item has its own ul
and all item’s data is inside child node of li – div.
Being based on these conclusions we can form a goods filter: 'ul.tile-list.tile-list-grid > li > div'.
One more thing before we dive into the code.
While writing and testing this script we noticed that Guzzle was redirected from some links to others. But HTTP client couldn’t do this automatically, so it threw an exception with a such message: “Was unable to parse malformed url: http://url/to/what/it/was/redirected”. But we caught this exception, got the URL from it and went ahead with the new URL.

PHP

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

$errorMessage='Was unable to parse malformed url: ';

$errorLength=strlen($errorMessage);

$goodsFilter='ul.tile-list.tile-list-grid > li > div';

foreach($subSubCatsas$catIndex=>$cats){

if(!empty($cats['subCats'])){

foreach($cats['subCats']as$subCatIndex=>$subCat){

if(empty($subCat['subSubCats'])){

$uri=$subCat['subCat']['href'];

$goodsHTML='';

$continue=true;

while(empty($goodsHTML)&&$continue){

if(strpos($uri,'http')===false){

$uri=$url.$uri;

}

$request=$client->get($uri,$headers,$options);

try{

$response=$request->send();

$goodsHTML=$response->getBody(true);

$crawler=newCrawler($goodsHTML);

$subSubCats[$catIndex]['subCats']

[$subCatIndex]['goods']=array();

$subSubCats[$catIndex]['subCats']

[$subCatIndex]['goods']=$crawler

->filter($goodsFilter)

->each(function(Crawler$node){

$html=$node->html();

returngetGoodsData(

$html,$node

);

});

unset($crawler);

}catch(Exception$e){

$message=$e->getMessage();

if(strpos($message,$errorMessage)===false){

$continue=false;

}else{

$uri=substr($message,$errorLength);

}

}

}

}else{

foreach(

$subCat['subSubCats']as$subSubCatIndex=>$subSubCat

){

$uri=$subSubCat['href'];

$goodsHTML='';

$continue=true;

while(empty($goodsHTML)&&$continue){

if(strpos($uri,'http')===false){

$uri=$url.$uri;

}

$request=$client->get($uri,$headers,$options);

try{

$response=$request->send();

$goodsHTML=$response->getBody(true);

$crawler=newCrawler($goodsHTML);

$subSubCats[$catIndex]['subCats']

[$subCatIndex]['subSubCats']

[$subSubCatIndex]['goods']=array();

$subSubCats[$catIndex]['subCats']

[$subCatIndex]['subSubCats']

[$subSubCatIndex]['goods']=

$crawler

->filter($goodsFilter)

->each(function(Crawler$node){

$html=$node->html();

returngetGoodsData(

$html,$node

);

});

unset($crawler);

}catch(Exception$e){

$message=$e->getMessage();

if(strpos($message,$errorMessage)===false){

$continue=false;

}else{

$uri=substr($message,$errorLength);

}

}

}

}

}

}

}

}

In the code above we used earlier undefined function getGoodsData($html, $node). Before we write it we need to know what data we want to get from goods. Looking at some sub-subcategory’s page (see screenshot below) we see that goods have such main attributes. But sometimes some fields may be absent: at the screenshots below there is no price or there are no rating and reviews.

We should prevent problems associated with these cases so it is obvious to set default values for these variables. Also it is necessary to check if these spans are present in the received HTML code.
Also let’s prepare filters for the data:

Now we can write the function.

PHP

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

functiongetGoodsData($html,Crawler$node)

{

$price=$rating=.0;

$reviews=0;

$image=$title='';

$priceSpan='<span class="price price-display">';

$priceFilter='span.price.price-display';

$ratingSpan='<span class="visuallyhidden">';

$ratingFilter='span.visuallyhidden';

$reviewsSpan='<span class="stars-reviews">';

$reviewsFilter='span.stars-reviews';

$imageFilter='a > img';

$titleFilter='h3.tile-heading';

if(!(strpos($html,$priceSpan)===false)){

$price=$node

->filter($priceFilter)

->text();

$price=trim($price);

$price=substr($price,1);

$price=str_replace(',','',$price);

$price=(float)$price;

}

if(!(strpos($html,$ratingSpan)===false)){

$rating=$node

->filter($ratingFilter)

->text();

$rating=(float)$rating;

}

if(!(strpos($html,$reviewsSpan)===false)){

$reviews=$node

->filter($reviewsFilter)

->text();

$reviews=trim($reviews);

$reviews=str_replace('(','',$reviews);

$reviews=str_replace(')','',$reviews);

$reviews=(int)$reviews;

}

$image=$node

->filter($imageFilter)

->attr('data-default-image');

$title=$node

->filter($titleFilter)

->text();

$title=trim($title);

returncompact('image','price','title','rating','reviews');

}

4. Push data into DB

4.0. Create DB

Let’s create a database called 'walmart'. We can do this easy within PHPMyAdmin or MySQL console. Here is an SQL script:

4.4. Insert goods into DB

At last we get to the final part.
Here we are going to walk through the array and insert every category, subcategory, sub-subcategory and goods item. Notice that we need to create variables for subcategory and sub-subcategory indices, and increment them at every iteration, because their indices in the array get to zero at every level.

PHP

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

$sql=array();

$sql['cat']='INSERT INTO `categories` '

.'(`category_id`, `category_title`)'

.'VALUES (?, ?);';

$sql['subCat']='INSERT INTO `subcategories` '

.'(`subcategory_id`, `category_id`, '

.'`subcategory_title`, `subcategory_href`) '

.'VALUES (?, ?, ?, ?)';

$sql['subSubCat']='INSERT INTO `subsubcategories` '

.'(`subsubcategory_id`, `subcategory_id`, '

.'`subsubcategory_title`, `subsubcategory_href`) '

.'VALUES (?, ?, ?, ?)';

$sql['goods']='INSERT INTO `goods` '

.'(`subcategory_id`, `subsubcategory_id`, '

.'`goods_image`, `goods_price`, `goods_title`, '

.'`goods_rating`, `goods_reviews`) '

.'VALUES (?, ?, ?, ?, ?, ?, ?)';

$subCatID=$subSubCatID=0;

foreach($subSubCatsas$catIndex=>$cat){

$conn->executeQuery($sql['cat'],array($catIndex+1,$cat['cat']));

foreach($cat['subCats']as$subCatIndex=&gt;$subCat){

$subCatID++;

$conn->executeQuery($sql['subCat'],array($subCatID,$catIndex+1,

$subCat['subCat']['title'],$subCat['subCat']['href']

));

if(!empty($subCat['subSubCats'])){

foreach(

$subCat['subSubCats']as$subSubCatIndex=&gt;$subSubCat

){

$subSubCatID++;

$conn->executeQuery($sql['subSubCat'],array($subSubCatID,

$subCatID,$subSubCat['title'],$subSubCat['href']

));

if(!empty($subSubCat['goods'])){

foreach($subSubCat['goods']as$item){

$conn->executeQuery($sql['goods'],array(null,

$subSubCatID,$item['image'],

$item['price'],$item['title'],

$item['rating'],$item['reviews']

));

}

}

}

}

if(!empty($subCat['goods'])){

foreach($subCat['goods']as$item){

$conn->executeQuery($sql['goods'],array($subCatID,null,

$item['image'],$item['price'],$item['title'],

$item['rating'],$item['reviews']

));

}

}

}

}

Now we have all goods from the first pages of every category located in our database. Let’s export table ‘goods’ to CSV file. Here we have 20 rows from this file:

2. Load HTML code of the website page
We create file, for example, scraper.php in the root of project’s directory. Firstly, we need to include file vendor/autoload.php to make everything work:
——————–
I am using symfony 4…. This does not seem right where do you put this file exactly? Because it seems to me it should go in the src folder?

thanks for commenting! This tutorial has been created with the thought, that you don’t use any MVC frameworks for the task. I.e. only guzzle+doctrine/dbal+symfony dom crawler (which is just a component of symfony framework). You definitely can use sf4 to create a similar scraper, but in this case I’d do it as a console symfony command (read more here: https://symfony.com/doc/current/console.html ). Note that the framework does all the autoloading tasks for you, so in case you prefer to go with sf console command, you just need to specify the namespaces used, no need to do any require/include’s.