Laravel import large CSV file

I recently had to import a 30,000 line CSV file, in this post, I document how I tackled this task.

large files can be read fairly easily with PHP the issue is when you start inserting the data into a database, even worse if you're first checking if the record already exists. In that case, say you're working with a 100 lined CSV file your end up doing 200 queries. You can't avoid needing to do the queries but doing lots in one go will use up your application's memory and will timeout when taking too long.

It's easier to break up a large file into smaller files and process them in batches.

Let's go through the steps achieved.

First 2 routes are required one for loading an import view and another to process the import.

In this case, we don't need to touch the construct method, move to the handle method. This will run when the command is executed

This opens the files inside the pendingcontacts folder we use array_slice(globa($path),0,2) to only open 2 files at a time. Then extract a line into an array then using Laravel's updateOrCreate method to update a record if the email matches otherwise import a new contact.

Now in order to run this command, we need to tell Artisan where to find it. We do this by registering it in a service provider I'll use the default AppServiceProvider.php.

Inside a boot method:

$this->commands([
\App\Console\Commands\ImportContacts::class
]);

Optionally you may want to use the scheduler to run this command at a given frequency. To do this create an instance of the schedule class then run the command pass in the signature and frequency.
In this example the import:contacts command will be called every minute.

That's it! now everytime you import a CSV file it will be broken into smaller CSV files added to the pendingcontacts folder and your command will run every minute processing 2 files at a time (you can increase the number).

It would be useful to inform the user how many contacts are left to be imported. This can be done in the import method of the controller:

<?php
namespace App\Http\Controllers;
use Illuminate\Http\Request;
class ContactsController extends Controller
{
public function import()
{
//setup an empty array
$records = [];
//path where the csv files are stored
$path = base_path('resources/pendingcontacts');
//loop over each file
foreach (glob($path.'/*.csv') as $file) {
//open the file and add the total number of lines to the records array
$file = new \SplFileObject($file, 'r');
$file->seek(PHP_INT_MAX);
$records[] = $file->key();
}
//now sum all the array keys together to get the total
$toImport = array_sum($records);
return view('import', compact('toImport'));
}

Pass $toImport to the view then in the view show the number as long as there is at least 1:

For the past 10 years, I’ve been developing applications for the web using mostly PHP. I do this for a living and love what I do as every day there is something new and exciting to learn.

In my spare time, the web development community is a big part of my life. Whether managing online programming groups and blogs or attending a conference, I find keeping involved helps me stay up to date. This is also my chance to give back to the community that helped me get started, a place I am proud to be part of.

Besides programming I love spending time with friends and family and can often be found together going out catching the latest movie, staying in playing games on the sofa or planning a trip to someplace I’ve never been before.