Exercises, examples and other material relating to training module P667. This topic is presented on public course Perl for Larger Projects

If you've so much data that it won't all fit into memory all at once, you may not be able to use conventional programming techniques to complete your task. We define a data set such as this as "huge data"; it's impossible to handle in some languages, but very practical in Perl. This module doesn't introduce many new language features; instead, it shows you how to use what you already know to handle huge data practically.

How to interact with a Perl program while it is processing dataIf you have a long running program, how do you monitor its progress?
You could use tail -f on an output file ... but there are other options too.
a) You can output a progress line; there's an example of this from the recent Perl for Larget Projects course ... [here].
b) You can trap ^C ... and even ...

Speeding up your Perl codeOn Friday morning - our Perl for Larger Project course - I was looking at coding efficiency / run speed with delegates. As an example, we took a data file from our web server logs - some 23 Mbytes of data, comprising about 121,000 lines, of which 1099 contained the word "melksham" in lower case.
The ...

Teaching examples in Perl - third and final partThree part article ... this is part 3. Jump back to part [1] [2]
Following on from two earlier posts, here is the final third of the new examples that I wrote during last week's Perl course, and to which I have added extra documentation over the last couple of days.
P212 More on Character Strings
"Does ...

How are you getting on?Have you ever asked someone to do something for you ... a long task, and you would like a progress report? "How are you getting on?" you'll ask ... and they'll give you an update - "I'm 75% of the way through" they'll say or - perhaps even more helpfully - "I'm nearly there, and I have some good results ...

Macho matching - do not do it!There's something vaguely macho about doing a grand regular expression match to do all your filtering in a single line of code - but being macho may be less than efficient. It may be far better to do two shorter matches, with the first quickly rejecting records which don't need to be handled in detail, ...

Long job - progress bar techniques (Perl)Here's a "Perl for Larger Projects" example --- for use in illustrating the "advanced file and directory handling" and "handling huge data set" modules.
Scenario ... I want to go through all the files and directories on a big drive, and find the largest file(s). It will take a while, so I want progress ...

Progress Bar Techniques - PerlHave you ever sat there and wondered "is this program nearly done ... is it still running ... how is it getting on" and wished you had a progress bar. But then have you ever watched a jerky progress bar and felt that it's more fiction than fact?
We were discussing these aspects on today's private Perl ...

Preventing ^C stopping / killing a program - PerlHere's a demonstration - in Perl - that shows you how to avoid a ^C (Control C) dropping you straight out of a program.
Have you ever accidentally hit ^C in the wrong window and terminated a long-running process just before it finished ... well, by setting $SIG{INT} to the address of a sub you want ...

Perl - progress bar, supressing ^C and coping with huge data flowsIf you're handling a huge amount of data (gigabytes!) in a Perl program, memory won't allow you to slurp it all into a list and you'll traverse the data with a loop from file or from database. And because of the sheer volume of data, it may take a while to process. During such proessing, you may wish ...

Answering ALL the delegate's Perl questionsDuring courses, questions arise. "I'll get back to that" could make people feel that I'm brushing something off ... except that I explain, early on, that some questions require a great deal of background knowledge to be answered sensibly. And I keep a list of topics that I'll be getting back to ...

Huge data files - what happened earlier?When I'm programming a log file analysis in Perl, I'll often "slurp" the whole file into a list which I can then traverse efficiently as many times as I need. If I need to look backwards from some interesting event to see what happened in the immediate lead up to it, I can do so simply by looking at ...

Progress bars and other dynamic reportsIf you've got a program that runs for a long time, your users will wish to be kept informed of progress and how much longer there is to go. Now that's not always easy to predict (and I'm sure that most of you have made fun of such forecasts in the past) but its's much much much better than sitting ...