Resources for the Check Point Community, by the Check Point Community.

Tim Hall has done it again! He has just released the 2nd edition of "Max Power".Rather than get into details here, I urge you to check out this announcement post. It's a massive upgrade, and well worth checking out. -E

If this is your first visit, be sure to
check out the FAQ by clicking the
link above. You may have to register
before you can post: click the register link above to proceed. To start viewing messages,
select the forum that you want to visit from the selection below.

fwm export - File size limit exceeded

SM225 running R77.30

Our logs rollover at midnight every day. Typical log file contains 6-7 million records and a size of approx 1.3 GB. Having a need to export multiple days of logs into a .csv format, I looked to using "fwm export". Tracker works fine for this but takes a long time to complete. I was hoping fwm export would be speedier and it seems to be. But there is this:

Re: fwm export - File size limit exceeded

I'll answer your second question first. Check Point's native log format records some fields like IP address in binary rather than text. An IP address in binary is four bytes. The same address as text would be up to 15 bytes in ASCII or potentially even more in various Unicode encodings. Yes, it is expected for logs exported as plain text to be significantly larger than the logs in the original binary format.

As for the file size limitation, what OS are you running on the box (check with 'uname -a')? What is the output of the command 'mount'? File size limitations most often come from the filesystem used on the drive. It's possible the fwm process doesn't like dealing with files larger than 2 GB, so you may have to use ordinary output redirection.

Peripherally related, I recommend against delimiting the fields with a comma. Certain fields can contain a comma, which makes automated processing more difficult. I generally export logs like this:

Code:

fwm logexport -s -z -n -p -i ./<file>.log > <file>.ffsv

-s sets the field delimiter to be ASCII character 0xff, which can never occur inside a log field.

Re: fwm export - File size limit exceeded

Originally Posted by Bob_Zimmerman

I'll answer your second question first. Check Point's native log format records some fields like IP address in binary rather than text. An IP address in binary is four bytes. The same address as text would be up to 15 bytes in ASCII or potentially even more in various Unicode encodings. Yes, it is expected for logs exported as plain text to be significantly larger than the logs in the original binary format.

As for the file size limitation, what OS are you running on the box (check with 'uname -a')? What is the output of the command 'mount'? File size limitations most often come from the filesystem used on the drive. It's possible the fwm process doesn't like dealing with files larger than 2 GB, so you may have to use ordinary output redirection.

Peripherally related, I recommend against delimiting the fields with a comma. Certain fields can contain a comma, which makes automated processing more difficult. I generally export logs like this:

Code:

fwm logexport -s -z -n -p -i ./<file>.log > <file>.ffsv

-s sets the field delimiter to be ASCII character 0xff, which can never occur inside a log field.

Thank you very much!

I was using CP_R77_CLI_ReferenceGuide for command structure, they do not note the -z or -s switches, this is nice to know. I will be trying your string shortly and reporting back. My requested outputs:

Re: fwm export - File size limit exceeded

x86_64 in the uname output indicates it's running in 64-bit mode, and vg_splat-lv_current indicates GAiA. Files bigger than 2 GB aren't an issue on ext3. This must be a limitation of the fwm binary file output functionality. Output redirection with > should work.

The -z switch just sets it to ignore non-fatal errors and keep exporting log data. This can cause corrupt entries in the output, but those can be handled after the fact.

Re: fwm export - File size limit exceeded

Originally Posted by Bob_Zimmerman

x86_64 in the uname output indicates it's running in 64-bit mode, and vg_splat-lv_current indicates GAiA. Files bigger than 2 GB aren't an issue on ext3. This must be a limitation of the fwm binary file output functionality. Output redirection with > should work.

The -z switch just sets it to ignore non-fatal errors and keep exporting log data. This can cause corrupt entries in the output, but those can be handled after the fact.

Well, "other duties as assigned" prevented my testing yesterday. I did so today and have encouraging results. Thank you for the great insights Zimmie.

I now have an Excel error stating not enough memory and to consider using 64-bit version of Excel, but that is a different story I will leave to our helpdesk.......

Re: fwm export - File size limit exceeded

Rather than importing into Excel, it may be worth processing the file with PowerShell first. For example, I use this to trim a file down to just the columns I care about, then process down to unique lines:

You should be able to chain them together without a round-trip through filtered.csv. I use intermediate files to be sure I only need to rerun a few shorter steps if something goes wrong.

Import-Csv has a -Delimiter option, but I haven't tried it with 0xff-separated files before. It may understand quote-delimited, comma-separated files, so you may not need to use the -s switch in the export. I have only started using PowerShell for this kind of processing relatively recently. Before, I used awk or Perl.