ImageMagick

Questions and postings pertaining to the usage of ImageMagick regardless of the interface. This includes the command-line utilities, as well as the C and C++ APIs. Usage questions are like "How do I use ImageMagick to create drop shadows?".

The problem is that you don't have enough memory for the pixel caches and other stuff. When you limit the memory used for pixel caches, you don't then hit problems when allocating memory for other structures.

You might try "-ping" as the first option. This may prevent IM from reading pixels.

I've been trying to read the documentation, and am I right in saying that ImageMagick will use MAGICK_MEMORY_LIMIT first, for the pixel cache, and if that's not enough, it moves on to MAGICK_MAP_LIMIT?

If so, I should be able to set MAGICK_MEMORY_LIMIT and MAGICK_MAP_LIMIT to "1GiB", to give a total of "2GiB"... and for this to work on a system with 4GB of RAM, and 4GB of SWAP (unfortunately it doesn't work).

As an aside, and while it's not good practice, I can set `MAGICK_TEMPORARY_PATH=/run/shm`, and process the image using the temporary filesystem (tmpfs) that resides in RAM.

ImageMagick attempts to allocate the pixel cache in memory since it has the lowest latency. The allocation might fail if the allocation exceeds any set memory limits or if the system request fails, e.g. the request exceeds the available free memory on the system. If the memory allocation fails, ImageMagick allocates the pixel cache on disk and then attempts to memory-map it which can be 5x faster when reading than direct disk access. The allocation might fail if the allocation exceeds any memory map limits or if the system request fails, e.g. the request exceeds the available memory map pages on the system. If memory-mapping fails, ImageMagick attempts to allocate the pixel cache on disk. This allocation might fail if the allocation exceeds any set disk limits or if the system call fails, e.g. you don't have permission to write on disk. The pixel cache on disk could fail while writing if the available free disk is exceeded. The pixel cache on disk has lazy allocation, meaning you might be well into writing before the system reports there is not enough space. To ensure there is enough disk space before processing begins on an image, set the MAGICK_SYNCHRONIZE environment variable which is ensures the entire file can fit in the disk partition before any processing begins.

You have 4 GB of RAM. But is that 4 GB the total, or the amount available for IM?

In total.

The server typically uses about 90MB of RAM, and when running with 1GiB for both MAGICK_MEMORY_LIMIT and MAGICK_MAP_LIMIT, I would assume it's hitting those limits, then move processing to disk... but it crashes instead.

If I set MAGICK_MAP_LIMIT to 768MiB, then it does successfully start using the disk.

But I would like to increase those limits, to use as much of the RAM as possible (i.e. slightly less than) 4GB of RAM... that said, even using some of the 4GB of SWAP would probably help.

To ensure there is enough disk space before processing begins on an image, set the MAGICK_SYNCHRONIZE environment variable which is ensures the entire file can fit in the disk partition before any processing begins.

Hi magick, thanks for the background.

I've tried `export MAGICK_SYNCHRONIZE="true"`, and I can't see any difference (it still fails):

Where I've run this `identify` command a few times (it's the same with `convert`) - and `free` does not get above 1.3GB of "used" memory.

And I should note that during `convert`, with MAGICK_MAP_LIMIT set to 768MiB (i.e. so it works), I do see the temporary files get created in /tmp/, so file permissions aren't an issue, nor is disk space (61GB free).

And if I set MAGICK_TEMPORARY_PATH="/run/shm/", it works faster (as it's RAM based), and does not have memory limit issues.

The pixel cache is consuming all the available memory so even some small allocation requests are refused by the system. The problem is likely that your TIFF image drops into the generic TIFF read method that requires width x height memory. Try the latest release of ImageMagick. It has been optimized, in most cases, to only use memory the size of a strip or tile. ImageMagick has control over how the pixel cache is allocated but does not have control over how much memory a delegate library consumes.

Unfortunately I'm not in a position to compile a new version of ImageMagick at the moment.

I did try using your suggested command, and it results in the same error.

The only way I can get `identify` or `convert` to work is by setting `-limit map 768MiB`, anything larger results in the "Cannot allocate memory" error (even when `free` reports that these commands never uses more than 1.3GB).

Don't worry about it, it might be something else going on, or it could have been fixed already; so I'll keep my current work around, and hope for the best in future.