I was under the impression that the maximum length of a single argument was not the problem here so much as the total size of the overall argument array plus the size of the environment, which is limited to ARG_MAX. Thus I thought that something like the following would succeed:

When the minus one is removed, the error returns. Seemingly the maximum for a single argument is actually ARG_MAX/16 and the -1 accounts for the null byte placed at the end of the string in the argument array.

Another issue is that when the argument is repeated, the total size of the argument array can be closer to ARG_MAX, but still not quite there:

3 Answers
3

Answers

Definitely not a bug.

The parameter which defines the maximum size for one argument is MAX_ARG_STRLEN. There is no documentation for this parameter other than the comments in binfmts.h:

/*
* These are the maximum length and maximum number of strings passed to the
* execve() system call. MAX_ARG_STRLEN is essentially random but serves to
* prevent the kernel from being unduly impacted by misaddressed pointers.
* MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
*/
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

As is shown, Linux also has a (very large) limit on the number of arguments to a command.

A limit on the size of a single argument (which differs from the overall limit on arguments plus environment) does appear to be specific to Linux. This article gives a detailed comparison of ARG_MAX and equivalents on Unix like systems. MAX_ARG_STRLEN is discussed for Linux, but there is no mention of any equivalent on any other systems.

The above article also states that MAX_ARG_STRLEN was introduced in Linux 2.6.23, along with a number of other changes relating to command argument maximums (discussed below). The log/diff for the commit can be found here.

It is still not clear what accounts for the additional discrepancy between the result of getconf ARG_MAX and the actual maximum possible size of arguments plus environment. Stephane Chazelas' related answer, suggests that part of the space is accounted for by pointers to each of the argument/environment strings. However, my own investigation suggests that these pointers are not created early in the execve system call when it may still return a E2BIG error to the calling process (although pointers to each argv string are certainly created later).

Also, the strings are contiguous in memory as far as I can see, so no memory gaps due do alignment here. Although is very likely to be a factor within whatever does use up the extra memory. Understanding what uses the extra space requires a more detailed knowledge of how the kernel allocates memory (which is useful knowledge to have, so I will investigate and update later).

ARG_MAX Confusion

Since the Linux 2.6.23 (as result of this commit), there have been changes to the way that command argument maximums are handled which makes Linux differ from other Unix-like systems. In addition to adding MAX_ARG_STRLEN and MAX_ARG_STRINGS, the result of getconf ARG_MAX now depends on the stack size and may be different from ARG_MAX in limits.h.

Normally the result of getconf ARG_MAX will be 1/4 of the stack size. Consider the following in bash using ulimit to get the stack size:

However, the above behaviour was changed slightly by this commit (added in Linux 2.6.25-rc4~121).
ARG_MAX in limits.h now serves as a hard lower bound on the result of getconf ARG_MAX. If the stack size is set such that 1/4 of the stack size is less than ARG_MAX in limits.h, then the limits.h value will be used:

Note also that if the stack size set lower than the minimum possible ARG_MAX, then the size of the stack (RLIMIT_STACK) becomes the upper limit of argument/environment size before E2BIG is returned (although getconf ARG_MAX will still show the value in limits.h).

A final thing to note is that if the kernel is built without CONFIG_MMU (support for memory management hardware), then the checking of ARG_MAX is disabled, so the limit does not apply. Although MAX_ARG_STRLEN and MAX_ARG_STRINGS still apply.

This is a good answer, certainly better than mine - I upvoted it. But the answer we ask for isnt always the answer we should get - thats why we're asking, because we dont know. It doesnt address the problem with your work flow that brought you head to head with this issue in the first place. I demonstrate how that might be mitigated in my own answer, and how single shell variable string arguments over 2mbs in length can be passed to newly execed processes with just a couple lines of shell script.
–
mikeservMar 21 '14 at 19:57

That ARG_MAX is the minimum guaranteed for the arg+env size limit, it's not the max size of a single argument (though it happens to be the same value as MAX_ARG_STRLEN)
–
Stéphane ChazelasMar 20 '14 at 20:48

Do you have a date for your eglibc-2.18/NEWS snippet? It would be good to pin this down to a particular kernel version.
–
GraemeMar 20 '14 at 21:06

@StephaneChazelas: I'm just too lazy to find the part, but if arg exceeds the max value it isn't necessary to figure out the env size.
–
user55518Mar 20 '14 at 21:06

@Graeme: I have also some older linuxes running where the getconf value shows 131072. I think this belongs to newer linuxes with eglibc > ?? only. Congrats, you found a bug BTW.
–
user55518Mar 20 '14 at 21:11

2

You're looking at glibc code, that's irrelevant here. The libc doesn't care what size of arguments you're passing. The code you're quoting is about sysconf, an API to gives users an idea of the maximum size (whatever that means) of argv+env passed to an execve(2). It's the kernel that accepts or not the arg and env list passed along an execve() system call. The getconf ARG_MAX is about the cumulative size of arg+env (variable in recent Linux, see ulimit -s and the other question I linked), it's not about the max length of a single arg for which there's no sysconf/getconf query.
–
Stéphane ChazelasMar 20 '14 at 21:24

So @StephaneChazelas rightly corrects me in the comments below - the shell itself does not dictate in any way the maximum argument size permitted by your system, but rather it's set by your kernel.

As several others have already said, it seems the kernel limits to 128kb the maximum argument size you can hand to a new process from any other when first execing it. You experience this problem specifically due to the many nested $(command substitution) subshells that must execute in place and hand the entirety of their output from one to the next.

And this one's kind of a wild guess, but as the ~5kb discrepancy seems so close to the standard system page size, my suspicion is that it is dedicated to the page bash uses to handle the subshell your $(command substitution) requires to ultimately deliver its output and/or the function stack it employs in associating your array table with your data. I can only assume neither comes free.

I demonstrate below that, while it might be a little tricky, it is possible to pass very large shell variable values off to new processes at invocation, so long as you can manage to stream it.

In order to do so, I primarily used pipes. But I also evaluated the shell array in a here-document pointed at cat's stdin. Results below.

But one last note - if you've no particular need for portable code, it strikes me that mapfile might simplify your shell jobs a little.

No, nothing to do with the shell, it's the execve(2) system call returning E2BIG when a single argument is over 128kiB.
–
Stéphane ChazelasMar 20 '14 at 20:49

Consider also that there is not limit on shell builtins - echo $(tr -dc [:alnum:] </dev/urandom | head -c $(($(getconf ARG_MAX)*10))) >/dev/null will run fine. It is only when you use an external command that there is a problem.
–
GraemeMar 20 '14 at 20:54

@Graeme Well, I did this with cat as well - no problem. The variable is evaluated in a heredoc at the end. See my last edit. I did cut down the total count to 33 because I'm adding in the last value each time. And the zero padding...
–
mikeservMar 20 '14 at 21:23

@StephaneChazelas - so am I getting around that by evaluating the argument in a heredoc stream? Or is bash compressing it somehow?
–
mikeservMar 20 '14 at 21:55

1

@mikeserv, I can't see anywhere in you're code any instance of you executing a command with a large arg list. printf is a builtin so is not executed, and AFAICT, your cat is not given any argument.
–
Stéphane ChazelasMar 20 '14 at 22:11