[PATCH] exec: allow core_pipe recursion check to look for a value of 1 rather than 0

Hey all- So, about 6 months ago, I made a set of changes to how thecore-dump-to-a-pipe feature in the kernel works. We had reports of severalraces, including some reports of apps bypassing our recursion check so that aprocess that was forked as part of a core_pattern setup could infinitely crashand refork until the system crashed.

We fixes those by improving our recursion checks. The new checkbasically refuses to fork a process if its core limit is zero, which works well.

Unfortunately, I've been getting grief from maintainer of user spaceprograms that are inserted as the forked process of core_pattern. They contendthat in order for their programs (such as abrt and apport) to work, all therunning processes in a system must have their core limits set to a non-zerovalue, to which I say 'yes'. I did this by design, and think thats the rightway to do things.

But I've been asked to ease this burden on user space enough times thatI thought I would take a look at it. The first suggestion was to make therecursion check fail on a non-zero 'special' number, like one. That way thecore collector process could set its core size ulimit to 1, and enable thekernel's recursion detection. This isn't a bad idea on the surface, but I don'tlike it since its opt-in, in that if a program like abrt or apport has a bug andfails to set such a core limit, we're left with a recursively crashing systemagain.

So I've come up with this. What I've done is modify thecall_usermodehelper api such that an extra parameter is added, a functionpointer which will be called by the user helper task, after it forks, but beforeit exec's the required process. This will give the caller the opportunity toget a call back in the processes context, allowing it to do whatever it needs toto the process in the kernel prior to exec-ing the user space code. In the caseof do_coredump, this callback is ues to set the core ulimit of the helperprocess to 1. This elimnates the opt-in problem that I had above, as it allowsthe ulimit for core sizes to be set to the value of 1, which is what therecursion check looks for in do_coredump.

This patch has been tested successfully by some of the Abrt maintainersin fedora, with good results. Patch applies to the latest -mm tree as of thisAM.

+/*+ * This is used as a helper to set up the task that execs+ * our user space core collector application+ * Its called in the context of the task thats going to + * exec itself to be the helper, so we can modify current here+ */+void core_pipe_setup(void)+{+ task_lock(current->group_leader);+ current->signal->rlim[RLIMIT_CORE].rlim_cur = 1;+ task_unlock(current->group_leader);+}