sort runs out of memory

I'm using a pipe including sort to merge multiple large textfiles and remove dupes. I don't have root permissions but the box isn't configured in any way to cut non root privileges further down than default debian jessie. The box has 32GB RAM and 16GB are in use. Regardless on how I call sort (GNU sort 8.13) it fills up all the remaining RAM and crashes with "out of memory". It really fills up all the memory before crashing. I followed the process in top.I tried to explicitly set the max memory usage with the -S parameter ranging from 80% to 10% and from 8G to 500M. The whole pipe looks similar to:

Always the same behavior. Does anyone know what could cause such issue? And of course how to solve it?

Best How To :

I found the issue myself. It's fairly easy. The "tr -cd '[:print:]'" removes line breaks and sort reads line by line. So it tries to read all the files as one line and the -S parameter can't do its job.

Q 1. why is ch empty even after fread() assignment? (Most probably) because fread() failed. See the detailed answer below. Q 2.Is this a portability issue between Solaris and Linux? No, there is a possible issue with your code itself, which is correctly reported by valgrind. I cannot quite...

You're not getting the results you want because you're not assigning the sorted user_infos back into the user_infos variable. You can do the following: user_infos = user_infos.sort {|a, b| - (a['can_go'] <=> b['can_go']) } # -or- user_infos.sort! {|a, b| - (a['can_go'] <=> b['can_go']) } The first version of sort creates...

I found the discussion in Valgrind mail list when someone had the same problem. The issue was that the kernel have been patched with PaX patches, one of which doesn't allow to look at the /proc/pid/maps. The quote about the patch from wikipedia The second and third classes of attacks...

as you see, in Edit1, you (make) try to run JavaScriptCore-4.0.gir instead of compile it with g-ir-compiler; I tried on my pc and my command is: cd /home/davide/src/webkitgtk-2.8.3/build/Source/JavaScriptCore && \ /usr/bin/g-ir-compiler /home/davide/src/webkitgtk-2.8.3/build/JavaScriptCore-4.0.gir \ -o /home/davide/src/webkitgtk-2.8.3/build/JavaScriptCore-4.0.typelib as a workaround, you cand edit: build/Source/JavaScriptCore/CMakeFiles/JavascriptCore-4-gir.dir/build here's the lines on my file (the last...

If you look at your inserstion sort As you already put count =1 because as for exits on exit condition of for loop. for same reason then it also make sense that when while loop cancels the count++ inside will not get executed but there was a comparison made. but...

The -v option to grep inverts the search, reporting only the lines that don't match the pattern. Since you know how to use grep to find the lines to be deleted, using grep -v and the same pattern will give you all the lines to be kept. You can write...

A workaround is to modify the sudoers file and remove the requirement of a password from your user ID for a particular script to have sudo privileges. Enter sudo visudo After this, add the details in the following manner. username ALL=(ALL) NOPASSWD: /path/to/script Another method would be to pipe the...

The only error that I see is this>current_generation_.end() instead that ->. In addition you should consider declaring your compare fuction as accepting two const FPGA* instead that just FPGA*. This will force you to declare fitness() as const int fitness() const but it makes sense to have it const. Mind...

The right way to do iterative code in R is to avoid explicit for loops. Use apply (and the company) instead. @jeremycg gave you the right R-ish answer. Regarding your code, you should make some editing to make it work. temp <- c() for (i in 1:length(data)){ temp[names(data)[i]] <- sum(is.na(data[i]))...

Your radixSort function violates the Compare requirements, namely irreflexivity (that is, radixOrder(x, x) must return false but it returns true because the execution goes to the first if branch). So you get a classic example of undefined behavior here. I believe that piece of code should be rewritten somehow like...

The problem is, you don't have debug info for the ptr type, so gdb treats it as integer. You can examine its real contents using: (gdb) x/a &ptr 0x600124 <ptr>: 0x7fffffffe950 (gdb) p/a $rsp $3 = 0x7fffffffe950 Of course I have a different value for rsp than you, but you...

As indicated in the comments, you need to provide "something" to your while loop. The while construct is written in a way that will execute with a condition; if a file is given, it will proceed until the read exhausts. #!/bin/bash file=Sheetone.txt while IFS= read -r line do echo sh...

The ret_from_syscall symbol will be in architecture-specific assembly code (it does not exist for all architectures). I would look in arch/XXX/kernel/entry.S. It's not actually a function. It is part of the assembly code that handles the transition from user-space into kernel-space for a system call. It's simply a label to...

This matches your input/output examples, but I had to use descending numbers to get the example answers. Are you sure your explanation is correct? If not, just use 0123456789 instead of 9876543210 in the code below. The algorithm is to provide a sorting key based on translating the digits of...

If you want the None and '' values to appear last, you can have your key function return a tuple, so the list is sorted by the natural order of that tuple. The tuple has the form (is_none, is_empty, value); this way, the tuple for a None value will be...

Unlike Windows, Linux does not have an implementation of "threads" in the kernel. The kernel gives us what are sometimes called "lightweight processes", which are a generalization of the concepts of "processes" and "threads", and can be used to implement either. It may be confusing when you read kernel code...

I took at look at your live site and injected the sorting function you used in your question. I noticed a few things. Firstly, the strings you are passing into your compare function look like this: "£38.89 ex. VAT" "£19.93 ex. VAT" "£44.44 ex. VAT" ... So parseInt("£38.89 ex. VAT")...

Here's a sed version: /^Host_Alias/{ # whenever we match Host_Alias at line start : /\\$/{N;b} # if backslash, append next line and repeat s/$/,host25/ # add the new host to end of line } If you need to add your new host to just one of the host aliases, adjust...

It looks like you're missing zlib; you'll want to install it: apt-get install zlib1g-dev I also suggest reading over the README and confirming you have all other dependencies met: https://github.com/dccmx/mysqldb/blob/master/README Also, I suggest using mysqlclient over MySQLdb as its a fork of MySQLdb and what Django recommends....

You can achieve that by using the env utility: timeout 10 /usr/bin/env LD_LIBRARY_PATH=/path/to/mod/libc/ cp a b Env will set the environment variable and exec the other utility with that environment....

You don't need the -s flag to determine the uptime. If you do something like this you have the time the server is running: $tmp = explode(' ', exec('uptime')); $uptime = $tmp[2]; // something like 2:14 (hh:mm) nb: an alternative would be to use the who -b command, which will...

Try this: find . -mmin +35 -or -mmin -25 find supports several logical operators (-and, -or, -not). See the OPERATORS section of the man pages for more details. ==================== EDIT: In response to the question about processing the two matches differently, I do not know of a way to do...

The output from set -x uses single quotes. So the outer double quotes were replaced with single quotes but you can't escape single quotes inside a single quoted string so when it then replaced the inner double quotes it needed, instead, to replace them with '\'' which ends the single...

wget expect can be tricky to work with so I'd prefer to use GNU Wget as an alternative. The following should work as long as you don’t have any spaces in any of the arguments. for v in "${files_to_download[@]}" do ftp_file="${v}.bz2" wget --user=${USER} --password=${PASSWD} ${HOST}/${ftp_file} done Request multiple resources using...

InputStream input = client.getInputStream(); BufferedReader in = new BufferedReader(new InputStreamReader(input)); Your problem is here. You can't use multiple inputs on a socket when one or more of them is buffered. The buffered input stream/reader will read-ahead and 'steal' data from the other stream. You need to change your protocol so...

.* is greedy: it matches all possible characters. This way, even sed 's/<?php.*//' file will also delete all the content in your file. To prevent this greediness of .*, say "everything but a ?" -> [^?]*: sed 's/<?php[^?]*?><?php[^?]*?>//' file Test $ cat a <?php echo 'first' ?><?php echo 'second' ?><?php...

You just have to check if it starts with "price". Note that I don't think that ToString() is appropriate; you should rather implement IComparer<T> and strongly type your objects in your listbox. public int Compare(object x, object y) { // test for equality if (x.ToString() == y.ToString()) { return 0;...

Try this to create a string variable n, with no leading whitespace (thanks @011c): n="10.0.0.135.527" wget http://infamvn:8081/nexus/content/groups/LDM_REPO_LIN64/com/infa/com.infa.products.ldm.ingestion.server.scala/"$n"-SNAPSHOT/com.infa.products.ldm.ingestion.server.scala-"$n"-20150622.210643-1-sources.jar ...

As PM 77-1 suggests, consider using the built–in Array.prototype.sort with Date objects. Presumably you want to sort them on one of start or end: jobs.sort(function(a, b) { return new Date(a.ys, a.ms-1) - new Date(b.ys, b.ms-1); }) ...