A Java program that writes each digit on a new line, from 1 to max integer. We should end up with 2.4 Billion lines. After looping to near the maximum integer, the text file becomes really huge. Like 55gb after running for 16 minutes. How can I optimize it to be faster, and also reduce the file size?

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

4

\$\begingroup\$We can't change the file size without changing what the code does. But you haven't specified what the code must achieve. So, how can we possibly help you?\$\endgroup\$
– 200_successMar 19 at 4:13

\$\begingroup\$@200_success I'm just writing a program that writes new digit on each line, max line of the max integer. I hope that clarifies the question.\$\endgroup\$
– king amadaMar 19 at 4:36

1

\$\begingroup\$@kingamada A) Your code doesn't go up to Integer.MAX_VALUE, but "only" Integer.MAX_VALUE - 2. B) You are multiplying by 2, so half way through you are overflowing the integer. C) Finally why do you think a 55 GB file is too big? I roughly estimate that size is to be expected.\$\endgroup\$
– RoToRaMar 19 at 8:29

1

\$\begingroup\$You can't expect a certain output and want the output to be smaller. That's not how it works.\$\endgroup\$
– MastMar 19 at 12:08

1 Answer
1

I have no idea why you need this text file. Making it faster should be possible though.

Your current code allocates a StringBuilder and a String for every line. You can avoid this by allocating a single StringBuilder and modifying it using append and setSize. This will reduce the time spent in the garbage collector.

Printing an integer involves splitting it into digits, and that involves integer division, one of the slowest opcodes a processor provides. In your case you can avoid integer division by defining a class called StrInt that stores the integer as a character array and as the number itself. You would need to define methods like add(n) and appendTo(StringBuilder) and take care of the wraparound into the negative numbers.

Using these two techniques will make your code much longer (especially the unit test for the StrInt class should be large) and probably also quite fast.

Writing 55 GB in 1000 seconds means the write rate is already 55 MB/s, which is not the worst.

You should also write to a byte stream instead of a character stream to avoid the unnecessary conversion. Simply changing the encoding from UTF-8 to US-ASCII may already speed it up measurably.

\$\begingroup\$On my system, using a StringBuilder in the manner described in your first paragraph only slows the program down. I am a little surprised how efficient plain old string addition is in this context. Your second idea is interesting, though I have not implemented it.\$\endgroup\$
– Benjamin KuykendallMar 19 at 6:54