I am tasked to build a program which takes input from a spss datafile and generates a script-textfile based on the datafile. The language of the script is rather esoteric (GESS) but tends to be verbose, so that the average file can easily reach 4.000 lins or more.
Since I like to use vb.net I am wondering if there is a "best way"?

Since strings are kind of memory intensive, I am wondering if it is more efficient to use stringbuilder instead of string?
Also i cannot the write the file to the disk - some elements have to go to the top of the file, wile others have to be appended to the end. So is it a problem to hold this file in memory, and what would be the best data structure?

Andy Ideas would be appreciated - I like to think this topic through before I got bogged down by a bad solution.

Does it need to run once, or often? If once, do not bother with optimization, 'programmer time is more expensive than computer time'. Also, 4000 lines, if they are 1000 characters wide, it only holds 4MB, which should be easy for any modern system.
–
Bart FriederichsOct 11 '12 at 6:56

It shall run 1 to 100 times, probably 50 would be a good average. Since it directly interacts with the customer (inhouse, but they need to accept the program) I am concerned about speed. ;)
–
Christian SauerOct 11 '12 at 7:54

You are concerned about speed, yet you talk about memory intensity. (And implemented it in VB, which is not a speed freak I reckon). Just write the program and do some benchmark tests. It is better to have a slow, functioning program than a fast program that doesn't work right.
–
Bart FriederichsOct 11 '12 at 8:09