I need to write numbers into a file upto 50mb and it should be fast can any one help me how to do that?i had written the following code..-----------------------------------------------------------------------------------------------------------def create_file_numbers_old(filename, size): start = time.clock()

print "time taken to write a file of size", size, " is ", (end -start), "seconds \n"------------------------------------------------------------------------------------------------------------------it takes about 34sec i need 5 to 10 times less than that.

You should also use timeit instead. Although i admit, i use the same sort of method to quickly aquire time stats of a general idea.

The more methods you call, the slower its going to be. Writing to disc is always going to be slow to a certain extent. While your code on my pc ran at 24 seconds ...For example this code snippets' code outcome is the same as it creates a 50MB file of ints, and on my pc runs between 2.0 and 2.4 seconds to write. But that is because i use write() once, and not constantly checking the size, nor checking the position. It depends on what you need to do.

And the final file size is 55 MB. So as you can see... the writing of the file was by far the fastest thing we did. The thing that took time was creating the data. This is where you need to optimize. As such we can't tell you how to go faster without seeing what your method of generating your data is.

VenugopalaRao wrote:thanks mek but i am getting the following memory errorFile "D:/work/pythoncourse/project_02.py", line 46, in create_file_numbers_new data = "\n".join(str(i) for i in xrange(size))+"\n"MemoryError

that is this not working for size = 50*1024*1024producing a memory error

Figure out a size that you can fit in memory and do it in chunks.

Seriously I have no clue why you would possibly need to do this. Computers are great at generating numbers; why save over 52 million consecutive ones in a file:

Lol I understand what you are doing... the argument you are passing is NOT the data size of the file to generate. It is the upper value of the integers to generate.

If you pass 50*1024*1024, using my function, you are asking for a file that prints all integers up to 52428800 which is almost a 500 megabyte file.

@Metul... the file time only starts to blow up once I use large numbers which is proportional to my processing speed. Try increasing the number drastically and see if they stay around even. It shouldn't stay even if you make them large enough unless something very strange is happening. You may just be using a much more powerful computer than mine.

def write_all_the_numbers(filename,lim,n=None): """lim is the max number to print up to (non-inclusive). n should be chosen to be as large as possible while still fitting in memory. If lim can already fit in memory do not pass a third argument.""" if n: gener=(xrange(i,i+n) if i+n<lim else xrange(i,lim) for i in xrange(0,lim,n)) else: gener = (xrange(lim),) with open(filename,'a') as myfile: for chunk_range in gener: data = "\n".join(str(num) for num in chunk_range)+"\n" myfile.write(data)

Note the above code WILL create a file with 52428800 numbers. It runs in 20 seconds on my machine. If you only want a 50 megabyte file lower the number to what Metul originally suggested of 6388889.

Edit: @Metul: When booting in linux mint I produced speeds similar to yours. I have no clue what retardation windows is doing to add 3 seconds on to the time.@OP: if you are running under windows, it seems that "\n" adds two bytes, not one. Try taking this into account.

On linux and in other worlds of sanity it adds 1 byte so your previous code might have worked.You might want to confirm with your instructor which he wants. The above should give you as close to 50 megs as you can get with the current criteria as I understand it. -Mek