Here is a sample of the output, It almost instantly uses 2GB and then starts
using VMem. This is probably the right suggestion but it's another thing to
install
> It's probably also worth being aware of guppy's heapy stuff:
http://guppy-pe.sourceforge.net/heapy_tutorial....<http://guppy-pe.sourceforge.net/heapy_tutorial.html>I
find it quite nice to have the following to get a quick point-in-time
estimate of my app's memory usage:
print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024))
19 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_nm_rep1.txt
3_veg_nm_rep1.txt has 228484 rows of data
3_veg_nm_rep1.txt has 0 rows of masked data
3_veg_nm_rep1.txt has 141 rows of outliers
3_veg_nm_rep1.txt has 0 modified rows of data
280bytes of memory used for 3_veg_nm_rep1.txt
20 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_nm_rep2.txt
3_veg_nm_rep2.txt has 228484 rows of data
3_veg_nm_rep2.txt has 0 rows of masked data
3_veg_nm_rep2.txt has 119 rows of outliers
3_veg_nm_rep2.txt has 0 modified rows of data
280bytes of memory used for 3_veg_nm_rep2.txt
21 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_phd_rep1.txt
3_veg_phd_rep1.txt has 228484 rows of data
3_veg_phd_rep1.txt has 0 rows of masked data
3_veg_phd_rep1.txt has 63 rows of outliers
3_veg_phd_rep1.txt has 0 modified rows of data
280bytes of memory used for 3_veg_phd_rep1.txt
22 files of 43 using 6424 bytes of memory
Loading into memory: 3g_c285-11.txt
3g_c285-11.txt has 228484 rows of data
3g_c285-11.txt has 0 rows of masked data
3g_c285-11.txt has 65 rows of outliers
3g_c285-11.txt has 0 modified rows of data
280bytes of memory used for 3g_c285-11.txt
23 files of 43 using 6424 bytes of memory
Loading into memory: 3g_c285-42.txt
3g_c285-42.txt has 228484 rows of data
3g_c285-42.txt has 0 rows of masked data
3g_c285-42.txt has 27 rows of outliers
3g_c285-42.txt has 0 modified rows of data
280bytes of memory used for 3g_c285-42.txt
24 files of 43 using 6424 bytes of memory
Loading into memory: A6AF.txt
A6AF.txt has 228484 rows of data
A6AF.txt has 0 rows of masked data
A6AF.txt has 36 rows of outliers
A6AF.txt has 0 modified rows of data
280bytes of memory used for A6AF.txt
25 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep1.txt
Grigg_3026_rep1.txt has 228484 rows of data
Grigg_3026_rep1.txt has 0 rows of masked data
Grigg_3026_rep1.txt has 949 rows of outliers
Grigg_3026_rep1.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep1.txt
26 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep2.txt
Grigg_3026_rep2.txt has 228484 rows of data
Grigg_3026_rep2.txt has 0 rows of masked data
Grigg_3026_rep2.txt has 361 rows of outliers
Grigg_3026_rep2.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep2.txt
27 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep3_both.txt
Grigg_3026_rep3_both.txt has 228484 rows of data
Grigg_3026_rep3_both.txt has 0 rows of masked data
Grigg_3026_rep3_both.txt has 41 rows of outliers
Grigg_3026_rep3_both.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep3_both.txt
28 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3131_rep1.txt
Grigg_3131_rep1.txt has 228484 rows of data
Grigg_3131_rep1.txt has 0 rows of masked data
Grigg_3131_rep1.txt has 537 rows of outliers
Grigg_3131_rep1.txt has 0 modified rows of data
280bytes of memory used for Grigg_3131_rep1.txt
29 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3131_rep2.txt
Grigg_3131_rep2.txt has 228484 rows of data
Grigg_3131_rep2.txt has 0 rows of masked data
Grigg_3131_rep2.txt has 238 rows of outliers
Grigg_3131_rep2.txt has 0 modified rows of data
280bytes of memory used for Grigg_3131_rep2.txt
30 files of 43 using 6424 bytes of memory
*Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>
On Sat, Feb 20, 2010 at 7:40 PM, ssteinerX at gmail.com <ssteinerx at gmail.com>wrote:
>> On Feb 20, 2010, at 9:21 PM, Vincent Davis wrote:
>> See this article for some more info about the reported sizes of things:
>>http://www.doughellmann.com/PyMOTW/sys/limits.html>>> <http://www.doughellmann.com/PyMOTW/sys/limits.html>Nice article but I
> must have missed something useful to my current issue. Do I get any hints?
>>> Oh, sorry, there was the part about getsizeof() not including attributes
> unless the class supplies a __sizeof__ method and a comment at the bottom:
>> It's probably also worth being aware of guppy's heapy stuff:
>http://guppy-pe.sourceforge.net/heapy_tutorial....<http://guppy-pe.sourceforge.net/heapy_tutorial.html>
>> I find it quite nice to have the following to get a quick point-in-time
> estimate of my app's memory usage:
>> print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024))
>> Put a few of those in at various places in the collection process and you
> should see where you're taking a substantial hit on each pass.
>> If I remember those numbers right, 5mb != 2.5gb so something along the way
> is doing something very strange.
>> S
>>-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20100220/a0850b01/attachment.html>