Folks,
I would like to test the integrity of a large dataset, for example by
finding the max value of some data field. The dataset is too large to be
fully loaded into memory, so I would like to read it slice by slice and
compute max in each slice, and then combine them together. To that end I
wrote the following script:
import yt, yt.funcs
import sys
import numpy as np
f = sys.argv[1]
ds = yt.load(f)
n = ds.domain_dimensions[0]
dmax = np.zeros((n,))
for i in range(n):
d = ds.r[i:(i+1),:,:][('artio', 'HVAR_GAS_DENSITY')]
dmax[i] = np.amax(d)
print(i,dmax[i],yt.funcs.get_memory_usage())
del d
print(np.amax(dmax))
However, when I run it, I get:
yt : [INFO ] 2018-08-28 10:52:58,075 Created 4945 chunks for ARTIO
0 16945762.0 202.5546875
yt : [INFO ] 2018-08-28 10:52:59,388 Created 1232 chunks for ARTIO
1 2416576.25 221.36328125
yt : [INFO ] 2018-08-28 10:53:01,599 Created 9635 chunks for ARTIO
2 10419311.0 269.06640625
...
yt : [INFO ] 2018-08-28 10:53:23,397 Created 1474 chunks for ARTIO
13 2395590.5 698.91015625
yt : [INFO ] 2018-08-28 10:53:25,594 Created 9747 chunks for ARTIO
14 11139424.0 739.16015625
The number of chunks created varies in each iteration of the loop, but
the total memory usage is still steadily climbing up.
Could you advise what I am doing wrong?
Many thanks,
Nick

Hello! I'm using the sph-viz branch of yt and I'm trying to make a density
projection plot from a gadget-2 snapshot. This works like a charm when I
use a snapshot with 428^3 particles in a 15 Mpc box. However, when I try to
do it on a snapshot with 1024^3 particles in a 40 Mpc box, my job keeps
getting killed because it is trying to exceed the available system memory.
I'm submitting it to a node with 256 GB of RAM, and I'm requesting all of
it, which means that more than this is trying to be used. The snapshot
itself is ~ 40 GB. The memory usage appears to spike when allocating for
the KDTree. Before that, it appears to generate the .ewah file without
issue. I was just wondering if anyone had any thoughts on why this large
spike in memory might be occurring and how I might go about fixing it? If
not, no worries, I can try and investigate it myself, but it's a pain to do
with pdb. My gadget simulation used a maximum of ~ 173 GB of RAM, so I feel
like the kdtree shouldn't be using as much memory as it's trying to.
Thanks, and sorry for the bother!
Sincerely,
-Jared

Dear yt-users,
I am having some trouble with the yt clump finder. Mostly, my issue is that
I don't understand what I am getting when I annotate the projectionplot.
Instead of the nice lines in the example, I get strange filled-in circles?
My figure is attached, and here is my code:
##################
ds =
yt.load("/mnt/ceph/users/stonnesen/runs/JO204wind/DD"+loop[i]+"/JO204cw"+loop[i])
ad = ds.all_data()
data_source = ad.cut_region(["obj['z'] > 0.27"]) #ds.disk([0.5, 0.5,
0.5], [0., 0., 1.],(80, 'kpc'), (145, 'kpc'))
# the field to be used for contouring
field = ("gas", "density")
# This is the multiplicative interval between contours.
step = 2.0
# Now we set some sane min/max values between which we want to find
contours
# This is how we tell the clump finder what to look for -- it won't
look for
# contours connected below or above these threshold values.
c_min = 2e-25 #10**np.floor(np.log10(data_source[field]).min()+3 )
c_max = 10**np.floor(np.log10(data_source[field]).max()+1)
# Now find get our 'base' clump -- this one just covers the whole
domain.
master_clump = Clump(data_source, 'density')
# Add a "validator" to weed out clumps with less than 16 cells.
# As many validators can be added as you want.
master_clump.add_validator("min_cells", 16)
# Calculate center of mass for all clumps.
master_clump.add_info_item("center_of_mass")
# Begin clump finding.
find_clumps(master_clump, c_min, c_max, step)
# Save the clump tree as a reloadable dataset
fn = master_clump.save_as_dataset(fields=["density"])
# Traverse the clump hierarchy to get a list of all of the 'leaf' clumps
leaf_clumps = get_lowest_clumps(master_clump)
# To visualize these clumps, a list of clumps can be supplied to
# the "clumps" callback on a plot. First, we create a projection plot:
prj = yt.ProjectionPlot(ds, 0, 'density',
center=[0.5,0.5,0.5],width=(160,'kpc'))
prj.set_buff_size(4000)
# Next we annotate the plot with contours on the borders of the clumps
prj.annotate_clumps(leaf_clumps)
# Save the plot to disk.
prj.save('JO204cw'+loop[i]+'_clumps_16pix')
##########################
Also, when I have
> print (leaf_clumps[0]["gas","density"])
I get 22 outputs, which seems like a lot less than what is getting put on
the figure?
Anyway, I am lost and any help would be greatly appreciated.
Best,
Stephanie
--
Dr. Stephanie Tonnesen
Associate Research Scientist
CCA, Flatiron Institute
New York, NY
stonnes(a)gmail.com

Hi yt-users!
So I have a question about set_buff_size. I would like my pixels to be the
same size as my more refined cell in a projectionplot, so have a couple
lines of code like this:
plot6z=yt.ProjectionPlot(ds,"x","density",center=[0.5,0.5,0.33],width=(76,'kpc'))
plot6z.set_buff_size(2000)
plot6z.save('JO201scc_76kpc_'+loop[i]+'t'+times[i])
When I take a look at the png file that is made, there are 800 pixels
across instead of the 2000 that I expect. I just checked the docs here:
http://yt-project.org/doc/visualizing/plots.html
and based on the SlicePlot example I feel like it should work! Then again,
it looks like in the example the number of pixels also doesn't change?
I would appreciate any/all advice!
Thanks!!
Stephanie
--
Dr. Stephanie Tonnesen
Associate Research Scientist
CCA, Flatiron Institute
New York, NY
stonnes(a)gmail.com

Hi everyone,
ytree version 2.2 is released!
ytree is an extension of yt for working with merger-tree data from
multiple sources.
This release adds a couple new features as well as improvements for
i/o and memory usage.
- New feature: vector fields return x/y/z components of a vector as a
single array with automatically created magnitude fields.
- New feature: a select_halos function to search all trees for halos
matching specified criteria.
- Enhancement: significant refactors of the field and i/o systems to
reduce memory, speed up field access, and simplify implementing new
frontends.
- Enhancement: upgrades to the testing infrastructure.
ytree can be installed with pip:
pip install ytree
or from source, by going to https://github.com/brittonsmith/ytree
git clone https://github.com/brittonsmith/ytree
cd ytree
pip install -e .
Full documentation can be found at http://ytree.readthedocs.io/.
As always, ytree would be nothing without the wonderful yt community.
Thanks to all those amazing people!
Britton

Folks,
I need to make a small change in the ARTIO backend - I added an option
to compress grid data, and that required a small change in the meaning
of some disk variables. In order to test these changes, I made them
inside yt/frontends/artio/artio_headers on my local anaconda install,
but I do not know how to compile them into yt - sorry for a such a dummy
question. Any advice?
Nick Gnedin

Hi,
I'm wondering if there's some way to format the way a YTQuantity prints,
for example the number of decimal places. Simply using print() leads to
many decimal places and includes the units. It's nice to have the units
included automatically, but would also be good to have control over the
number of decimal places.
Actually, from looking at the source I see how the __str__ method for
YTArray and YTQuantity objects is defined, which would indicate that no
such formatting is possible, as built in -- though I suppose one could use
the value and units parts to do the outputting. It might be nice to have a
simple method built in like say valunit(x) which would return a tuple of
the value and unit so then you could do
print('{:.4f} {}'.format(x.valunit()))
Jon
--
________________________________________________________
Jonathan D. Slavin Harvard-Smithsonian CfA
jslavin(a)cfa.harvard.edu 60 Garden Street, MS 83
phone: (617) 496-7981 Cambridge, MA 02138-1516
cell: (781) 363-0035 USA
________________________________________________________