The web-based interface (API) can respond to a variety of user requests and queries, and can be used in
addition to, or in place of, the download and local analysis of large data files. At a high level, the API
allows a user to search, extract, visualize, and analyze. In each case, the goal is to reduce the data response
size, either by extracting an unmodified subset, or by calculating a derivative quantity.

Note! IDL v8.2+ required, for the
json_parse()
function. Otherwise, you will have to parse JSON responses yourself.

Note! Matlab needs some help. In particular, the
JSONlab and
urlread2 packages are required.

API Getting Started Guide

First, start up your interface of choice and define a helper function, whose purpose is
to make a HTTP GET request to a specified URL ("endpoint"), and verify that the response is successful.
If the response type is JSON, then automatically decode
it into a dict-like object.

We see the three resolution levels of Illustris, the three dark matter only runs, and the four
subboxes per "full physics" run, as expected. Each entry has only three fields: name,
num_snapshots, and url. We can retrieve the full metadata for a particular
simulation by submitting a request to the specified url.

Let's look at Illustris-3 by determining which entry in r it is, then requesting
the url field of that entry.

Notice how we do not actually need to construct the URL by hand.

This is in general true: whenever an
API response refers to another resource or endpoint, it does so with an absolute URL, which can be
directly followed to retrieve that resource. Meaning, that there is no need to know the structure
of the API in order to navigate it.

In this case, we could have seen from the reference table at the bottom of this page, that the endpoint which
retrieves the full metadata for a given simulation is /api/{sim_name}/. Therefore, we could
manually construct the URL www.illustris-project.org/api/Illustris-3/ and send a request.
Alternatively, we can simply follow the url field that we already have to arrive at the same place.

In addition to numeric meta-data fields such as num_gas at this snapshot, we have the
url field, describing the location of this particular snapshot, which is exactly the URL
which we just requested. The simulation links back to the parent simulation which owns this
snapshot, while subhalos links deeper, to all the child subhalos which belong to this
snapshot. Finally, the files dict contains entries for all "raw file" downloads available
specifically for this snapshot. These are the same links you will find embedded in the wget
commands for snapshot 135 on the Illustris-3 Downloads
page.

Each element of results contains minimal information: the subhalo id,
its total mass (log solar units), and its unique URL. Note that, although this is the first result
of the first page, the ID may not necessarily be zero!

Note: Ordering of subhalo searches.

Return order is arbitrary unless specified.

Request the first twenty subhalos at this snapshot, sorted by descending stellar mass.

Note: Ordering

As expected, because subhalo IDs are assigned in order of descending total mass, the most
massive subhalo (with ID==0) also has the most stars. The next ID (1030) is likely a
central subhalo of a subsequent FoF halo.

The response is a combination of numeric fields and links to related objects, as well as additional data.

For example, desc_sfid = -1 and desc_snap = -1 indicate that this subhalo has no
descendant in the SubLink trees (as expected, since we are at $z=0$). On the other hand, prog_sfid = 1004
and prog_snap = 134 indicate that the main progenitor of this subhalo has ID 1004 at
snapshot 134. The related['sublink_progenitor'] link would take us directly there.

We also have id = 1030, a good sanity check. grnr = 2 indicates that this subhalo is a
member of FoF 2. primary_flag = 1 indicates that this is the central (i.e. most massive, or "primary")
subhalo of this FoF halo.

Let us directly request a group catalog field dump of the parent FoF halo.

Note: info.json endpoints

The subhalos/N/info.json and halos/N/info.json endpoints provide a raw
extraction from the group catalogs, so the fields are named accordingly.

In this case, we see that subhalo 1030 is indeed the central for this FoF 2, which has 366 total subhalos.

Let us return to the subhalo itself, and make some requests which return HDF5 data. First, extend our
helper function so that if it recieves a binary response, it saves it to a file with the appropriate
name (in the current working directory, customize as needed).

Note: Available fields in tree returns

By default, extracting the main progenitor branch, or the full tree, for a given subhalo will return all fields.
That is, in addition to telling you the Subfind IDs and snapshot numbers of all progenitors, the properties
of each progenitor subhalo will also be returned.

We see this subhalo was tracked back to snapshot 32 in the SubLink tree.
For comparison, get the main progenitor branch from the LHaloTree.

So the LHaloTree goes back to the same snapshot. Further inspection would show that in this case
the tracking is similar (same $z=0$ descendant), but not identical, which is often the case. Let's plot the evolution
the subhalo position, along each coordinate axis, back in time.

Task 2: for Illustris-1 at $z=2$, search for all subhalos with total mass
$10^{11.9} M_\odot < M < 10^{12.1} M_\odot$, print the number returned, and the Subfind IDs
of the first five results (arbitrarily ordered, you may get different ids).

>>># first convert log solar masses into group catalog units>>>mass_min=10**11.9/1e10*0.704>>>mass_max=10**12.1/1e10*0.704>>># form the search_query string by hand for once>>>search_query="?mass__gt="+str(mass_min)+"&mass__lt="+str(mass_max)>>>search_query'?mass__gt=55.9207077246&mass__lt=88.6283489903'>>># form the url and make the request>>>url="http://www.illustris-project.org/api/Illustris-1/snapshots/z=2/subhalos/"+search_query>>>subhalos=get(url)>>>subhalos['count']550>>>ids=[subhalos['results'][i]['id']foriinrange(5)]>>>ids[109974,110822,123175,107743,95711]

>>mass_min=10^11.9/1e10*0.704;>>mass_max=10^12.1/1e10*0.704;>>>>% form the search_query string by hand for once>>search_query=['?mass__gt='num2str(mass_min)'&mass__lt='num2str(mass_max)]search_query=
?mass__gt=55.9207&mass__lt=88.6283>>url=['http://www.illustris-project.org/api/Illustris-1/snapshots/z=2/subhalos/'search_query];>>subhalos=get_url(url);>>subhalos.('count')ans=550>>ids=[];>>fori=1:5,ids(i)=subhalos.('results'){i}.('id');,end>>idsids=113525525657412718

julia>

Task 3: for Illustris-1 at $z=2$, retrieve all fields for five specific Subfind IDs
(from above: 109974, 110822, 123175, 107743, 95711), print the stellar mass and
number of star particles in each.

Task 6: for Illustris-1 at $z=2$ for Subfind ID 109974, get a cutout including only the positions and
metallicities of stars, and calculate the mean stellar metallicity in solar units within the annuli
$3 \rm{kpc} < r < 5 \rm{kpc}$ (proper) centered on the fiducial subhalo position.

>>>importh5py>>>importnumpyasnp>>>>>>id=109974>>>redshift=2.0>>>params={'stars':'Coordinates,GFM_Metallicity'}>>>>>>scale_factor=1.0/(1+redshift)>>>little_h=0.704>>>solar_Z=0.0127>>>>>>url="http://www.illustris-project.org/api/Illustris-1/snapshots/z="+str(redshift)+"/subhalos/"+str(id)>>>sub=get(url)# get json response of subhalo properties>>>saved_filename=get(url+"/cutout.hdf5",params)# get and save HDF5 cutout file>>>>>>withh5py.File(saved_filename)asf:>>># NOTE! If the subhalo is near the edge of the box, you must take the periodic boundary into account! (we ignore it here)>>>dx=f['PartType4']['Coordinates'][:,0]-sub['pos_x']>>>dy=f['PartType4']['Coordinates'][:,1]-sub['pos_y']>>>dz=f['PartType4']['Coordinates'][:,2]-sub['pos_z']>>>metals=f['PartType4']['GFM_Metallicity'][:]>>>>>>rr=np.sqrt(dx**2+dy**2+dz**2)>>>rr*=scale_factor/little_h# ckpc/h -> physical kpc>>>>>>w=np.where((rr>=3.0)&(rr<5.0))>>>printnp.mean(metals[w])/solar_Z0.248392603881

Task 8: for Illustris-1 at $z=2$, for five specific Subfind IDs (from above: 109974, 110822,
123175, 107743, 95711), locate the $z=0$ descendant of each by using the API to walk down the
SubLink descendant links.

>>>ids=[109974,110822,123175,107743,95711]>>>z0_descendant_ids=[-1]*len(ids)>>>>>>fori,idinenumerate(ids):>>>start_url="http://www.illustris-project.org/api/Illustris-1/snapshots/68/subhalos/"+str(id)>>>sub=get(start_url)>>>>>>whilesub['desc_sfid']!=-1:>>># request the full subhalo details of the descendant by following the sublink URL>>>sub=get(sub['related']['sublink_descendant'])>>>ifsub['snap']==135:>>>z0_descendant_ids[i]=sub['id']>>>>>>ifz0_descendant_ids[i]>=0:>>>print'Descendant of '+str(id)+' at z=0 is '+str(z0_descendant_ids[i])>>>else:>>>print'Descendant of '+str(id)+' not followed to z=0!'Descendantof109974atz=0is41092Descendantof110822atz=0is338375Descendantof123175atz=0is257378Descendantof107743atz=0is110568Descendantof95711atz=0is260067

>>ids=[109974,110822,123175,107743,95711];>>z0_descendant_ids=zeros(numel(ids))-1;>>>>fori=1:numel(ids)>>start_url=['http://www.illustris-project.org/api/Illustris-1/snapshots/68/subhalos/'num2str(ids(i))];>>sub=get_url(start_url);>>>>whilesub.('desc_sfid')~=-1>>% request the full subhalo details of the descendant by following the sublink URL>>sub=get_url(sub.('related').('sublink_descendant'));>>ifsub.('snap')==135,z0_descendant_ids(i)=sub.('id');,end>>end>>>>ifz0_descendant_ids(i)>=0>>fprintf('Descendant of %d at z=0 is %d\n',ids(i),z0_descendant_ids(i));>>else>>fprintf('Descendant of %d not followed to z=0!\n',ids(i));>>end>>endDescendantof109974atz=0is41092Descendantof110822atz=0is338375Descendantof123175atz=0is257378Descendantof107743atz=0is110568Descendantof95711atz=0is260067

julia>

Task 9: for Illustris-1 at $z=2$ track Subfind ID 109974 to $z=0$, using the API to walk down the
SubLink descendant links, and plot the mass evolution of each component (gas, dark matter, stars,
and black holes).

>>>id=109974>>>url="http://www.illustris-project.org/api/Illustris-1/snapshots/68/subhalos/"+str(id)>>>sub=get(url)# get json response of subhalo properties>>>>>># prepare dict to hold result arrays>>>fields=['snap','id','mass_gas','mass_stars','mass_dm','mass_bhs']>>>r={}>>>forfieldinfields:>>>r[field]=[]>>>>>>whilesub['desc_sfid']!=-1:>>>forfieldinfields:>>>r[field].append(sub[field])>>># request the full subhalo details of the descendant by following the sublink URL>>>sub=get(sub['related']['sublink_descendant'])>>>>>># make a plot (notice our subhalo falls into a much more massive halo around snapshot 105)>>>forpartTypein['gas','dm','stars','bhs']:>>>mass_logmsun=np.log10(np.array(r['mass_'+partType])*1e10/0.704)>>>plt.plot(r['snap'],mass_logmsun,label=partType)>>>>>>plt.xlabel('Snapshot Number')>>>plt.ylabel('Mass [log $M_\odot$]')>>>plt.legend(loc='lower right');

>>id=109974;>>url=['http://www.illustris-project.org/api/Illustris-1/snapshots/68/subhalos/'num2str(id)];>>sub=get_url(url);% get json response of subhalo properties>>>>% prepare struct to hold result arrays>>fields={'snap','id','mass_gas','mass_stars','mass_dm','mass_bhs'};>>fori=1:numel(fields),r.(fields{i})=[];,end>>>>whilesub.('desc_sfid')~=-1>>fori=1:numel(fields),r.(fields{i})=[r.(fields{i})sub.(fields{i})];,end>>% request the full subhalo details of the descendant by following the sublink URL>>sub=get_url(sub.('related').('sublink_descendant'));>>end>>>>% make a plot (notice our subhalo falls into a much more massive halo around snapshot 105)>>fori=1:numel(partTypes)>>mass_logmsun=log10(r.(['mass_'partTypes{i}])*1e10/0.704);>>plot(r.('snap'),mass_logmsun);>>holdall>>end>>>>xlabel('Snapshot Number');>>ylabel('Mass [log $M_\odot$]');>>legend(partTypes,'Location','best');

>>>importmatplotlib.imageasmpimg>>>fromStringIOimportStringIO>>>>>>ids=[41092,338375,257378,110568,260067]>>>>>>sub_count=1>>>plt.figure(figsize=[15,3])>>>>>>foridinids:>>>url="http://www.illustris-project.org/api/Illustris-1/snapshots/135/subhalos/"+str(id)>>>sub=get(url)>>>>>># it is of course possible this data product does not exist for all requested subhalos>>>if'stellar_mocks'insub['supplementary_data']:>>># download PNG image, the version which includes all stars in the FoF halo (try replacing 'fof' with 'gz')>>>png_url=sub['supplementary_data']['stellar_mocks']['image_fof']>>>response=get(png_url)>>>>>># make plot a bit nicer>>>plt.subplot(1,len(ids),sub_count)>>>plt.text(0,-20,"ID="+str(id),color='blue')>>>plt.gca().axes.get_xaxis().set_ticks([])>>>plt.gca().axes.get_yaxis().set_ticks([])>>>sub_count+=1>>>>>># plot the PNG binary data directly, without actually saving a .png file>>>file_object=StringIO(response.content)>>>plt.imshow(mpimg.imread(file_object))

>>ids=[41092,338375,257378,110568,260067];>>>>fig=figure();>>set(fig,'Position',[001250200]);>>>>fori=1:numel(ids)>>url=['http://www.illustris-project.org/api/Illustris-1/snapshots/135/subhalos/'num2str(ids(i))];>>sub=get_url(url);>>>>% it is of course possible this data product does not exist for all requested subhalos>>if~isfield(sub.('supplementary_data'),'stellar_mocks'),continue,end>>>>% download PNG image, the version which includes all stars in the FoF halo (try replacing 'fof' with 'gz')>>png_url=sub.('supplementary_data').('stellar_mocks').('image_fof');>>response=get_url(png_url);>>>>% save PNG to temporary file and read it (cannot easily decode PNG format in-memory)>>f=fopen('out.png','w');>>fwrite(f,response);>>fclose(f);>>>>image_data=imread('out.png');>>>>% plot image and add text annotation>>subplot(1,numel(ids),i);>>image(image_data);>>text(0,-5,['\color{blue} ID='num2str(ids(i))]);>>set(gca,'XTick',[]);>>set(gca,'YTick',[]);>>end

julia>

Task 11: download the entire Illustris-1 $z=0$ snapshot including only the
positions, masses, and metallicities of stars (in the form of 512 HDF5 files).

Note: Extracting only certain particle types or fields from full snapshots

In the above example, since we only need these three fields for stars only, we can
reduce the download and storage size from ~1.5TB to ~17GB, in 512 files which are
an easy to handle ~35 MB each.

Note: Filenames of cutouts and subsets

It is the responsibility of the user to organize the data they download, particularly
important when extracting cutouts and subsets. In the above example, the download
filenames are completely unmodified -- they have no indication of the fields they contain
(this information can be found in a HDF5 Header group). They also do not uniquely identify
the files across different simulations. This is also true of halo and subhalo cutouts,
whose filenames include only a single ID by default. Therefore, they will collide with
different {cutout_query}'s, snapshots, or simulations. The user should implement a
directory structure and/or file naming scheme as needed.

Task 12? If you are having trouble figuring out if (or how) a specific task can be accomplished with the API,
please request an example!

API Reference

Endpoint Listing and Descriptions

Endpoint

Description

Return Type

/api/

list all simulations currently accessible to the user

json,api (?format=)

/api/{sim_name}/

list metadata (including list of all snapshots+redshifts) for {sim_name}

json,api (?format=)

/api/{sim_name}/snapshots/

list all snapshots which exist for this simulation

json,api (?format=)

/api/{sim_name}/snapshots/{num}/

list metadata for snapshot {num} of simulation {sim_name}

json,api (?format=)

/api/{sim_name}/snapshots/z={redshift}/

redirect to the snapshot which exists closest to {redshift} (with a maximum allowed error of 0.1 in redshift)

retrieve only SubLink main progenitor branch (towards higher redshift for this subhalo)

HDF5,json (.ext)

[base]/subhalos/{id}/sublink/mdb.hdf5

retrieve only SubLink [main] descendant branch(es) (towards lower redshift for this subhalo) (this is the single main branch only if this subhalo lies on the MPB of its z=0 descendant,
otherwise it is its full descendant sub-tree containing its z=0 descendant as the first entry)

HDF5,json (.ext)

[base]/subhalos/{id}/sublink/simple.json

retrieve a simple representation of the SubLink tree, the Main Progenitor Branch (in 'Main') and a list of past mergers (in 'Mergers')
only. In both cases a snapshot number and subhalo ID pair is given.

list of each 'files' type available for this simulation (excluding those attached to specific snapshots)

json,api (?format=)

[base]/snapshot-{num}/

list of all the actual file chunks to download snapshot {num}

json,api (?format=)

[base]/snapshot-{num}.{chunknum}.hdf5

download chunk {chunknum} of snapshot {num}

HDF5 (.ext)

[base]/snapshot-{num}.{chunknum}.hdf5?{cutout_query}

download only {cutout_query} of chunk {chunknum} of snapshot {num}

HDF5 (.ext)

[base]/groupcat-{num}/

list of all the actual file chunks to download group catalog (fof/subfind) for snapshot {num}

json,api (?format=)

[base]/groupcat-{num}/?{subset_query}

download a single field, specified by {subset_query}, from the entire group catalog for snapshot {num}

HDF5 (.ext)

[base]/groupcat-{num}.{chunknum}.hdf5

download chunk {chunknum} of group catalog for snapshot {num}

HDF5 (.ext)

[base]/lhalotree/

list of all the actual file chunks to download LHaloTree merger tree for this simulation

json,api (?format=)

[base]/lhalotree.{chunknum}.hdf5

download chunk {chunknum} of LHaloTree merger tree for this simulation

HDF5 (.ext)

[base]/sublink/

list of all the actual file chunks to download Sublink merger tree for this simulation

json,api (?format=)

[base]/sublink.{chunknum}.hdf5

download chunk {chunknum} of Sublink merger tree for this simulation

HDF5 (.ext)

Search and Cutout requests

Several API functions accept additional, optional parameters, which are described here.

{search_query} is an AND combination of restrictions over any of the supported fields, where the
relations supported are 'greater than' (gt), 'greater or equal to' (gte), 'less than' (lt), 'less than or equal to'
(lte), 'equal to'. The first four work by appending e.g. '__gt=val' to the field name (with a double underscore).
For example:

mass_dm__gt=90.0

mass__gt=10.0&mass__lte=20.0

vmax__lt=100.0&len__gas=0&vmaxrad__gt=20.0

{cutout_query} is a concatenated list of particle fields, separated by particle type.
The allowed particle types are 'dm','gas','stars','bhs'. The field names are exactly as in the snapshots
("all" is allowed). Omitting all particle types will return the full cutout: all types, all fields. For example:

dm=field0,field1,field2&stars=field0,field5,field6&gas=field3

dm=Coordinates&stars=all

{subset_query} is a concatenated list of group catalog fields, separated by object type.
The allowed object types are 'Group' and 'Subhalo'. The field names are exactly as in the group catalogs. Currently, only one single field can be requested at once. For example:

Group=GroupMass

Subhalo=SubhaloMassInRadType

Subhalo=SubhaloSFR

[base]/subhalos/{id} fields

The endpoint for an individual subhalo returns many numeric fields, as well as links
to additional data, when available:

All Subfind group catalog fields are given, flattened into separate values if a vector,
and with modified names. In particular, the 'Subhalo' prefix is removed. For values by
type, the 'Type' postfix is removed from the end of the name and '_gas', '_dm', '_stars', and
'_bhs' added. For 3-vectors, '_x', '_y', and '_z' are added. For example, SubhaloPos
becomes pos_x,pos_y,pos_z.

Main branch merger tree links from SubLink are given in terms of their Subfind IDs and snapshots.
For first progenitors, the fields are prog_sfid and prog_snap. For descendants,
the fields are desc_sfid and desc_snap. If no such relation exists, both values
equal -1.

The supplementary_data field contains any pre-computed data products which are available
for this subhalo (may be empty).

Response codes:

302 - Redirection.
client must submit a second, otherwise identical request to the new URL specified in the
Location header field. (All browsers, libraries and commands will do this automatically and transparently).
For example, all raw file download requests will receive a redirect type response.

400 - Bad Request.
The server will not process the request, because it appears to contain an error.
For example, we return code 400 if you make a cutout request but include a particle type or field name
which is not known (e.g. typo).

401 - Unauthorized.
Standard response requesting authentication if it was not provided.
You will receive this response if you do not include your API Key with a request.

404 - Not Found.
Either there is no API endpoint at the URL which you requested, or you have requested a
data resource which does not exist. For example, requesting a 'stellar mocks' broadband file for a subhalo
for which it is not currently available.

500 - Server Error.
Our fault, this is not good! Please let us know if this happens.

Specifying request formats

Each API eachpoint can return a response in one or more data types. When multiple options exist, a specify
return format can be requested through one of the following methods.

(?format=) indicates that the return type is chosen by supplying such a querystring, appended to the URL.

(.ext) indicates that the return type is chosen by supplying the desired file extension in the URL.

Sending authentication

All API requests require authentication, and therefore also user registration. Each request must provide,
along with the details of the request itself, the unique "API Key" of the user making the request. You can
send your API Key in two ways:

in the querystring, by appending it to the URL (the following are just examples,
replace with your actual key): ?api_key=INSERT_API_KEY_HERE.

in HTTP header. This is particularly useful for wget commands or
within scripts: api-key: INSERT_API_KEY_HERE.

Note: If you are logged in to the website, then requests from your browser are automatically authenticated.
Navigating the Browsable API works in this way.

Full-path examples

To be explicit by way of example, the following are absolute URLs for the Illustris API
covering some of its functionality, where the type of the request and the expected response should be
clear from the preceding documentation.