Chapter 30: Sparse Arrays

30.1 Introduction

The sparse array facility of J allows a large array
to be stored in the computer in a moderate amount of
memory if many of the array's elements
are all the same. In this case a value which occurs many times
need be stored only once.

For an example, sparse representation might be considered for
a connection matrix describing a network. In this chapter
we will look at the J machinery for handling sparse arrays.

This array can be stored in a compact form, called a "sparse array",
where only its non-zero elements occupy storage.
An ordinary array which is not sparse may be called a "dense" array.

There is a built-in function, $. (dollar dot)
to compute a sparse array from a dense.

S =: $. D

For many purposes dense matrix D and sparse matrix S are equivalent:
S matches D, and therefore it has the same dimensions, and
gives the same result on indexing:

S -: D

($S) -: ($D)

((< 0 0){ S) -: (<0 0) { D

1

1

1

30.2 Sparse Array is Compact

Compared to matrix D, matrix S is economical in storage because the value
which occurs many times in D is stored only once in S.
This value is known as the "sparse element" of S, or the "zero" element
of S.
It happens to be 0 in the case of S, but need not be 0 always.

We can measure the size of the storage occupied by an array
with the built-in 7!:5.
We see that the size of S (which the sparse form of D)
is smaller than the size D itself:

7!:5

7!:5

384

2048

30.3 Inspecting A Sparse Array

There is a useful function datatype in the standard library.
It shows the type of its argument.

datatype D

datatype S

integer

sparse integer

Recall that the built verb 3!:0 also gives the type of its argument.
For a sparse array, the possible types reported by 3!:0 are

1024

sparse boolean

2048

sparse character

4096

sparse integer

8192

sparse floating point

16384

sparse complex

32768

sparse boxed

If we display S in the usual way , we see, not the familiar representation of a matrix,
but instead
a list of index-value pairs, one pair for each (in this example)
non-zero element.

S 2 2 | 23 6 | 34 4 | 4

This display does not show that the sparse element of S is in fact integer zero.
To show this, we can extract the sparse element with the verb 3 & $. .

se =: 3 $. S

datatype se

0

integer

If we now compute a new matrix from S

T =: S + 5

we see that T is sparse, and the sparse element of T is not zero but 5

T

3 $. T

2 2 | 7
3 6 | 8
4 4 | 9

5

Another way to view a sparse array
is simply to convert it to dense with 0 & $.

30.4 Computing with Sparse Arrays

Computations with sparse arrays are pretty much the same as
with dense arrays, except that they tend to produce sparse arrays
as results. We saw this with S+5 above. Here is another example.
Summing over T produces a vector of column-sums which is sparse

] V =: +/ T2 | 824 | 846 | 83

but the "zero" element of V is the sum of a column of "zero" elements
of T

3 $. V 80

At the time of writing, there are still some limitations on what
can be done with sparse arrays compared with dense arrays.
See the Dictionary under $. for more information.

30.5 Constructing A Sparse Array

At this point it will be helpful to define a few terms.
First note that, according to context,
the numerals 0 or 1 or 0.0 or 1.0
could be valid as boolean or integer or real. However
in the absence of any context the J system takes them all to be in fact
boolean.

datatype 0

datatype 1

datatype 0.0

datatype 1.0

boolean

boolean

boolean

boolean

It will be useful to define some values of unambiguous type.

INTEGERZERO =: 3 - 3

datatype INTEGERZERO

0

integer

INTEGERONE =: 3 - 2

datatype INTEGERONE

1

integer

REALZERO =: 0.0*0.1

datatype REALZERO

0

floating

REALONE =: ^ 0

datatype REALONE

1

floating

Returning now to sparse arrays, the recommended method of constructing them
is to begin by making an empty array of
the required shape and type, but with no actual data.

An empty array is built by evaluating the expression

1 $. shape;axes;zero

where

shape specifies the dimensions

axes specifies which of those dimensions will be sparse,
as a list of axis-numbers. For example,
with 2 dimensions both sparse the
list would be 0 1

So far, in the examples
of sparse arrays, all axes have been sparse but we will see below
mixed sparse and dense axes.

zero specifies the value of the "zero" element,
and hence the type of the array as a whole.
An unambiguous value is evidently needed.

If zero is omitted the default is REALZERO.
If both axes and zero are omitted, the default is all axes sparse
and REALZERO.

So to build a 6 by 6 matrix, sparse in all dimensions (that is, on axis 0 and axis 1),
of type integer with "zero" element of 0 we can write:

U =: 1 $. 6 6 ; 0 1; INTEGERZERO

At this point, U is empty, that is, all "zero",
so displays as nothing: