The latest Superpixel Benchmark [] also includes revised implementations of the graph-based image segmentation algorithm by Felzenswalb and Huttenlocher [] — referred to as FH — and SEEDS []. Separately from the benchmark, the implementations are discussed here.

Overview

FH [] and SEEDS [] are two very popular and efficient superpixel algorithms — actually, FH is a general image segmentation algorithm that is commonly used to generate oversegmentations. As part of the Superpixel Benchmark presented in [], revised implementations of FH and SEEDS are used for comparison purposes. The presented implementations demonstrate improved connectivity which can also be linked to better performance with regard to common metrics such as Boundary Recall [] and Undersegmentation Error []. The revised implementation of SEEDS — reSEEDS — additionally reduces runtime, while the performance improvement of the revised implementation of FH — reFH — comes at the cost of increased runtime. This page discusses both the algorithm's background and implementation. Additionally, a qualitative and quantitative comparison is provided.

SEEDS Revised (reSEEDS)

Given an image $I: \{1, \ldots, W\} \times \{1, \ldots, H\} \rightarrow \{0, \ldots, 255\}^3$, the algorithm starts by grouping pixels into blocks of size $w^{(1)} \times h^{(1)}$. These blocks are then further arranged into blocks of $2 \times 2$. This scheme is applied recursively to form a hierarchy of blocks such that blocks at level $l$ have size

This is repeated for a total of $L$ levels. Therefore, the number of superpixels is calculated as follows:

$S = \frac{W}{w^{(L)}} \cdot \frac{H}{h^{(L)}}$

where we assume the divisions to be integer divisions. In practice, this means that the number of levels $L$ and the initial block size $w^{(1)} \times h^{(1)}$ need to be dervied from the desired number of superpixels $K$.

The initial superpixel segmentation is refined by exchanging blocks of pixels as well as single pixels between neighboring superpixels. Therefore, let $B_i^{(l)} \subseteq \{1,\ldots, W\}\times\{1,\ldots, H\}$ be a block of pixels at level $l$ and $S_j$ be a superpixel. With $h_{B_i^{(l)}}$ we denote the color histogram of block $B_i^{(l)}$. Then, the similarity of block $B_i^{(l)}$ and superpixel $S_j$ is given by the intersection of the corresponding histograms:

Algorithm 1: The basic algorithm of SEEDS. In practice, both block updates as well as pixel updates can be iterated more than once.

As alternative, the similarity of a pixel $x_n$ and a superpixel $S_j$ can be expressed as distance:

$d(x_n, S_j) = \|I(x_n) - I(S_j)\|_2$

where $I(S_j)$ is the mean color of the superpixel $S_j$. This variant of pixel updates is called mean pixel updates.

Compactness and Smoothness

[] discuss both a compactness and a smoothness term. The former is comparably easy to implement — adapting the distance used for mean pixel updates as follows:

$d(x_n, S_j) = \|I(x_n) - I(S_j)\|_2 + \lambda\|x_n - \mu(S_j)\|_2$

where $\mu(S_j)$ is the center position of superpixel $S_j$.

The latter tries to encourage smooth superpixel boundaries. For each pair of pixels $(x_n, x_m)$ belonging to different superpixels, a $3 \times 4$ or $4\times 3$ neighborhood is considered. Then, the number $N_{n,m,j}$ of pixels belonging to superpixel $S_j$ as well as the number $N_{n,m,k}$ of pixels belonging to superpixel $S_k$ is counted. Then, the histogram-based criterion for pixel updates is weighted by these counts. However, the smoothness term is not implemented for mean pixel updates.

Implementation

The implementation was done with ease-to-use and efficiency in mind. It also easily integrates with OpenCV. In the following, we provide building and usage instructions. More details can be found in the GitHub repository.

Usage

SeedsRevised.h provides two classes: SEEDSRevised implements the original SEEDS algorithm including smoothness term; SEEDSRevisedMeanPixels implements the variant with mean pixel updates and compactness term. Usage is comparably simply given an OpenCV image and a desired number of superpixels:

FH Revised (reFH)

Given an image $I$, $G = (V, E)$ is the graph with nodes corresponding to pixels, and edges between adjacent pixels. Each edge $(n,m) \in E$ is assigned a weight $w_{n,m} = \|I(x_n) - I(x_m)\|_2$. Then, for superpixels $S_i,S_j \subseteq V$ we define $MSTS_i)$ to be the minimum spanning tree of $S_i$ and

where $\tau$ is a threshold parameter and $MInt(S_i,S_j)$ is called the minimum internal difference between superpixels $S_i$ and $S_j$. Starting from an initial superpixel segmentation where each pixel forms its own superpixel the algorithm processes all edges in order of increasing edge weight. Whenever an edge connects two different superpixels, these are merged if the edge weight is small compared to the minimum internal difference.

Implementation

reFH was originally meant as an experimental prototype. Thus, the algorithm is flexible in which distance is used to compute the edge weights and allows to easily implement custom distances as well as threshold mechanisms. The implementation interfaces to OpenCV.

Results

As subsumed in the latest Superpixel Benchmark, qualitative and quantitative results are presented. We compare the original implementations with the presented implementations. The parameters for all algorithms were optimized separately on the validation set of the BSDS500 dataset [] as described in []. Used metrics are discussed first.

Metrics

Boundary Recall [3] quantifies the fraction of boundary pixels in $G$ correctly captured by $S$. Let $F(G,S)$ and $TP(G,S)$ be the number of false negatives and true positive boundary pixels. Then Boundary Recall is defined by

Qualitative

Slider 1 (click images to enlarge): Superpixels generated by SEEDS on three images from the BSDS500 test set. The image captions refer to the number of superpixels $K$. Different visualizations are shown.

Slider 2 (click images to enlarge): Superpixels generated by reSEEDS on three images from the BSDS500 test set. The image captions refer to the number of superpixels $K$. Different visualizations are shown.

Slider 3 (click images to enlarge): Superpixels generated by FHon three images from the BSDS500 test set. The image captions refer to the number of superpixels $K$. Different visualizations are shown.

Slider 4 (click images to enlarge): Superpixels generated by reFH on three images from the BSDS500 test set. The image captions refer to the number of superpixels $K$. Different visualizations are shown.

Quantitative

Figure 1 shows Boundary Recall (Rec), Undersegmentation Error (UE) and runtime (t) plotted against the number of generated superpixels for all four evaluated implementations, SEEDS, reSEEDS, FH and reFH. As can be seen, reFH clearly outperforms FH with respect to Rec and UE, however, the increased performance comes at the cost of increased runtime. SEEDS, in contrast, demonstrates very high Rec such that reSEEDS is not able to outperform reSEEDS significantly. Regarding UE, reSEEDS is able to outperform SEEDS. However, the difference is not as significant as in the cases of FH and reFH. Finally, reSEEDS runs significantly faster than SEEDS. Here, we note that SEEDS represents the original implementation — the OpenCV implementation of SEEDS is significantly more efficient that#n the original one. However, we do not have experimental results comparing reSEEDS to the OpenCV implementation.

Figure 1 (click to enlarge): Quantitative results on the BSDS500 test set. Left: Boundary Recall (Rec) plotted against the number of generated superpixels; for clarity, the x-axis is shown in log-scale. Middle: Undersegmentation Error plotted against the number of generated superpixels. Right: runtime on an Intel® Core™ i7-3770@3.4GHz, 64bit with 32GB RAM plotted against the number of generated superpixels.

Conclusion

We presented two revised implementations, reSEEDS and reFH, of popular superpixel algorithms, namely SEEDS [] and FH []. Overall, our revised implementations illustrate that revisiting the original implementations of superpixel algorithms might be beneficial to increase performance and reduce runtime. Note that similar experiments are discussed in [] in more detail.