Segmenting large images with fewer resources
Exact out-of-core computation and incremental interaction, from a CIFRE PhD at LIGM and Thermo Fisher Scientific.
English · Lire en français
Objects of interest in an image can occur at different scales: a cell and its organelles, an animal and its body parts. Segmentation delineates these regions. A single level of detail is therefore not always enough to represent the structures relevant to an analysis.
My CIFRE PhD, Scalable Algorithms for Hierarchical Image and Data Segmentation, carried out at ESIEE Paris between LIGM and Thermo Fisher Scientific, aims to make hierarchical segmentation practical for large images despite its memory and computation costs.
One image, several scales
A segmentation hierarchy brings together several partitions of the same image, from fine to coarse. Small regions progressively merge into larger ones. These partitions are nested: two regions in the hierarchy are either disjoint, or one contains the other.
A whole bird can thus be found at one scale and its beak at another. The hierarchy lets us choose regions at the level of detail appropriate to each object, without imposing a single scale on the whole analysis. The figure below connects the image partitions to their tree representation.
Move the slider, then click the image or a node in the hierarchy.
Loading the hierarchy…
Building and using this representation has a cost: relationships between regions must be stored and processed alongside the pixels. For the 3D volumes containing billions of voxels produced by electron microscopy, RAM becomes a limitation. When a user refines a segmentation, response time also matters. The two contributions address these constraints: computing hierarchies with less memory, and accelerating interactive corrections.
First contribution: exact out-of-core computation
An in-core method keeps the image and its computation structures in RAM. Those structures can be much larger than the pixels: a 3.35 GB image can require hundreds of gigabytes to construct its hierarchy. The thesis experiments include an 8-bit FIB-SEM volume of 2048 × 1536 × 1065 voxels, approximately 3.35 billion voxels. The classical in-core algorithm could not process the complete volume on the 256 GB workstation used in the study.
An out-of-core algorithm keeps data on disk and loads only the blocks needed for the current computation.
The scientific challenge is preserving region inclusion relationships across the complete image, including regions that span several blocks. Processing each block independently can introduce artificial boundaries:
Our algebraic framework constructs a distribution of the binary partition hierarchy: local trees consistent with one global representation. Three operations organize the computation: select extracts relevant information, join combines information from neighboring blocks, and insert completes and corrects local hierarchies.
Mathematical guarantees establish that blockwise computation recovers the hierarchy of the global calculation, under the same construction conventions. The causal scheme also bounds auxiliary structures relative to neighboring elements of the distribution.
Mathematical foundations
Chapter 4 and its supporting appendix contain five definitions and nine mathematical results: three properties, five lemmas, and one corollary. They formalize the operations, relationships between representations, and bounds on intermediate structures.
For a global hierarchy \(H\) and a block \(S_i\), the intended local element is
\[H_i = \operatorname{select}(S_i,H).\]The challenge is constructing these elements without first storing the complete hierarchy \(H\) in RAM. This exactness guarantee concerns the computation; whether a segmentation is suitable for an application is a separate question.
The framework extends to regional attributes—area, volume, height, and minima—and to watershed segmentation, including its hierarchical version.
How large an image can we process?
File size alone does not tell us whether an image can be processed: the hierarchy and computation also require memory. Inverting the models derived from the experiments gives an estimated maximum image size for a given RAM budget.
With 32 GB for computation, the models suggest approximately 196 MB of image data in core, compared with 10.7 GB using an optimized out-of-core representation, for uncompressed 8-bit images. The compact representation is explained in the next section.
Estimate the maximum image size
Choose the RAM available for computation. The table estimates the maximum uncompressed 8-bit image size under the two models.
| Method | Estimated maximum image size | 8-bit voxel count |
|---|---|---|
| In core | ||
| Optimized out of core |
Estimates without an additional safety margin. The optimized representation and its parameters are explained in the next section. Results depend on the data, partitioning, and implementation; they are not capacity guarantees.
Calculations and assumptions
With image size \(S\) and peak RAM \(M\) in GB, fitting the six points of Figure 4.11(a) gives
\[M_{\mathrm{in\ core}}\approx162.94S+0.062.\]The optimized out-of-core configuration reaches a minimum close to three times the image size. For a RAM budget \(B\), the estimated maximum image sizes are therefore
\[S_{\max,\mathrm{in\ core}}\approx\frac{B-0.062}{162.94}, \qquad S_{\max,\mathrm{out\ of\ core}}\approx\frac{B}{3}.\]For \(B=32\) GB, this gives approximately \(0.196\) GB in core and \(10.7\) GB out of core.
Continuing the in-core trend to the complete 3.35 GB volume predicts about 550 GB, compared with the optimized measured minimum of 10 GB: roughly 98% less RAM by estimation. This comparison between the Chapter 4 and Chapter 6 implementations is distinct from the measured 84% gain between out-of-core configurations, presented below.
The fit was reconstructed from plotted points rather than raw measurement logs. The out-of-core coefficient depends on image content, partitioning, and the representation choices explained in the next section. It is not a universal capacity limit. GB are decimal: for an uncompressed 8-bit image, 1 GB corresponds to one billion voxels.
More compact representations
To reduce resource use further, we group pixels into superpixels defined by quasi-flat zones. A parameter \(\alpha\) controls the granularity of this representation.
For marker-based watershed, the compact representation preserves the corresponding pixel-level result when markers occupy distinct quasi-flat zones. This condition retains the information needed to separate them.
Experiments constructing the hierarchy distribution with the in-house implementation on a 256 GB workstation gave the following results:
| Resource | Result at α = 111 |
|---|---|
| RAM | Approximately 84% lower minimum peak: about 10 GB versus 62 GB. |
| Temporary storage | Approximately 40 GB versus 299 GB, an 87% reduction. |
| Construction time | Approximately 40 min versus 100 min, a 60% reduction. |
Both configurations are out-of-core. They use pixel-level and superpixel representations with different partitioning strategies. The 84% compares the minima over a sweep of block counts; storage and time are compared at 80 blocks.
Second contribution: faster interactive corrections
The first contribution makes large images tractable despite limited RAM. A different obstacle appears during use: waiting for the result after each correction. This second contribution uses the hierarchy to accelerate interactive watershed segmentation, when the image and computation structures fit in memory.
In interactive segmentation, the user identifies objects of interest and the background with markers, inspects the result, then adds or erases markers to refine the boundaries.
The workflow is incremental by nature: the image remains the same while the user’s instructions evolve step by step. Yet non-incremental watershed implementations treat each correction as a new problem, recomputing the segmentation across the entire image even for a local edit. Repeating this computation makes the user wait after each correction.
Our incremental method preserves the computation state between interactions. It reuses results that remain valid and updates only regions affected by adding or removing a marker. The update cost is proportional to the affected pixels, allowing local corrections to be incorporated quickly as the user progressively refines the segmentation.
A parallel strategy accelerates larger updates. Across the five 3D benchmark volumes, the mean time was 0.34 seconds per interaction, compared with 9.26 seconds for ITK: approximately 27 times faster, excluding initialization.
These timings describe the experimental in-core implementation with simulated 3D interactions, not the Amira™ demonstrator. The practical benefit is a shorter wait between a correction and its visible effect.
An industrial and scientific partnership
The collaboration fulfilled industrial and scientific objectives together. Thermo Fisher Scientific’s operational needs informed the research questions; LIGM contributed expertise in mathematical morphology and graph algorithms. The resulting methods combine mathematical guarantees, measured performance, and software development.
The work produced two journal articles and three international conference publications while preparing transfer into the company’s tools. An Amira™ demonstrator was presented at VivaTech in 2025. The public C++ implementation of incremental interactive watershed accompanies the associated article and maintains a direct connection with the research community.
People behind the work. Jean Cousty and Benjamin Perret at LIGM/ESIEE Paris, and Harold Phelippeau at Thermo Fisher Scientific, supervised the thesis. The interactive segmentation contribution was developed with Quentin Lebon. My intern Matheus Sano also contributed to the work. I also thank Nicolas Constanty, Franck Fauverte, and Daniel Lichau for their contributions to the Amira™ demonstrator.
Next steps
The next research directions are to:
- reduce repeated region information by constructing hierarchies across successive ranges of scales, making full-resolution processing more economical;
- bring incremental interactive updates into the out-of-core setting;
- extend the computation of geometric region attributes;
- explore parallel, distributed, and GPU architectures, together with connections to deep learning.
These directions support further maturation of the demonstrator and evaluation of its integration into Amira™/Avizo™, across a wider range of data and user workflows.
Figures adapted from my thesis and defense slides. Out-of-core results: Chapters 4 and 6; interactive segmentation: Chapter 7 and the associated paper. See also the public thesis record.