Schematic view of the SuperPoint Algorithm [1]

Motivation

The co-registration of satellite images is a crucial process for planetary sciences and remote sensing. Many subsequent processes, e.g., surface reconstruction and data normalization, benefit from a pixel-perfect overlap between the images. Due to uncertainties in the satellite positions and camera models, there is an offset between images even after they have been calibrated and mapped. The offset is particularly severe for images from different instruments, such as the Moon Mineralogy Mapper (M3) and the Lunar Reconnaissance Orbiter Wide Angle Camera (LRO WAC). A simple euclidean or affine transformation is usually not enough to align the images. While this work will focus on lunar images from the instruments mentioned above, the principle also applies to other planetary bodies (e.g. Mars, Mercury).

WAC image of the same crater.

M3 image of the Buys-Ballot crater on the Moon.

Challenge

The challenge of this thesis is to align distinct images of the same area on the Moon to achieve a pixel-wise overlap. The input images have different spatial resolutions and were captured from varying viewing angles. Furthermore, the images were taken at different times and therefore have varying illumination conditions.

To co-register two distinct images, various control points in each image and their corresponding feature vectors must be identified. Due to the different imaging conditions detailed above, automatically detecting suitable points is challenging. Although unsupervised approaches have been tried, the most successful approach currently is identifying and selecting points by hand. This repetitive manual task is time-consuming and error-prone.

In order to address these issues, this thesis will employ newly developed state-of-the-art machine learning techniques, including the SuperPoint [1] algorithm, to extract points and features from a collection of distinct images. As shown in the top figure, control points created by humans (already available in a database) can be used to fine-tune a new Base Detector (illustrated on the left side of the image). Another objective of this thesis is to replace the encoder CNN from the original paper with a newer Vision Transformer [2] to evaluate the effectiveness of this model change. Additionally, various homographic transformations will be applied to enhance the algorithm further, utilizing a large volume of unsupervised lunar data image pairs that do not contain control points.

After successfully finding robust control points, algorithms like RANSAC [3] or LightGlue [4] can be utilized to match the two distinct images.

Preferred Frameworks and programming languages

Python 3

PyTorch

Pytorch Lightning

MATLAB

References

[1] DeTone, D., Malisiewicz, T., & Rabinovich, A. (2017). SuperPoint: Self-Supervised Interest Point Detection and Description (Version 4). arXiv. https://doi.org/10.48550/ARXIV.1712.07629
[2] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Version 2). arXiv. https://doi.org/10.48550/ARXIV.2010.11929
[3] Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus. In Communications of the ACM (Vol. 24, Issue 6, pp. 381–395). Association for Computing Machinery (ACM). https://doi.org/10.1145/358669.358692
[4] Lindenberger, P., Sarlin, P.-E., & Pollefeys, M. (2023). LightGlue: Local Feature Matching at Light Speed (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2306.13643