CNN-BASED PLANT SPECIES CATEGORIZATION USING NATURAL IMAGES

Abstract

Automatic identification of plants from natural images is a challenging problem that is relevant to both the disciplines of Botany and Computer Science. The classification of plant images at the species level is a computer vision task called fine-grained categorization. This categorization problem is particularly complicated due to a large number of plant species, the inter-species similarity, the large-scale variation in appearance, and the lack of annotated data. Despite the availability of dozens of plant identification mobile applications, categorizing plant species from natural images remains an unsolved problem - e.g., most of the existing applications do not address the multi-scale nature of this type of image. Furthermore, an automated system capable of addressing the complexity of this computer vision problem has important implications for society at large, not only in preserving ecosystem biodiversity and public education but also in numerous agricultural activities such as detecting abnormalities in plants and analyzing food crops. In this dissertation, I present a new approach to the problem of automatically categorizing plant species using photos taken in nature. Essentially, this approach assembles a collection of Convolutional Neural Networks (CNN-based) to create a plant categorization system that I named WTPlant (What's That Plant?). One of the novelties of this system is a preprocessing method that extracts multi-scale samples from natural images, making the classification models more robust to variations in the scale of the plant. A comprehensive experimental evaluation of this new preprocessing method compares its performance with frequently used data augmentation techniques over different classification models of the system. WTPlant also enables the categorization of multiple plant components simultaneously by employing distinct classification pipelines for plants (leaves, branches, bushes, and trees) and flowers. The combination of these multi-organ analyses ensures a broader categorization process. It can be further extended by adding pipelines for fruits, barks, roots, etc., depending on the availability of annotated images. In summary, this new approach locates multiple plant organs in a natural image and guides the extraction of representative samples at various scales used to train and test state-of-the-art CNN classification models. To apply the WTPlant system in a real-world environment, I implement a scale-up process that adapts the classification models. In this process, models have their top classification layers replaced to accommodate a more significant number of plant species. But due to a lack of training data, these models have to be pre-trained to achieve satisfactory performance. As a result, I also implement the integration of domain-specific knowledge to create plant and flower expert classification models. Initially focusing on the University of Hawai'i Manoa campus plants, this research aims to produce the most accurate system for classifying Hawaiian plants and make it available to botanists, tourists, and the entire community to use. As a case study, I create a mobile version of the WTPlant system to categorize plant species from the Harold L. Lyon Arboretum, a University of Hawai'i Research Unit located at the upper end of the Manoa Valley.