AI-assisted quantitative tissue pathology identifies nuclear morphometric features distinguishing OSCC from normal oral epithelium

Early detection and risk stratification of oral squamous cell carcinoma (OSCC) remain major clinical challenges, and traditional histopathology is limited by subjectivity and interobserver variability. The objective of this study is to investigate the effectiveness using image-analysis pipeline integrating deep-learning nuclei segmentation and multifeature risk scoring of microscopic nuclear morphometric features to distinguish OSCC from normal oral tissue. We hypothesize that nuclear appearance alone contains sufficient biological signal to stratify malignant from non-malignant tissue. The dataset comprised 34 OSCC and 25 normal mucosa tissue microarray cores from 59 patients that were stained with stoichiometric Feulgen-Thionin stain. Scanned images were converted to 8-bit grayscale, and background brightness was automatically normalized. Nuclear segmentation was performed using a two-stage deep-learning UNet pipeline implemented in PyTorch, with one network detecting nuclear centers and a second network generating full nuclear masks. Following segmentation, 84 nuclear features describing DNA content, morphology, and chromatin texture and spatial organization were extracted. These features were used to train and evaluate three supervised machine-learning classifiers distinguishing OSCC from normal group: Random Forest, XGBoost, and LightGBM. All models were trained on the same feature matrix and identical training/test splits using 5-fold cross-validation. Hyperparameters were tuned using grid search. Model performance was assessed using accuracy, sensitivity, and specificity. This improved segmentation approach identified 96,000 and 78,000 nuclei in OSCC and normal, respectively. Across all classifiers, LightGBM performed the best (94.2%), followed by XGBoost (93.5%), while Random Forest showed slightly lower, but still strong, performance (91.5%). Malignant nuclei showed consistently larger size, greater irregularity, and more heterogeneous chromatin texture (p < 0.0001). Applied to a separate set of 9 oral premalignant samples, the LightGBM model correctly identified 4/5 high-grade lesions, 2 low-grade progressors, and 2 low-grade non-progressors, with 100% specificity. This approach identifies distinct nuclear morphometric signatures associated with OSCC and provide an objective, quantitative framework for diagnostic classification. Nuclear appearance alone can effectively differentiate malignant from normal epithelium, supporting its potential application in risk assessment and early detection. Ongoing efforts focus on refining the segmentation model and cell-type classification model to retain only squamous epithelial nuclei. Future work will extend this framework to oral epithelial dysplasias to evaluate its utility for progression prediction.