Loading…
Crop phenotyping vision-language model that answers questions about arbitrarily shaped image regions, routing each task to its own expert branch.
No providers recorded yet. Browse all providers
A breeder looking at a photograph of a plot rarely has a question about the photograph. The question is about this lesion on that leaf margin, or about the one fruit in the frame that is far enough along to pick. A general-purpose vision-language model answers about the picture as a whole, and the usual workaround — crop to a bounding box and ask again — throws away exactly the information that makes the answer possible: the shape of an irregular lesion, the boundary between diseased and healthy tissue, the part of a fruit that is occluded by a leaf.
iPheno attacks that mismatch by making the region, rather than the image, the unit of analysis. Its spatial-aware feature extractor takes an arbitrary mask — any shape the user draws — and samples it into local K-nearest-neighbour graphs, so a long thin lesion and a round fruit are both represented in their own geometry instead of being forced into a rectangle. A second mechanism, a task-aware Mixture-of-Experts router, activates a specialized module for whichever phenotyping question is being asked. Together the authors call this a dual-aware architecture: aware of where the question points, and aware of what kind of question it is.
The model was built by the AI4Bread lab at Xi'an Jiaotong University, with collaborators at Northwest A&F University, the Chinese Academy of Sciences and the Max Planck Institute for Plant Breeding Research, and published in Plant Communications in September 2026. Pairing expert routing with region-level grounding has a close analogue in biomedical imaging, where MedPLIB answers questions about medical images and returns pixel-level masks from a mixture-of-experts design; plant-facing vision models in the catalog, such as the Root Foundation Model, have so far specialized in a single task.
iPheno is trained on iPheno-120K, a corpus drawn from public sources and manually collected field samples that spans nearly 40 major crop types and more than 70,000 original images, expanded into roughly 120,000 visual question-answering items. It covers four task families: pest and disease detection, ripeness assessment, freshness evaluation, and crop object detection. Evaluated on the iPheno-120K test set and on other publicly available datasets, iPheno improved F1-score over every baseline it was compared against — all of which had been fine-tuned for the task — by margins running from 17.1% against LLaVA-1.6-13B to 28.9% against MiniCPM-o-9B, while recording the fastest inference and the lowest memory footprint in the comparison.
The released artifacts are the iPheno-120K corpus, published on HuggingFace under Apache-2.0, and the stand-alone PC client, an Apache-2.0 PyQt5 desktop front end that posts an image, a drawn region and a question to a configurable inference endpoint rather than bundling weights. Trained weights and training code are not distributed; access to the model itself is through the hosted web server at ipheno.ai4bread.com, the mobile application, or a self-configured endpoint. The paper is open access under CC BY-NC-ND 4.0.
The four tasks map onto work that happens continuously in breeding programs and in the supply chain behind them: scoring disease pressure on trial plots, deciding when a line is ready to harvest, grading post-harvest freshness, and counting or locating fruit and plants in field imagery. Because a question can be pinned to a region the user outlines, the model fits annotation-style workflows where a phenotyper scores specific structures rather than whole plots, and the mobile and web clients put that within reach of field staff who have a phone but no imaging pipeline.
Fine-scale crop phenotyping has been a persistent bottleneck in breeding: the measurements that matter are cheap to photograph and expensive to score. iPheno's contribution is twofold — a concrete mechanism for asking a vision-language model about an arbitrarily shaped part of an image, and an openly published multi-task corpus that gives the field a shared training and evaluation target for agricultural VQA. Its limits are worth stating plainly: the evaluation covers four task families rather than phenotyping broadly, and without distributed weights the reported gains cannot be reproduced independently or adapted to a new crop offline.
Much of this page is generated or calculated automatically. Flag anything that looks off and we will re-run it.