Preferred Label : multimodal;
EMFI synonym : Multimodal Approach; Multimodal Learning;
EFMI definition : An approach for processing and integrating multiple different data types, aiming to
capture and leverage the relationships between them for a better understanding of
the input information or improved prediction performance. These data types may include
text, images, audio, video, genomics, sensor data, etc. These different data types
may be processed using a single multimodal network (e.g., based on neural network,
or other architectures) or through separate unimodal networks (e.g., LLMs for text
and CNNs for images) where the unimodal outputs are combined. For example, in healthcare,
data from electronic health records and wearable biosensors can be combined to enable
remote monitoring of patients.; Source: Adapted from: Acosta, J.N., Falcone, G. J., Rajpurkar, P., & Topol, E.
J. (2022). Multimodal biomedical AI. Nature Medicine, 28(9), 1773–1784. https://doi.org/10.1038/s41591-022-01981-2External
Link Disclaimer Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang,
F., Cheng, F., & Luo, Y. (2022). Multimodal machine learning in precision health:
A scoping review. Npj Digital Medicine, 5(1). https://doi.org/10.1038/s41746-022-00712-8;
Origin ID : 300243;
Validated automatic mappings to BTNT
An approach for processing and integrating multiple different data types, aiming to
capture and leverage the relationships between them for a better understanding of
the input information or improved prediction performance. These data types may include
text, images, audio, video, genomics, sensor data, etc. These different data types
may be processed using a single multimodal network (e.g., based on neural network,
or other architectures) or through separate unimodal networks (e.g., LLMs for text
and CNNs for images) where the unimodal outputs are combined. For example, in healthcare,
data from electronic health records and wearable biosensors can be combined to enable
remote monitoring of patients.
Source: Adapted from: Acosta, J.N., Falcone, G. J., Rajpurkar, P., & Topol, E.
J. (2022). Multimodal biomedical AI. Nature Medicine, 28(9), 1773–1784. https://doi.org/10.1038/s41591-022-01981-2External
Link Disclaimer Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang,
F., Cheng, F., & Luo, Y. (2022). Multimodal machine learning in precision health:
A scoping review. Npj Digital Medicine, 5(1). https://doi.org/10.1038/s41746-022-00712-8
