In the EU, treating patients with prostate (PCa) and kidney cancer (KC) costs more than €6.6 billion annually. Yet, PCa and KC are often managed inadequately, which is associated with high costs and negative consequences such as hospitalisation, psychosocial stress and poorer chances of survival. Diagnostic and therapeutic effectiveness depends on multimodal information, including cancer type, stage, and location as well as the patient’s age and health. Current clinical methods do not effectively use the large amount of mostly unstructured data. The main challenge in developing multimodal models is the lack of access to data sources and missing joint validation of data through collaboration between clinicians and computer scientists. A strength of our consortium is access to multiple sources of medical data, including the largest expert-annotated database for PCa and KC to date. Our overall goal is to develop and deploy marketable datadriven multimodal decision support systems to improve clinical prognosis, patient stratification and individual therapy for patients suffering from PCa or KC, defining a new state-of-the-art for the development of multimodal medical AI applications. We will develop AI models for PCa and KC that incorporate multimodal data, e.g., image data, unstructured medical text notes, laboratory information and biomarkers, and perform a prospective validation of the models in a large prospective multicentric international study. At the same time, we will assess the trust of healthcare professionals and patients in such AI tools and explore how this trust can be increased. By providing improved, personalised diagnosis and prognosis assessment, the multimodal models will ultimately contribute to better patient outcomes and quality of life. The models developed in this study can be used as basis for any use case where imaging and electronic medical records are relevant, as they are easily adaptable and can help combat different types of cancer.