Efficient Vision-Language Models for Resource-Constrained Edge AI System

Primary supervisor

Adamu Muhammad Buhari

Co-supervisors

Vision-Language Models (VLMs) are increasingly capable of understanding visual scenes, interpreting natural-language instructions, and reasoning across visual and textual information. These capabilities create new opportunities for intelligent systems that can interact with users and understand their surroundings. However, most state-of-the-art VLMs are designed for cloud infrastructure or high-performance GPU systems, making their direct deployment on embedded platforms challenging because of memory, computation, latency, and energy constraints.

This project explores how Vision-Language Models can be adapted and optimised for efficient real-time operation on resource-constrained edge AI platforms. Rather than training a large foundation model from scratch, the work will build on suitable open-source VLMs and investigate techniques that reduce their computational requirements while preserving useful multimodal reasoning capabilities.

Possible directions include model compression, quantisation, knowledge distillation, efficient multimodal representations, and hardware-aware inference optimisation. Particular attention will be given to the trade-off between reasoning performance and deployment efficiency: a smaller or faster model is useful only if it retains sufficient visual-language understanding for the intended task.

The developed methods will be evaluated using public vision-language and embodied AI benchmarks before deployment on embedded AI hardware such as the NVIDIA Jetson AGX Thor and Jetson Orin NX. Evaluation will consider not only task performance, but also inference latency, memory utilisation, computational efficiency, and energy consumption.

As a final demonstration, the resulting framework may be integrated with the Unitree Go2 quadruped robot for real-time semantic perception and autonomous task execution in an indoor environment. The robot serves as an application and validation platform; the main research contribution remains the development and evaluation of efficient Vision-Language reasoning methods for edge AI.

The project is research-focused and is intended to produce results suitable for a Minor Thesis/Honours dissertation and a publication-quality research manuscript.

Aim/outline

The aim of this project is to develop and evaluate an efficient Vision-Language reasoning framework that enables real-time multimodal AI capabilities on resource-constrained edge computing platforms.

The project will focus on:

  • Selecting and benchmarking suitable open-source Vision-Language Models for edge deployment.
  • Investigating efficient multimodal representation and inference techniques.
  • Applying model compression, quantisation, knowledge distillation, and hardware-aware optimisation where appropriate.
  • Optimising inference on NVIDIA Jetson platforms using frameworks such as TensorRT.
  • Studying the trade-off between reasoning capability and computational requirements.
  • Evaluating performance using public vision-language and embodied AI benchmark datasets.
  • Measuring reasoning accuracy, inference latency, memory utilisation, computational efficiency, and energy consumption.
  • Demonstrating the resulting system on an embodied robotic platform where feasible.

The project will not require training a large foundation VLM from scratch. The emphasis is on adapting and improving existing open-source models for efficient edge deployment.

Expected research outcomes include an experimentally validated VLM framework for edge AI, a detailed performance comparison of optimisation strategies, practical insights into accuracy-efficiency trade-offs, reproducible experimental results, a completed thesis, and a manuscript prepared for submission to an appropriate peer-reviewed conference or journal. These outcomes preserve the objectives and expected outputs specified in the original proposal.

URLs/references

  • Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., et al. (2025). “Multimodal fusion and vision-language models: A survey for robot vision.” Information Fusion, 103652.
  • Zhang, C., Chen, J., Li, J., Peng, Y., & Mao, Z. (2023). “Large language models for human–robot interaction: A review.” Biomimetic Intelligence and Robotics, 3(4), 100131.
  • Mon-Williams, R., Li, G., Long, R., Du, W., & Lucas, C. G. (2025). “Embodied large language models enable robots to complete complex tasks in unpredictable environments.” Nature Machine Intelligence, 7(4), 592–601.
  • Sun, F., Chen, R., Ji, T., Luo, Y., Zhou, H., & Liu, H. (2024). “A comprehensive survey on embodied intelligence: Advancements, challenges, and future perspectives.” CAAI Artificial Intelligence Research, 3.
  •  

Required knowledge

Students should have:

  • Good Python programming skills.
  • A foundation in machine learning, deep learning, artificial intelligence, or computer vision.
  • Basic understanding of neural networks and modern deep-learning architectures.
  • Experience with PyTorch or a similar deep-learning framework.
  • An interest in Vision-Language Models, multimodal AI, edge AI, or embedded intelligent systems.

Experience with Hugging Face Transformers, CUDA, TensorRT, NVIDIA Jetson, Linux, ROS/ROS2, model quantisation, or embedded systems would be advantageous but is not essential.

The student should also be comfortable conducting experimental research, analysing quantitative results, reading research papers, and working independently with new AI frameworks and hardware.