Sparse domain-conditioned neuron activation in multimodal transformer networks: computational analysis and cross-backbone empirical validation
Keywords:
Multimodal transformers, Sparse activation, Domain-conditioned routing, Computational efficiency, Cross-backbone validationAbstract
Multimodal transformer networks couple a vision encoder, a learnable projector, and a language decoder to reason jointly across text and image inputs. Dense activation of every feed-forward neuron at every layer makes per-inference computational cost increase with model size even when the task does not require the network's full representational capacity. We present NEUROBALANCE, a domain-conditioned sparse-activation framework combining per-head attention reweighting, sparse feed-forward neuron gating trained with the straight-through estimator, and Elastic-Weight-Consolidation-inspired retention regularisation. A single soft domain embedding computed from input tokens drives all three mechanisms at inference. We characterise the resulting savings theoretically through a closed-form floating-point-operations (FLOPs) analysis as a function of active-neuron density and empirically on LLaVA-NeXT-7B and InstructBLIP across PMC-VQA, PathVQA, and LingoQA. NEUROBALANCE reduces forward-pass FLOPs by 36.4% and 36.6%, A100 wall-clock latency by 32.4% and 36.1%, and peak memory by 18.1% and 25.8%, respectively. Benchmark accuracy is preserved, with mean gains of +0.6 percentage points on LLaVA-NeXT and +0.5 percentage points on InstructBLIP; all per-benchmark differences are statistically significant under a paired sign-flip permutation test after Holm--Bonferroni adjustment. Per-inference energy consumption falls by at least 32.4% under a conservative latency-proportional estimate and by 43.9% when measured power is used for both dense and sparse terms. The estimated saving is 58.5 kg CO2-equivalent per million inferences at the stated global-average grid intensity. Cross-backbone consistency indicates that the computational savings arise from architectural conditioning rather than a particular backbone.
References
[1] H. Liu, C. Li, Y. Li & Y. J. Lee, Improved baselines with visual instruction tuning, IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2024, pp. 26296--26306. https://doi.org/10.1109/CVPR52733.2024.02484.
[2] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung & S. Hoi, ``InstructBLIP: Towards general-purpose vision-language models with instruction tuning'', Advances in Neural Information Processing Systems 36 (2023) 49250. https://doi.org/10.48550/arXiv.2305.06500.
[3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser & I. Polosukhin, ``Attention is all you need'', Advances in Neural Information Processing Systems 30 (2017) 5998. https://doi.org/10.48550/arXiv.1706.03762.
[4] C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. A. Aga, J. Huang, C. Bai, M. Gschwind, A. Gupta, M. Ott, A. Melnikov, S. Candido, D. Brooks, G. Chauhan, B. Lee, H.-H. S. Lee & K. Hazelwood, ``Sustainable AI: Environmental implications, challenges and opportunities'', Proceedings of Machine Learning and Systems 4 (2022) 795. https://doi.org/10.48550/arXiv.2111.00364.
[5] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu & D. Amodei, ``Scaling laws for neural language models'', arXiv preprint (2020) arXiv:2001.08361. https://doi.org/10.48550/arXiv.2001.08361.
[6] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton & J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, International Conference on Learning Representations, Toulon, France, 2017. https://doi.org/10.48550/arXiv.1701.06538.
[7] S. Han, H. Mao & W. J. Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding, International Conference on Learning Representations, San Juan, Puerto Rico, 2016. https://doi.org/10.48550/arXiv.1510.00149.
[8] B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam & D. Kalenichenko, Quantization and training of neural networks for efficient integer-arithmetic-only inference, IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, USA, 2018, pp. 2704--2713. https://doi.org/10.1109/CVPR.2018.00286.
[9] T. Tang, W. Luo, H. Huang, D. Zhang, X. Wang, X. Zhao, F. Wei & J.-R. Wen, Language-specific neurons: The key to multilingual capabilities in large language models, in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, 2024, pp. 5701--5715. https://doi.org/10.18653/v1/2024.acl-long.309.
[10] J. Huo, Y. Yan, B. Hu, Y. Yue & X. Hu, MMNeuron: Discovering neuron-level domain-specific interpretation in multimodal large language model, 2024 Conference on Empirical Methods in Natural Language Processing, Miami, USA, 2024, pp. 6801--6816. https://doi.org/10.18653/v1/2024.emnlp-main.387.
[11] T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden & A. Peste, ``Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks'', Journal of Machine Learning Research 22 (2021) 241. https://doi.org/10.48550/arXiv.2102.00554.
[12] S. M. Shagari, D. Gabi, N. M. Dankolo & N. N. Gana, ``Countermeasure to structured query language injection attack for web applications using hybrid logistic regression technique'', Journal of the Nigerian Society of Physical Sciences 4 (2022) 832. https://doi.org/10.46481/jnsps.2022.832.
[13] X. Zhu, Y. Guan, D. Liang, Y. Chen, Y. Liu & X. Bai, ``MoE Jetpack: From dense checkpoints to adaptive mixture of experts for vision tasks'', Advances in Neural Information Processing Systems 37 (2024) 12094. https://doi.org/10.52202/079017-0386.
[14] D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer & Z. Chen, GShard: Scaling giant models with conditional computation and automatic sharding, International Conference on Learning Representations, Virtual, 2021. https://doi.org/10.48550/arXiv.2006.16668.
[15] O. M. Nwakeze, N. U. Mohammed, O. C. Akachukwu, U. A. Tochukwu, O. N. Blessing, I. S. Chinasa & O. Christopher, ``Dynamic-kernel CNN-LSTM for real-time intrusion detection in low-power healthcare IoT systems'', Journal of the Nigerian Society of Physical Sciences 8 (2026) 3230. https://doi.org/10.46481/jnsps.2026.3230.
[16] G. Hinton, O. Vinyals & J. Dean, Distilling the knowledge in a neural network, NeurIPS Deep Learning Workshop, Montreal, Canada, 2014. https://doi.org/10.48550/arXiv.1503.02531.
[17] N. M. Chittaiah & S. H. Manjula, ``Adaptive sparse ternary compression with dynamic gradient thresholding for communication-efficient federated learning'', Engineering, Technology & Applied Science Research 16 (2026) 35281. https://doi.org/10.48084/etasr.18548.
[18] C. X. Liang, P. Tian, C. H. Yin, Y. Yua, A.-H. Wei, M. Li, X. Song, T. Wang, Z. Bi, M. Liu, R. Bao & P. Feng, ``A comprehensive survey and guide to multimodal large language models in vision--language tasks'', Computation 14 (2026) 125. https://doi.org/10.3390/computation14060125.
[19] M. M. Liman, R. Prasad & H. A. Amshi, ``Feature-optimized hybrid CNN--ViT architecture for sustainable vision-based condition assessment in agriculture'', Journal of the Nigerian Society of Physical Sciences 8 (2026) 3301. https://doi.org/10.46481/jnsps.2026.3301.
[20] X. Zhang, C. Wu, Z. Zhao, W. Lin, Y. Zhang, Y. Wang & W. Xie, ``PMC-VQA: Visual instruction tuning for medical visual question answering'', arXiv preprint (2023) arXiv:2305.10415. https://doi.org/10.48550/arXiv.2305.10415.
[21] X. He, Y. Zhang, L. Mou, E. Xing & P. Xie, ``PathVQA: 30,000+ questions for medical visual question answering'', arXiv preprint (2020) arXiv:2003.10286. https://doi.org/10.48550/arXiv.2003.10286.
[22] A. Marcu, L. Chen, J. Hunermann, A. Karnsund, B. Hanotte, P. Chidananda, S. Nair, V. Badrinarayanan, A. Kendall, J. Shotton & O. Sinavski, LingoQA: Visual question answering for autonomous driving, European Conference on Computer Vision, Milan, Italy, 2024. https://doi.org/10.48550/arXiv.2312.14115.
[23] B. Hanindhito, B. Patel & L. K. John, Large language model fine-tuning with low-rank adaptation: A performance exploration, 16th ACM/SPEC International Conference on Performance Engineering, Toronto, Canada, 2025, pp. 92--104. https://doi.org/10.1145/3676151.3719377.
[24] A. Lacoste, A. Luccioni, V. Schmidt & T. Dandres, ``Quantifying the carbon emissions of machine learning'', arXiv preprint (2019) arXiv:1910.09700. https://doi.org/10.48550/arXiv.1910.09700.
[25] P. Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky & J. Pineau, ``Towards the systematic reporting of the energy and carbon footprints of machine learning'', Journal of Machine Learning Research 21 (2020) 248. https://doi.org/10.48550/arXiv.2002.05651.
[26] E. Strubell, A. Ganesh & A. McCallum, Energy and policy considerations for deep learning in NLP, 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019, pp. 3645--3650. https://doi.org/10.18653/v1/P19-1355.
[27] N. Belrose, Z. Furman, L. Smith, D. Halawi, I. Ostrovsky, L. McKinney, S. Biderman & J. Steinhardt, ``Eliciting latent predictions from transformers with the tuned lens'', arXiv preprint (2023) arXiv:2303.08112. https://doi.org/10.48550/arXiv.2303.08112.
[28] Y. Bengio, N. Leonard & A. Courville, ``Estimating or propagating gradients through stochastic neurons for conditional computation'', arXiv preprint (2013) arXiv:1308.3432. https://doi.org/10.48550/arXiv.1308.3432.
[29] D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier & J. Dean, ``Carbon emissions and large neural network training'', arXiv preprint (2021) arXiv:2104.10350. https://doi.org/10.48550/arXiv.2104.10350.
[30] U. Gupta, Y. G. Kim, S. Lee, J. Tse, H.-H. S. Lee, G.-Y. Wei, D. Brooks & C.-J. Wu, Chasing carbon: The elusive environmental footprint of computing, 2021 IEEE International Symposium on High-Performance Computer Architecture, Seoul, Korea, 2021, pp. 854--867. https://doi.org/10.1109/HPCA51647.2021.00076.
[31] A. S. Luccioni, S. Viguier & A.-L. Ligozat, ``Estimating the carbon footprint of BLOOM, a 176B parameter language model'', Journal of Machine Learning Research 24 (2023) 253. https://www.jmlr.org/papers/v24/23-0069.html.
[32] B. Acun, B. Lee, F. Kazhamiaka, K. Maeng, U. Gupta, M. Chakkaravarthy, D. Brooks & C.-J. Wu, Carbon Explorer: A holistic framework for designing carbon-aware datacenters, 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vancouver, Canada, 2023, pp. 118--132. https://doi.org/10.1145/3575693.3575754.
Published
How to Cite
Issue
Section
Copyright (c) 2027 Austin Olom Ogar, Joshua Abah, Aliyu Suleiman Muhammed, Oluwatobi Noah Akande, Faruk Obansa Muhammed, Ibrahim Anka Salihu (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
