The integration of optical fibre communication with multiple input multiple output-non-orthogonal multiple access (MIMO-NOMA) waveforms in cognitive radio (CR) systems is examined in this study. The proposed system leverages the advantages of optical fibre, including high bandwidth and immunity to electromagnetic interference to facilitate the transmission and reception of MIMO-NOMA signals in a CR environment. Moreover, MIMO-NOMA signal was detected and analysed by the hybrid-discrete cosine transform-Welch (H-DCT-W) method. Based on the modes results, a detection probability greater than 0.96%, a false alarm probability equal to 0.06, and a global system error probability equal to 0.09% were obtained with a signal-to-noise ratio (SNR) less than 0 dB, while maintaining a simple level of complexity. The results obtained in this paper indicate the potential of the optical fibre-based MIMO-NOMA system based on H-DCT-W technology in CR networks. Therefore, its suitability for practical CR applications is demonstrated by the improvements obtained in false alarms, detection probability, and error rates at low levels of SNR. This study contributes to the development of efficient and reliable wireless communication systems by linking cooperation and synergy concerning MIMO-NOMA, optical fibres, as well as the proposed detection technique (H-DCT-W).