Electronic restorative imaging and examination techniques utilizing different modalities have encouraged early determination. The development of the computer-aided retrivel systems in recent years turned them into a nondestructive and popular method for diagnosis the disease in medical images. In this work, adaptive Gabor wavelet filter bank and Texton based a feature descriptor is developed for medical image retrieval. The design of the proposed descriptor basis provides flexibility in order to extract the dominant directional features from medical images.. Also, we present a novel end-to-end integrated deep learning model using Convolutional Neural Network (CNN) and the Long Short-Term Memory cell (LSTM). The proposed integrate deep learning descriptor is compared to other descriptor such as CCM, CHD, MTH and MSD using the datasets such as New Caltech , Corel-1000,Oliva and Corel10,000.