Object detection plays a significant role in an intelligent system equipped with a fisheye camera. The fisheye image captures a wide field-of-view but deforms in the radial direction. The deformation changes the relative angle between the edge of objects and the image. Therefore, a horizontal bounding box cannot perform an accurate description of an object's location and dimension in advanced neural network training. In this paper, we build a rotation sensitive neural network targeting to realize one-stage regression on the fisheye image detection. The oriented bounding box is applied in the object's description and detection. To evaluate our proposed method, we develop a new labelled fisheye image dataset that contains two categories. The network model training takes around 3 hours and achieves 100% precious by the test set.