The Xiaotaoyuan Mosque was originally built in 1917 by Jin Ziyun, the Director of the Shanghai Islamic Board of Directors, who had bought the land. It was then rebuilt in 1925 and completed two years later in the shape it is today. The site of the mosque was the original site for the Shanghai Islamic Normal School which was later moved to Qinglian Street and renamed Pingliang Islamic Normal School. Other schools established by the mosques were the Islam Normal School, Primary School for Muslims, Mingcheng Primary School, Chongben Primary School, Shanghai Islam Orphanage etc.
After the founding of the People's Republic of China on 1 October 1949, Shanghai Mayor Chen Yi inspected the mosque and approved appropriate funds for its maintenance. On 15 February 1994, the mosque was listed as heritage architecture by the Shanghai Municipal Government.