Reference: Fataawa al-Hajj wal-‘Umrah waz-Ziyaarah – Page 104
Question: What is the ruling regarding one who remains in Minaa until midnight, then leaves for Makkah and does not return until after Fajr?
Response: If the time is midnight, then that is the middle of the night in Minaa and in that case there is no harm in leaving Minaa after this time, even though it is better to remain in Minaa the day and night. However, if the time is midnight before the middle of the night, then do not leave because the condition for remaining in Minaa is to spend most of the night there as has been mentioned by our Fuqahaa, may Allaah (Subhaanahu wa Ta’aala) have Mercy upon them.