Is there a convenience store near the museum? Anything like ice cream, cold drinks, or coffee.

Yes, there are some near the East Gate or near Wenchang-ge.
There are some, near the Eastern Gate or Wenchang-ge.