Also inside temples, and in 'traditional' areas of a restaurant. Stores and such you generally leave your shoes on, although some places still request you remove them. (Disclaimer: Answer based on a two week vacation in Japan, so I don't know best obviously)
In my experience, anything traditional, such as temples, shrines, or even ryokan (Japanese style hotel) you take off your shoes. I have also had to take them off at a chiropractic clinic, changing rooms, and Japanese style restaurants. All of these places will have slippers provided, so if you aren't sure, take a quick look around for slippers or cubby holes for shoes.
If you find a difference in level on the floor, especially near the entrance, it means you should take off your shoes. But there are many differences. Follow guidances by staff and watch what locals do. I also do so.
I have been in many small store-front businesses where you had to take off your shoes & step up to enter the office. I recently read an article about the famous Matsuzakaya department store which stated that its Ginza branch became the first department store in Japan to allow customers to wear their shoes everywhere in the store instead of having to check them in at the entrance cloak room. Granted that was 1924, but that's how deep that custom was/is in their culture. I think there are still pre-schools and elementary schools that have their students change into slippers. In larger environments it becomes a logistics issue, so they won't do it. It seems like the smaller or more intimate the environment (like homes/apartments/small offices) the more likely one is to see this. In modern hotels (rooms) and Japanese airline flights, slippers are offered.