A questionnaire survey on the usage of portable audio devices for university students was conducted to clarify the attitudes of listeners toward music and environmental sounds. The optimum listening level of music was measured in both quiet and noisy conditions to clarify the preferred listening environments of portable audio device users. Furthermore, the effect of listening music on the perception of environmental sounds was examined by a field survey. The questionnaire survey showed widespread and long-term use of portable audio devices among younger generations; they listened to music in various places. Twenty percent of the users typically listened to music for longer than 3 hours. Fifteen percent experienced dangerous situations while listening to music via earphones. Twenty percent felt annoyed by environmental sounds. The reasons for listening to music for these users were mainly to pass time, to refresh the mind, and to mask environmental sounds; however, some also used their portable audio devices to avoid conversation with others. Although the optimum listening level was found to be around 58 dB in a quiet condition, it rose to over 70 dB when environmental sound was present. These results suggest that listening to music via portable audio devices causes difficulty in hearing environmental sounds. The field survey on environmental sounds heard while listening to music via earphones in an outdoor environment suggested that music masked various environmental sounds and losing opportunities to notice pleasant sounds.