Heterogeneous deformation during rolling is a crucial issue for elucidating recrystallization behavior. The progress thus far in our understanding of heterogeneity has been reviewed focusing on grain boundary and shear band. A statistical study on heterogeneous deformation structure using EBSD revealed that heterogeneity along the grain boundary can be classified into three types: 1) relatively flat boundary, 2) irregularly serrated boundary, and 3) boundary associated with fine grains. The fine grains in type 3 seem to be dynamically recovered as a cold-rolled state. Shear band formation is considered to be caused by plastic instability that is accelerated, for instance, by dynamic strain aging. A shear band is revealed to have a feature of recovered fine cells with Goss orientation already embedded in the shear band. The application of the phase-field method is exploited to predict recrystallization behavior and texture evolution during annealing based on the subgrain growth model. In simulation, a bulging mechanism seems to be dominant. Thus, a more rigorous description of the heterogeneous deformation structure is needed in the future.