Recovery and localization of handwritings by a camera-pen based on tracking and document image retrieval

We propose a camera-based method for digital recovery of handwritings on ordinary paper. Our method is characterized by the following two points: (1) it requires no special device such as special paper other than a camera-pen to recover handwritings, (2) if the handwriting is on a printed document, the method is capable of localizing it onto an electronic equivalent of the printed document. The above points are enabled by the following processing. The handwriting is recovered by the LK tracking to trace the move of the pen-tip. The recovered shape is localized onto the corresponding part of the electronic document with the help of document image retrieval called LLAH (locally likely arrangement hashing). A new framework for stably estimating the homography from a camera-captured image to the corresponding electronic document allows us to localize the recovered handwritings accurately. We experimentally evaluate the accuracy, processing time and memory usage of the proposed method using 30 handwritings. From the comparison to other methods that implement alternative ways for realizing the same functionality, we have confirmed that the proposed method is superior to those other methods.