This paper presents the implementation for hand tacking and gesture recognition on Altera FPGA. For hand tracking, an improved hand segmentation algorithm combined YCbCr color space and region growing algorithm is used for valid region separation, and then morphological operations including erosion and dilation are applied to this system in order to remove extra noise. The three-frame-difference motion detection method is used to detect moving hand. Finally the centroid of hand can be located. For gesture recognition, we develop a hand model based on finger state, converting pose recognition to finger state detection. The hand recognition system has been entirely implemented on Altera Cyclone II FPGA