Existing compressed ultrafast photography (CUP) mainly acquires the light intensity and loses the phase information that characterizes the spatial features. In this paper, we propose a compressed ultrafast holographic imaging system to achieve phase measurement in ultrafast dynamic samples. In the proposed imaging system, the Mach-Zehnder interference structure is incorporated into compressed ultrafast photography. Based on this imaging system, a reconstruction method combining the plug-and-play generalized alternating projection (PnP-GAP) framework and the depth filtering of prior FastDVDNet is proposed to recover the hologram sequence in ultrafast temporal resolution. FastDVDNet employs the correlation between frames to enhance denoising performance and maintain temporal continuity, thus obtaining better results for ultrafast hologram sequence reconstruction. The simulation results indicate that the proposed method outperforms other reconstruction methods in terms of average peak sig