Graphene is an attractive material for all-optical modulation because of its ultrafast optical response and broad spectral coverage. However, all-optical graphene modulators reported so far require high pump fluence due to the ultrashort photo-carrier lifetime and limited absorption in graphene. We present modulator designs based on graphene-metal hybrid plasmonic metasurfaces with highly enhanced light- graphene interaction in the nanoscale hot spots at pump and probe (signal) wavelengths. Based on this design concept, we have demonstrated high-speed all optical modulators at near and mid-infrared wavelengths (1.56 um and above Gum) with significantly reduced pump fluence (1-2 orders of magnitude) and enhanced optical modulation. Ultrafast near-infrared pump-probe measurement results suggest that the modulators' response times are ultimately determined by graphene's ultrafast photocarrier relaxation times on the picosecond scale. The proposed designs hold the promise to address the challenges in the realization of ultrafast all-optical modulators for mid-and far-infrared wavelengths.