Also near this spot, a fine assortment of conglomerate boulders crops out on a hummocky hillside. This spot is a landslide scar, and perhaps the boulders came from across the San Andreas fault and have emerged as the landslide material eroded away. See a closeup of the rock on this page.
Fault motion has commonly traded packets of rock back and forth across the fault zone, so it is not clear to me which side it originated on. Because of California's tectonic history, though, I believe the high mountains that shed the large cobbles of this conglomerate were somewhere east of the fault in the eastern Transverse Ranges.