There was no "Jump the Shark" moment on Five-O, but I understand why Chin's death ("A Death in the Family") was picked. It wasn't because that was a bad episode, but because it was the last episode where Five-O was recognizable as the quality show it had always been.

Season 11 as substantially worse, and now only had McGarrett and Danno left from the original team.

Season 12 was even worse than that, and now only McGarrett was left from the originals, and Duke was the only other character we recognized from other seasons.

So my answer would be, "Right after season 10 finished!"