My best guess was that Alex and Scott finally had enough with the series. They had both been considering leaving since the end of Season 7.

However, it doesn't look like they were sure they were ending despite that, because Cole was brought in to presumably take over the lead of Five-Zero. Even the series finale showed that happening, with only McGarrett being shown written out of the series. Had it been renewed, they could have gone onto a Season 11 with Cole in charge, with or without Danno.

It's clear to me why they didn't want Grover in charge. Chi McBride is almost 60, and overweight. He couldn't do the crazy action scenes, nor would it be believable even if they got a stuntman to do it.

If I had to guess what happened, it went like this:

Near the end of production of Season 10, Alex said, "I'm not coming back guys. Really, I can't be talked into it this time."

They scrambled to find a replacement, and got the Cole character. They pitched it to the network as, "Young, likable guy takes over when McGarrett bows out because of being worn down from 10 tough years on the job. Cole has a military past just like McGarrett, and we will introduce him so fans already get to know and like him before Season 10 ends."

Network executives decided to pass, which was probably smart. Long-running shows can't usually survive when the main star leaves and main character is written out.