Sour grapes.

There is nothing popular fiction hates more than somebody doing something everyone wants to do but can’t. Impossibility, when possible, becomes cast as immorality or immaturity or otherwise something arbitrarily undesirable.

To be a ghost is depressing and/or monstrous because when we die in real life we don’t stick around. Time travel overwrites reality with a worse version of the present because in real life we can’t change the past. Resurrection brings people back as monsters because in real life we can’t have our lost loved ones back. Immortality is sad and lonely and often requires you to do evil things to sustain it because in real life we can’t live forever. Traveling to alternate lifetimes where you’re more successful is emotionally hollow because you had the most important emotional stuff in your life all along and you wouldn’t trade that for the world.

These and other speculative crises always have to be fixed by making the fictional world abide be the limitations of the real one. Aren’t we so lucky that our world is randomly already like this?

Disclaimer that I don’t believe in ghosts, but in fiction at least, I think the usual implication is that a ghost is someone who hasn’t passed on correctly. A few people have brought up unfinished business already, but even in stories that don’t bring that up, ghosts are often people who died horribly, prematurely, and/or violently. Sometimes they’re explicitly under a curse keeping them from moving on.

Basically, the circumstances of their existence are wrong, and they’re stuck due to forces beyond their control. That’s kind of the tragedy of being a ghost; they’re often a whittled-down, corrupted version of their living self.

Most of the ghosts at Hogwarts in Harry Potter seem to either actively enjoy being a ghost or at least not mind it and just carry on doing stuff (like assist the pupils)

That aside, plenty stories have the ghost relive the moments before their death involuntarily, over and over. So for them it’s nothing they can control.

But I’m with you: If I were not trapped in this deathloop (or bound to one location) and were able to be seen by other humans, I would definitely not be gloomy, once I got over the fact that I died and can no longer interact in all the usual ways with my spouse and dogs. Even if I could not do crazy telekinetic stuff, I could at least wander nature, haunt bad politicians and give my spouse inside knowledge, which she can possibly make a lot of money from to buy a house for us.

I’d imagine that the ghosts who ditched their unfinished business to explore the world beyond would be doing that exploring in a world beyond so we wouldn’t see them or anything.

Popular beliefs influences people’s beliefs, which reinforces popular beliefs. Step back even farther from the question for a moment and ask, “why do you think of ghosts as dead human spirits at all?” That a “ghost” is some sort of dead human spirit is a concept that has been built into Western society for a long time. It is something we just accept in story telling and mythological belief systems because it’s been in them so long and is told to us via authoritative figures in our lives from an early age. To tell a story where a ghost is anything other than a dead human spirit or the echo of a dead human, makes people call bullshit on the story, because the story has broken a long standing societal expectation. Sure, some stories can get away with it, and more so in the modern age where we are starting to appreciate stories which subvert long standing expectations. But, we still tend to fall back on old tropes and devices which we can expect readers to understand, without having to spend too much time on building a world. It’s far easier to save the term “ghost” for something much like a dead human spirit and just create a new term when trying to describe something else.