I track the passage of time in my RPGs, and so should you, and it's actually easy. This blog post tells you how.
A deck of cards. If possible, use Pathfinder or Starfinder cards, or something similar. You'll see why.
Set your deck of cards on the table. When an hour of in-game time passes, take a card from the deck and place it in the discard pile.
How do you know when an hour passes in-game? It depends.
Tracking time in a dungeon (or a similarly insulated and linear environment) is easy: 1 room on your map equals 1 hour.
Yes, sometimes players go into a room, do a single perception check, and then leave upon finding it empty. It takes 20 seconds of real time. But in-game, this simple interaction represents player characters moving cautiously into a darkened room, shining their torch into every corner, and painstakingly searching every crevice for a hidden panel or secret door. Even when nothing happens in a room, the assumption is that the players tried to make something interesting happen. That's what they're there for. They're delving into dungeons for wealth or for fame or to find a kidnapped noble or to defeat a nefarious plot. They are NOT breezing past alcoves, they're not obliviously speed-walking through corridors. They're moving slowly, methodically, and carefully, and it takes time. One location on the map = 1 hour, no exceptions.
when something does happen in a room (PCs find some loot, or a secret door, or engage in combat), you have the option of adding another hour to that location. Even though combat happens very quickly, it's fair to assume that recovering from combat (calming your nerves after a near-death experience, cleaning your weapons, looting the body) adds considerable time to the event.
When gameplay is happening outside of a dungeon, it's a little harder to track time. Story progression in dungeons usually rely on changes in location. When players move from one room to another, they encounter the next plot point. Outside a dungeon, the location may not need to change as frequently for the story to progress.
I find that a good indication of stuff happening outside the dungeon are dice rolls. Whether it's a nature or orienteering roll to find your way through wilderness, or a persuasion or diplomacy roll to convince an NPC to help you, or just plain old combat, when there's dice rolling, it probably means an hour passes. As in the dungeon, sometimes an hour seems like a lot. After all, combat happens in 6-second rounds. A simple persuasion check to convince a vendor to give you a discount on chainmail surely wouldn't take a whole hour. But actually, I find that when dice roll, it's usually because activity is happening. Maybe the roll itself is for a simple interaction, but the player characters had walk to the store, they had to meander around, get the shopkeep's attention, assess the chainmail they want to buy, and so on. Forty-five minutes, thirty minutes, even twenty minutes. Round up, call it an hour. Flip a card.
Sometimes a gaming session consists mostly of roleplay. Many game groups have at least one session in which nobody rolled even a single die. Some groups have that kind of session more often than not. I admit, I rarely have those sessions, and when I do the stakes are low enough that tracking time isn't that important. Unless there's a significant in-game time constraint, it's often not significant to me whether the PCs spend 3 hours or 12 hours asking around town for clues about the location of the mythical dungeon. However, sometimes there is a ticking clock, and time does matter, so you have to track it.
When a session consists mostly of roleplay, I count an encounter as an hour. You talk to a city guard: 1 hour. You talk to a shopkeep: 1 hour. You talk to each other: 1 hour.
A benefit of using Pathfinder or Starfinder item cards is that you have a source of random stuff when you need it. And as a Dungeon Master, you're gonna need it. It's got nothing to do with time, but inevitably there will be times when a player looks behind a curtain or in the hollow of a tree or in a wardrobe, and you have to come up with something better than "you find nothing." When your time deck happens to contain random RPG content, you can fall back on it when you need a quick idea. Even if you just write some ideas on a deck of poker cards and use it, I think having a deck of random stuff is a really useful DM hack.
Once you've started tracking time with cards, you have in-world data you can use for all kinds of mechanics.
Implement it in your next game.