A lot of what it does is "proprietary" because, much like with SEO, if the details were public it becomes a lot easier to bypass.
My understanding is that it monitors the activity of the user to see if they are a bot or not.
Over the years, I've seen a lot of talk that the main way it does this is by monitoring mouse movements. Humans have no pattern to mouse movements, whereas bots almost always do (even if there's an element of randomness to the movement, there is still a "pattern").
If reCaptcha doesn't detect any strangeness in the way you interact with the checkbox, it will simply let you continue without doing anything further. If it does find problems it will stop the user from carrying on.
If the system isn't sure whether the user is a bot or a human, it has further checks (like clicking every square with a bike/bridge/traffic light etc etc.).
I'm sure there's more behind the scenes, probably even some details that are public or at least have good guesses, but the long and short of it is "It's an industry-recognised system for preventing bots from a reputable source".