You have to factor in time. We have only been able to send radio waves for 120 years, so anything farther than 60 light years away we won't have got a reply from yet. Anything farther than 120 light years away, haven't heard us yet.
The greater the distance, the longer a civilisation has to exist with the technology to send and receive signals in order for a reply to be interpreted. Our galaxy is 50,000 light years across, so we need to be around for the next 100,000 years to be sure we're "alone".
Although you're correct in that 'space is huge' and these signals can take thousands of years to reach us, scientists have countered that argument with the fact that our solar system is relatively young compared to the rest of our galaxy. Earth is around 4.54 billion years old. Of the 1 billion (minimum) Earth-Like planets in our galaxy, there will be millions that are over 8 billion years old - meaning theoretical life on those planets has had an extra 3.5 billions years worth of evolution/scientific advances/time to send out signals.
Running the maths and various hypothesis, a really, really, really conservative estimate would be around 100,000 alien civilisations within our galaxy. Even if a tiny fraction of those civilisations sent out signals (that we could monitor), we should have heard something by now, because they theoretically could have been sending them for billions of years. They have had more than enough time to reach us.
This means either:
- There is very little life out there (either the great filter is behind us, or we're one of the first to evolve to this point - which is unrealistic given the age of our solar system).
- Most intelligent life purposefully avoids sending out signals (perhaps they are fearful of being discovered - again unrealistic as rules out even a tiny fraction from breaking this rule).
- We're too primitive to understand the abundant signals (again unlikely as other civilisations are likely to use similar methods at some point in their existence).
- We are receiving signals, but the information is being withheld from us by people in power.
- We are completely wrong about everything we perceive the universe to be.
Again, Fermi's paradox is based on the presumption that mathematically (and scientifically) speaking, the universe should be teeming with life. It shouldn't be rare. It should have existed for billions of years. There should have been enough time for more advanced civilisations to 'contact' us. Even in the most incredibly conservative predictions, we should have 'some' signs of other life, yet we're met with complete and utter silence.
It's very, very interesting and unfortunately, it's something I highly doubt we'll ever have answered in our lifetimes. :')