There are three main planet-hunting techniques astronomers use to spot exoplanets: radio velocity, transit luminosity dips, and a chronograph.
Large planets can have a slight gravitational pull on its host star, and the system looks slightly like a binary system. These small wobbles in the star’s position can be measured to find exoplanets.
By measuring the magnitude of the Doppler shift of the star as it moves towards and away from us, astronomers can infer the mass of the orbiting planet, and hey presto! You’ve found an exoplanet.
This only works for large planets. An object like Mercury or Pluto would have a far too small effect on the star’s rotation for us to detect.
The most frequently used method is the transit method. If the object passes in front of a star, it blocks a small amount of emitted light. If there’s a regular dip (regular in period and dip in brightness), then you’ve found an exoplanet.
This method is ideal because it can give a good estimate on planet size (by relative amount of light blocked) and also tells us how far away the planet is from it’s host star.
However, only close planets can be detected. The star needs to be constantly measured for multiple regular dips, so looking at a planet with an orbit of 248 years (Pluto) would be impossible to do. There’s so many things that could block starlight, so a one-off dip in brightness is not enough to detect a planet.
If Earth is facing the star’s orbital plane from a bird’s eye view, we can never detect any dips in brightness from planets.
The rarest and most difficult of all: actually seeing the planets with a telescope.
The very close planet will be indistinguishable from the star, but a far away planet will not have enough incident light from the star to see it. For this reason, seeing the planets with a telescope is incredibly difficult.
If the star is too bright, planets are difficult to distinguish. Brown dwarfs tend to be the best targets.
My favourite solar system (well it’s not really solar, but never mind) is HR8799. You can really see the individual planets around the star. This is what a direct image usually looks like. The star has been covered, otherwise the light would be overpowering.