We enjoyed two wonderful gardens while in Victoria; the famous Buchart Gardens and its nextdoor neighbor, The Butterfly Garden.
We used the HAL tour just because it went to both, and it seemed the easiest way to get there. There is public transportation as well and probably other tour operators are available too, we've been told. These two gardens are essentially on adjacent properties although the lengthy driveways might make walking between them consume more time than you want to spend doing that, if your time is limited. While the butterflies are confined to a fairly small area, the Butchart Gardens are very large and even just a quick stroll through the highlights would probably take at least a couple of hours.