AusBonsai

I picked this up around the corner from home as a guy was re-doing his garden.

Should I chop it off even more?

If you just kept the smaller of the two trunks it would improve the taper and you could eliminate the bit of inverse taper. But that is the boring version and hopefully you will get plenty of other suggestions.

I would just leave it as is for now. The tree has gone through a lot so far. Now needs time to start recovery. There is plenty of time to consider and make any further reduction when the new shoots and roots are growing strongly.
It is possible to push a tree too far too fast.

I agree with the assessment that the taller trunk is too thick, straight and has little taper and probably will need reduction at some stage. Just where and when will depend on the shoots that sprout from the trunks. It would be a shame to cut off one only to find that the other one did not sprout.

Remember you may want to put some carving into that larger trunk in years to come. If this is a fruiting olive it will not be as rewarding as if a wild olive with smaller leaves
You could also split the 2 trunks completely having 2 smaller bases and later carving the cut sides

Sounds like I'll just leave it and see what happens for now.

It is the fruiting olive, very common around houses in this area.

My advice is to use it later when it has shoots and roots next year to practice carving
And join the Bonsai Society of WA so you can join in the Spring Wild olive dig early Aug usually and gain more suitable stock to work with