This does not look like European olive - Olea europaea which usually has white bark at this age and slightly wider, more grey leaves. It could be African olive (Olea europaea subspecies cuspidata)which I believe is an invasive weed in the Sydney basin.