The olive is an oddity - a fruit he is mostly raised for the oil in its flesh. (There is also a high-quality oil in its seed, which is where you find the oil in most plants.) For this reason, and because it is bitter, we do not think of it as a fruit.