There are different causes for damage on avocado fruit. For help with identifying the cause of the problem, see the photos and links here: Avocado-Causes of Fruit Damage.
An avocado tree typically can produce up to about one million flowers but will typically only set about 100 to 200 fruit per tree. Or in other words, 1 fruit in 10,000. Sometimes they will set fruit but then drop them when they are pea to walnut size. Again this is typical.
To minimize fruit drop of good, fertilized fruit, avoid stressing the tree. That is, don't under or overwater the tree. There has been research in Israel which suggests that fruit retention is also facilitated when there are other avocado varieties present to provide cross-pollination and that these crossed fruit have a higher tendency to stay on the tree. We also recommend not to fertilize with nitrogen from about April through mid-June, or apply only very low amounts during this time.
If your tree has been dropping small size avocados and it appears as if the stems are being damaged somehow, you might suspect avocado thrips. Avocado thrips will feed on the stem and cause fruit drop. The pest damage and the natural fruit fall occur at the same time, so it is hard to distinguish the two unless an inspection of the fruit is made for thrips. Thrips would be found on the fruit under the callyx.
If the leaves and fruit of avocado have spots and discoloration, and there is some webbing on the underside of the leaves, it is possible that the tree has mites. Click to see photos of the damage caused by Persea mites and spider mites.
In the home garden, some mites can be tolerated. However, if numbers get too high, treat carefully with insecticidal oil (horticultural oil) and water. Be sure the spray gets on the undersides of leaves, where mites are located. Water alone applied with a high-pressure hose or sprayer will often control mites.
Mites have several natural enemies which often limit populations, including predatory mites. Avoid fertilizing infected trees as this can promote mite populations; avoid broad-spectrum insecticide treatments as this may kill natural enemies and lead to an increase in mite populations.