If we want to stop this, the only solution is to recycle the plastic we use! Obviously it would be a lot better if companies started to use other types of packaging for their products, but it is generally more expensive and less durable, so that’s why they go for the easy choice and make plastic packaging.

Even with recycling, one of the issues is with the little plastic bits used as feedstock for manufacturing. These are called 'nurdles' and, as you might imagine for a tiny & light thing, it is easy for them to accidentally enter the environment. They are a significant part of plastic pollution.

Between nurdles and, frankly, the impossibility of getting everyone to recycle, more must be done. As you pointed out, expense and durability of non-plastics are issues, but biodegradable plastics are an increasingly promising option. BUT it's important for us as consumers to know not all degradable plastics are the same with respect to how they degrade.

Going roughly from worst to best (terminology my own, as the industry standards are in flux):

1. Disintegratable These are generally the plastics we're used to, but with weak points inserted throughout the polymer. This can be things like starch or UV reactive compounds. They certainly do break into smaller and smaller pieces, but generally the degree of substitution is not sufficient to produce small enough fragments to actually biodegrade. What you actually get are millions of microscopic bits of plastic. Note that the UV sensitive plastic will not degrade while in sediment, underwater, or in a fish.

2. Compostable/Oxobiodegradable In some cases, these are similar disintegratable plastics, but with a different degradable substitution. In other cases, these are different polymers from traditional plastics. However, in both situations there is an added requirement that oxygen is present for degradation to occur. Often the assumption is lots of oxygen, and increased temperature. This is why plastics which are technically compostable are really only meant to be composted at the industrial scale and may never break down in someone's backyard pile. As you can imagine, they have many of the same issues as disintegratable plastics with respect to oceanic pollution.

3. Biodegradable These truly mineralize (become CO₂, energy, and biomass) over time under aerobic and anaerobic conditions. They are almost all alternative polymers (although some are not new just not as popular as PE and PS, like polycaprolactone(PCA) and polyvinyal acetate). Unfortunately they are also, unfortunately, moreexpensive to produce, due to both the price of feedstock and the fact that we have tons of infrastructure already set up for traditional polymers. Beyond that, many don't have quite the same material properties, PCA, for example, can't replace plastic coffee cups because of its low melting point.