Many RNAs, proteins, and organelles are present in such low numbers per cell that random segregation of individual copies causes large "partitioning errors" at cell division. Even symmetrically dividing cells can then by chance produce daughters with very different composition. The size of the errors depends on the segregation mechanism: Control systems can reduce low-abundance errors, but the segregation process can also be subject to upstream sources of randomness or spatial heterogeneities that create large errors despite high abundances. Here we mathematically demonstrate how partitioning errors arise for different types of segregation mechanisms and how errors can be greatly increased by upstream heterogeneity but remarkably hard to avoid through controlled partitioning. We also show that seemingly straightforward experiments cannot be straightforwardly interpreted because very different mechanisms produce identical fits and present an approach to deal with this problem by adding binomial counting noise and testing for convexity or concavity in the partitioning error as a function of the binomial thinning parameter. The results lay a conceptual groundwork for more effective studies of heterogeneity among growing and dividing cells, whether in microbes or in differentiating tissues.