This is a small refactoring that removes the main ejector cache. The
cache is now tracked by slots and ejector components.
It avoids a large array allocation and many small object allocations,
but adds many small array allocations. It's net neutral for performance.