You will notice it, eventually! A few years ago I was profiling some shadow volume extrusion code and it turned out that using std::vector's array operator was killing performance. Casting my array's front() to a raw pointer, and accessing that brought CPU hit in the internal loops from something like 80% of CPU time down to essentially nothing.

I'm not suggesting premature optimization, but keep in mind to shark your code now and then. You might write a correct but slow path and forget all about it until, 6 months from now when you're really stress testing your code, you wonder "Why is this so slow?"

What I'd do, personally, is let NSArray manage your storage while you're building or otherwise manipulating your data. But once your data's ready, copy it over to a raw array and let go of the NSArray.