Well, actually you don't have to delete the Limited Access itself because it only shows you that some other levels of access are assigned to this user somewhere inside this container (site/web/list/folder) in another container that does not inherit permissions from its parent.