Why Java compiler restricts adding even the right type of element to the List when using wildcard with extends

I understand the PECS rule with regards to extend and super from other posts, but I am a bit confused about the restrictions below code

List<Integer> integerList = new ArrayList<>();
integerList.add(1);
// integerList.add(3.14); => understandably this will not compile,
// as we cannot add double to a list of Integer

List<? extends Number> numberList = integerList;
// numberList.add(3); => will not compile!
// Why the compiler cannot infer from the above line that
// adding an Integer to a list of Integer is OK, the same
// way it inferred that line 3 was not OK?

Are there any reasons why the compiler would impose the restriction on line 5, given that we are just trying to add an Integer to a list of Integers and there is sufficient information available to the compiler?