to have been released and gone but they are all there. I would appreciate it a lot if someone can point it to me where am I keeping a strong reference or if there is a circular referencing issue? or perhaps something else?

You have two strong references to your views, your custom property and the view hierarchy reference established when you call addSubview. When you remove the view from the view hierarchy, your class, itself, still has its strong reference to it.

You could solve this by making your reference optional, and when you call removeFromSuperview, also manually set your reference to nil. Or, perhaps easier, you might resolve this by using weak references, letting the view hierarchy maintain the strong references for you. And because your custom property is weak, when you remove it from the view hierarchy (thus eliminating the only strong reference to it), your weak reference will automatically become nil:

class FirstView: UIView {
weak var secondView: SecondView? // note the `weak` reference, which is obviously an optional
//sometime later by clicking on a button
func doSomething() {
let subview = SecondView()
subview.backgroundColor = .red
self.addSubview(subview)
secondView = subview
}
// a button will be tapped to remove secondView;
// later will be called at some point upon tapping ...
func removingSecondViewByTapping() {
secondView?.removeFromSuperview()
}
}