The public interface right now makes it impossible to implement dynamic_pointer_cast, const_pointer_cast, static_pointer_cast, and reinterpret_pointer_cast for deferred pointers.

The deferred_ptr's converting constructor makes it possible to covert between any pointers that for which is_convertible<T,U> is true but that is not enough to do all the common casts. The converting constructor could be made private with the enable_if removed and then the casts could implemented as friend functions?