Yes you need to SAVE them as they are simply HTML layers saved over the screen.

So listen for the annotation events and save the annotations to a file,
ex.
unsubscribeAnnotationEvents = annotationApi.onAnnotationEvent((event) => {
if (event.committed && (event.type === 'create' || event.type === 'update')) {
const data = roundCoords(event, ['x', 'y', 'width', 'height']);
console.log(annotation save: , JSON.stringify(data));
saveAnnotation(fileId, data); <-- a function YOU build to save it to a file, database, innodb, etc.
} else if (event.committed && event.type === 'delete') {
deleteAnnotation(event.annotation.id); <!-- each annotation generates their own id, so you tracked that in the saves, now you can go and delete whichever corresponds to the delete.
}

Finally on the loading of the PDFView you need to import all those saved annotations:
so my loader function is called getAnnotations, then you see I call the annotationApi (see their headless sample code here) and their method importAnnotations() with an array of all of those 'events' generated on their onAnnotationEvent.

getAnnotations(fileId).then((annotations) => {
if (annotations?.length) annotationApi.importAnnotations(annotations);
});

Good luck! I had to figure this stuff out myself too as the poor guy building this clearly is so busy we don't have documentation for all of this yet!