Background: The efficacy and safety of human epidermal growth factor receptor 2 (HER2)-targeted therapy and androgen deprivation therapy (ADT) for locally advanced or recurrent or metastatic (LA/RM) salivary duct carcinoma (SDC) have been reported in prospective studies. However, the survival benefit of these therapies to conventional therapy remains controversial, and whether HER2-targeted therapy or ADT should be chosen in HER2- and androgen receptor (AR)-positive SDC patients remains unknown. Methods: Overall, 323 LA/RM SDC patients treated at seven institutions between August 1992 and June 2020 were retrospectively enrolled. The primary aim was to analyze the effect of HER2-targeted therapy and ADT on overall survival from the diagnosis of LA/RM disease to death from any cause (OS1). The secondary indicators included the overall response rate (ORR), clinical benefit rate (CBR), overall survival from therapy initiation for LA/RM disease (OS2), progression-free survival (PFS), time to second progression (PFS2), duration of response (DoR), and duration of clinical benefit (DoCB) of HER2-targeted therapy or ADT as first-line therapy for HER2-positive/AR-positive SDC. Results: Patients treated with HER2-targeted therapy or ADT had longer OS1 than those treated without these therapies (Median OS1: historical control, 21.6 months; HER2-targeted therapy, 50.6 months; ADT, 32.8 months; HER2-targeted therapy followed by ADT, 42.4 months; and ADT followed by HER2-targeted therapy, 45.2 months, p < 0.001). Among HER2-positive/AR-positive SDC patients, although HER2-targeted therapy had better ORR, CBR, and PFS than those of ADT as first-line therapy, we found no significant differences between HER2-targeted therapy and ADT regarding OS2, PFS2, DoR, and DoCB. Conclusion: Patients treated with HER2-targeted therapy and ADT showed longer survival in LA/RM SDC. HER2-targeted therapy can be recommended prior to ADT for HER2-positive/AR-positive SDC. It is warranted to establish a biomarker that could predict the efficacy of clinical benefit or better response in ADT.