Inclusion Criteria: - Localized invasive breast ductal carcinoma, confirmed by the local pathologist, that includes the following combined primary tumor and clinical node (cN) categories: T1c (tumor size = 2 cm)-T2 (tumor size > 2 cm), cN1-N2 OR T3-T4, cN0-cN2. Note: Axillary lymph node status must be assessed by fine needle biopsy or core biopsy. - Estrogen receptor-positive (ER+) breast cancer (BC) and with or without progesterone receptor (PgR) expression (determined on the most recently analyzed tissue sample, tested locally, and confirmed by the central laboratory), as defined in the relevant American Society of Clinical Oncology (ASCO)- College of American Pathologists (CAP) Guidelines. - Human epidermal growth factor receptor 2 (HER2-) BC tested in the local laboratory, defined as a negative in situ hybridization test or an immunohistochemistry (IHC) status of 0, 1+, or 2+. - Tumor Grade 3 of ductal histology, Or Tumor Grade 2 of ductal histology having an ER expression level percentage between 1-10% - Must agree to provide primary breast tumor tissue at baseline and at surgery - Must be deemed eligible for surgery - Males and females must agree to follow specific methods of contraception, if applicable, while participating in the trial - Must have an Eastern Cooperative Oncology Group (ECOG) scale performance status of 0 or 1 Exclusion Criteria: - Breastfeeding, pregnant, or expecting to conceive or father children within the projected duration of the study, starting with the screening through 12 months for participants who receive cyclophosphamide, or 6 months for participants who do not receive cyclophosphamide, after the last dose of study treatment - Prior treatment with chemotherapy, endocrine therapy (ET), targeted therapy, and/or radiation therapy for the currently diagnosed breast cancer prior to enrollment - Prior treatment with an anti-PD-1, anti-PD-L1, anti-PD-L2, or anti-CTLA-4 antibody, or any other antibody or drug specifically targeting T-cell co-stimulation or checkpoint pathways - Significant cardiovascular disease such as left ventricular ejection fraction (LVEF) < 50% at baseline as assessed by echocardiography (ECHO) or multigated acquisition (MUGA) scan performed at screening, or Class III or IV myocardial disease as described by the New York Heart Association - History of ipsilateral invasive BC, regardless of treatment, ipsilateral ductal carcinoma in situ treated with radiation, or contralateral invasive BC, at any time - Definitive clinical or radiologic evidence of metastatic disease - Multicentric BC (the presence of > 1 tumor in different quadrants of the breast) - Bilateral invasive BC Other protocol-defined inclusion/exclusion criteria apply