For more information regarding Bristol-Myers Squibb Clinical Trial participation, please visit www.BMSStudyConnect.com Inclusion Criteria: - Must have metastatic colorectal or pancreatic cancer - Eastern Cooperative Oncology Group (ECOG) performance status of ≤1 - Ability to swallow pills or capsules - Required to undergo mandatory pre and on-treatment biopsies - Adequate marrow function - Adequate other organ functions - Ability to comply with study visits, treatment, procedures, pharmacokinetic (PK) and pharmacodynamic (PD) sample collection, and required study follow-up Exclusion Criteria: - Histology other than adenocarcinoma (neuroendocrine or acinar cell) - Suspected, known, or central nervous system (CNS) metastases (Imaging required only if participants are symptomatic) - Active, known or suspected autoimmune disease - Condition requiring systemic treatment with either corticosteroids (> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study treatment administration - Interstitial lung disease that is symptomatic or may interfere with the detection or management of suspected treatment-related pulmonary toxicity - Prior treatment with cysteine-cysteine chemokine receptor 2 (CCR2) and/or cysteine-cysteine chemokine receptor 5 (CCR5) inhibitors, programmed death-1 receptor (PD-1), programmed death-ligand 1 [PD(L)-1] or cytotoxic T-lymphocyte antigen-4 (CTLA-4) antibodies - History of allergy to study treatments or any of its components of the study arm that participant is enrolling Other protocol-defined inclusion/exclusion criteria apply