For more information regarding Bristol-Myers Squibb Clinical Trial participation, please visit www.BMSStudyConnect.com Inclusion Criteria: - Participants must have metastatic colorectal or pancreatic cancer - Eastern Cooperative Oncology Group (ECOG) performance status of ≤1 - Ability to swallow pills or capsules - All participants will be required to undergo mandatory pre and on-treatment biopsies - Adequate marrow function - Adequate other organ functions - Ability to comply with study visits, treatment, procedures, PK and PD sample collection, and required study follow-up Exclusion Criteria: - Histology other than adenocarcinoma (neuroendocrine or acinar cell) - Suspected, known, or progressive CNS metastases (Imaging required only if participants are symptomatic) - Participants with active, known or suspected autoimmune disease - Participants with a condition requiring systemic treatment with either corticosteroids (> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study treatment administration - Interstitial lung disease that is symptomatic or may interfere with the detection or management of suspected treatment-related pulmonary toxicity - Prior treatment with CCR2 and/or CCR5 inhibitors, PD-1, PD(L)-1 or CTLA-4 antibodies - History of allergy to study treatments or any of its components of the study arm that participant is enrolling Other protocol defined inclusion/exclusion criteria could apply