This is a Phase I/Ib study testing the safety and preliminary efficacy of olaparib and vorinostat when used together in participants with relapsed/refractory and or metastatic breast cancer. Cancer cells grow in an uncontrolled manner and this causes damage to their DNA (genetic makeup). Cancer cells that cannot repair this damage will not survive and die. Unfortunately, cancer cells contain certain proteins whose job is to repair DNA damage. Poly (adenosine 5' diphosphoribose) polymerase (PARP) and histone deacetylase (HDAC) are two such proteins. Olaparib stops PARP from working, and vorinostat stops histone deacetylase from working. The use of olaparib and vorinostat together may better block the ability of cancer cells to repair their DNA damage. This may lead to even better killing of cancer cells. The study will be done in two parts. In part one of the study, different dose levels of olaparib and vorinostat will be tested in several study participants. This part of the study will allow us to see the doses of olaparib and vorinostat that can be used safely together in participants with relapsed/refractory and/or metastatic breast cancer. Up to 4 different dose levels will be studied. In part two of the study, the dose level of olaparib and vorinostat found to be the safest in the first part of the study will be tested. This part of the study will allow us to see how well relapsed/refractory and/or metastatic breast cancer responds to treatment with olaparib and vorinostat. Participants who received the dose level of olaparib and vorinostat found to be the safest in the first part of the study will also take part in part two of the study.