$category->getStoreId() returns the current store instead of the store that it actually belongs to

I have a category 1 which belongs to store A and category 2 which belongs to store B. When I am in store A and run the query $category->getStoreId(), both categories returns A, and when I am in store B, both categories return B.

.....
MagentoCatalogModelCategoryRepository $categoryRepository,
......

$categoryIds = $product->getCategoryIds();

foreach($categoryIds as $categoryId) {
   $category = $categoryRepository->get($categoryId);
   echo $category->getStoreId();
} 

I tried everything using categorycollection, categoryfactory, and categoryrepository.
How do I get the store id of that category instead of current store id.
FYI, setStore, setProductStoreId(), addStoreFilter etc, none of them worked