Product Filter Count Showing Wrong Count After Custom Product Collection – Magento Layered Navigation

I have applied custom filter on product collection to hide certain products from listing page. I have used below event for same

<event name="catalog_product_collection_load_after">
    <observer name="abc_productrestriction_collectionfilter" instance="AbcProductRestrictionObserverFrontendCatalogCollectionFilter" />
</event>

and in the observer, I am removing the product from collection using

public function execute(
        MagentoFrameworkEventObserver $observer
    ) {
        $productCollection = $observer->getEvent()->getCollection();
        $productCollection->load();
        foreach($productCollection as $key => $product) {
            if(!$this->isProductAllowed($product))
                $productCollection->removeItemByKey($key);
        }
        $productCollection->getSize();
        $productCollection->load();

        return $this;
    }

The products are removed correctly however, the filters and product count is layered navigation is not updated.

I also tried apply collection filter on Layer.php file and CollectionFilter.php in module

module-catalog
module-catalog-search
module-elastic-search

However, none of them are affecting layered navigation filter. Any help will be appreciated in guiding to filter product in correct way so it is affected at all places.

I also used below solution, and it works however, it doesn’t seem to be correct way to do it

Custom product collection filter doesnt work with Magento 2 elasticsearch ( but work with mysql)