Filter by attribute is slow for customer collection using addAttributeToFilter() function

I have created a custom attribute called username for Customer which is searchable in admin grid. But whenever I search by username for SSO it takes much time (2-3 sec). Below are the code:

$objectManager = MagentoFrameworkAppObjectManager::getInstance();
        $customerObj = $objectManager->create('MagentoCustomerModelResourceModelCustomerCollection');
        $collection = $customerObj->addAttributeToSelect('entity_id')
                    ->addAttributeToFilter('username', $username)
                    ->load();

Below is the query from sql log:

SELECT `e`.*, `at_username`.`value` AS `username` FROM `customer_entity` AS `e` LEFT JOIN `customer_entity_varchar` AS `at_username` ON (`at_username`.`entity_id` = `e`.`entity_id`) AND (`at_username`.`attribute_id` = '181') WHERE (at_username.value = 'vaibhav')