Get customer orders excluding ones with a specific order item

I want to retrieve all customer orders’ except the ones that have a specific item (walle product)

Currently, I get all orders that have the item and then filter them from orderCollection

$orderItemCollection = $this->orderItemCollectionFactory->create();
$orderItemCollection->addFieldToFilter('sku', Data::WALLET_PRODUCT_SKU);

$walletOrders = [];
foreach ($orderItemCollection->getItems() as $orderItem) {
    $walletOrders[] = $orderItem->getOrderId();
}
$orderCollection->addFieldToFilter('entity_id', ['nin' => $walletOrders]);

Is there a better way to do this without getting order IDs first?