Undefined array key 3796 in Eav.php on line 604

I am getting this error, when I try to open a product in the admin:

Undefined array key 3796 in vendor/magento/module-catalog/Ui/DataProvider/Product/Form/Modifier/Eav.php on line 604

The code in Eav.php on line 604 looks like this:

$attributeGroupId = $mapAttributeToGroup[$attribute->getAttributeId()];

So I guess $attribute->getAttributeId() returns a number which is not in $mapAttributeToGroup. I var_dumped the attribute code to see if it exists and it does, so the attribute wasnt deleted, its there as it has been all the time.

When I, just for testing, put in a continue for ID 3796, it gives another number for another attribute, seems to be something general.

Ideas?