Custom attribute shipping address update to null after invoice or shipment

I have a plugin with multiple functionalities and one of them is to add a custom attribute on address and on checkout I am updating the attribute value via AJAX with on click javascript. The value is in DB and also after order is placed is moved correctly from quote to order table, but when I am invoicing the order or doing a shipment, the attribute value is NULL. Based on debug, this value is not from my code as I’ve alter the code and add different values instead of NULL to check.

What can be the reason for this?

In order to save attribute from quote to order, I am using an event

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Event/etc/events.xsd">
    <event name="sales_order_save_after">
        <observer name="save_data" instance="TestTestObserverAddPudoFieldValue"  />
    </event>
</config>

With observer

<?php
namespace TestTestObserver;
class AddPudoFieldValue implements MagentoFrameworkEventObserverInterface
{
    /**
     * @var MagentoQuoteModelQuoteFactory
     */
    protected $quoteFactory;

    public function __construct(
        MagentoQuoteModelQuoteFactory $quoteFactory
    )
    {
        $this->quoteFactory = $quoteFactory;
    }

    /*
     * Fetch Quote Factory and add field to this function
     */
    public function execute(MagentoFrameworkEventObserver $observer)
    {
        $order = $observer->getEvent()->getOrder();
        $quote = $this->quoteFactory->create()->load($order->getQuoteId());
        if ($quote) {
            $pudoValue = $quote->getShippingAddress()->getTestPudoId();
            $order->getShippingAddress()->setTestPudoId($pudoValue);
            $order->save();
        }
    }
}

I have extension_attributes.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Api/etc/extension_attributes.xsd">
    <extension_attributes for="MagentoQuoteApiDataAddressInterface">
        <attribute code="test_pudo_id" type="int"/>
    </extension_attributes>
</config>

Saving from Quote to order is working. I’ve checked the DB and I see the value there.

My issue now is that when doing invoice/shipment on order, I am loosing the value from DB