Upgrade on Docker
To run OpenMetadata with Docker, you can simply download the docker-compose.yml
file. Optionally, we added some
Named Volumes to handle data persistence.
Note
You can find more details about Docker deployment here
Below we have highlighted the steps needed to upgrade to the latest version with Docker. Make sure to also look here for the specific details related to upgrading to 0.13
Warning
It is advised to go through openmetadata release notes before starting the upgrade process.
Upgrade from 0.13.1 to 0.13.2
Upgrading from 0.13.1 to 0.13.2 can be done easily as version 0.13.1 compose files already provided volumes for the databases.
Let's go through the required steps:
1. Backup 0.13.1 data
- Make sure your instance is connected to the Database server
- Create a virtual environment to install an upgraded
metadata
version to run the backup command:python -m venv venv
source venv/bin/activate
pip install openmetadata-ingestion~=0.13.2
- Validate the installed
metadata
version withpython -m metadata --version
, which should tell us that we are indeed at 0.13.2. Notice thepython -m metadata
vs.metadata
. - Run the backup using the updated
metadata
CLI: if using Postgres:python -m metadata backup -u openmetadata_user -p openmetadata_password -H mysql -d openmetadata_db --port 3306
python -m metadata backup -u openmetadata_user -p openmetadata_password -H postgresql -d openmetadata_db --port 5432 -s public
- This will generate the .sql file which can be used for the backup
In our case, the backup file was named
openmetadata_202212201528_backup.sql
. You can copy the name from the backup command output.
2. Update the docker compose file
- Stop the running compose deployment with
docker compose down
. - On the compose file we ran the 0.13.1 version, update the image tag in the
ingestion
andopenmetadata-server
to 0.13.2. E.g.,image: openmetadata/server:0.13.2
. - Start the updated compose file.
- Run the reindex from the UI.
Upgrade from 0.12.3 to 0.13.2
Your production deployment should go from stable version to stable version. This translated to moving from 0.12.3 to 0.13.2 to get the latest stable OpenMetadata release.
Database volumes were just introduced in the 0.13.0 feature release. This means that when moving from 0.12.3 to 0.13.2, we'll need to do some extra steps to ensure that all your metadata is safe and sound. In this section, we'll guide you step by step on what you'll need to do to:
- Backup your data,
- Add volumes and port mapping to the databases and restore your data,
- Start OpenMetadata with the 0.13.2 release.
These steps are based on MySQL, but are analogous for Postgres.
Please, validate the process in your development or staging environment first before going into PROD.
Note
This guide is specific to installations using the default docker compose file we shipped on the 0.12.3 release. The whole process is about making sure that all the data will be properly backed up considering that there is no port mapping or volumes available.
If your docker compose already had volumes for the database, then upgrading the version of OpenMetadata and the Ingestion container should be good to go (as shown in step 4 below).
1. Backup 0.12.3 data
- Get inside the
openmetadata_ingestion
container withdocker exec -it openmetadata_ingestion bash
. - Create a virtual environment to install an upgraded
metadata
version to run the backup command:python -m venv venv
source venv/bin/activate
PIP_USER=false pip install openmetadata-ingestion~=0.13.2
- Validate the installed
metadata
version withpython -m metadata --version
, which should tell us that we are indeed at 0.13.2. Notice thepython -m metadata
vs.metadata
. As we're inside the container, the raw metadata command will use the system-wide one, which is at 0.12.3. - Run the backup using the updated
metadata
CLI: if using Postgres:python -m metadata backup -u openmetadata_user -p openmetadata_password -H mysql -d openmetadata_db --port 3306
python -m metadata backup -u openmetadata_user -p openmetadata_password -H postgresql -d openmetadata_db --port 5432 -s public
- From outside the container, copy the backup file to safety. For example
docker cp openmetadata_ingestion:/opt/airflow/openmetadata_202212201528_backup.sql .
In our case, the backup file was namedopenmetadata_202212201528_backup.sql
. You can copy the name from the backup command output.
2. Add volumes and port mapping
This step is only required if you have not done that yet.
- Stop docker compose
- Update the
mysql
service in the compose file so that it looks like: Note how we added themysql: container_name: openmetadata_mysql image: openmetadata/db:0.13.2 restart: always environment: MYSQL_ROOT_PASSWORD: password expose: - 3306 ports: - "3306:3306" volumes: - ./docker-volume/db-data:/var/lib/mysql networks: - app_net healthcheck: test: mysql --user=root --password=$$MYSQL_ROOT_PASSWORD --silent --execute "use openmetadata_db" interval: 15s timeout: 10s retries: 10
ports
andvolumes
sections. If you're using postgres, these would look like:ports: - "5432:5432" volumes: - ./docker-volume/db-data-postgres:/var/lib/postgresql/data
- Start docker compose with
docker compose up
on the updated file.- This will run a clean database instance with ports and volumes.
- The OpenMetadata server will run the 0.12.3 migrations to populate the starting tables.
3. Restore the backup
We will now use the backup generated in the previous step to repopulate the database.
- On your laptop/VM, prepare a virtual environment and install
openmetadata-ingestion==0.13.2
. It is important that the instance has access to the database.python -m venv venv
source venv/bin/activate
pip install openmetadata-ingestion~=0.13.2
- Validate the metadata version with
metadata --version
. it should be 0.13.2. - Run the restore with your file name
if using Postgres:metadata restore -H localhost -u openmetadata_user -p openmetadata_password -d openmetadata_db --port 3306 --input openmetadata_202212201528_backup.sql
metadata restore -H localhost -u openmetadata_user -p openmetadata_password -d openmetadata_db --port 5432 -s public --input openmetadata_202212201528_backup.sql
- Stop the docker compose
4. Upgrade OpenMetadata versions
- On the compose file we previously updated the database settings, update the image tag in the
ingestion
andopenmetadata-server
to 0.13.2. E.g.,image: openmetadata/server:0.13.2
. - Start the updated compose file.
- Run the reindex from the UI.
Now you should still have all your data with OpenMetadata version 0.13.2.
Upgrade ingestion patch versions
During the release lifespan we may publish new patch versions of openmetadata-ingestion
. If you deployed
the ingestion container and require one of the fixes or improvements from a new patch release, there's usually no need
to re-deploy the full ingestion container.
Note
Note that this process will only work if we are moving from PATCH versions. For example: 0.13.1.1
-> 0.13.1.2
.
This method won't work when upgrading from 0.13.1.X
-> 0.13.2.X
, as that will also require to upgrade the
server version.
The steps to follow are:
- Connect to the ingestion container. If using our docker compose files or
metadata docker
CLI, this translates todocker exec -it openmetadata_ingestion bash
. - Validate your
metadata
version viametadata --version
. You will get back something like:metadata 0.13.1.5 from /home/airflow/.local/lib/python3.9 (python 3.9)
- Upgrade the
openmetadata-ingestion
package viapip install "openmetadata-ingestion==0.13.1.X"
, for example,pip install "openmetadata-ingestion==0.13.1.7"
. You can find the list of all released versions of theopenmetadata-ingestion
package here. - Exit the container by typing
exit
. - Restart the ingestion container with
docker restart openmetadata_ingestion
. This will need a few minutes to to stop the container and start it again. Now, Airflow will start with the upgradedmetadata
version. - Connect to the ingestion container and validate the
metadata
version:docker exec -it openmetadata_ingestion bash
metadata version
: where we expect to get the same version that was previously installed.
Troubleshooting
metadata openmetadata-imports-migration
Permission Denied when running If you have a Permission Denied
error thrown when running metadata openmetadata-imports-migration --change-config-file-path
you might need to change the permission on the /opt/airflow/dags
folder. SSH into the ingestion container and check the permission on the folder running the below commands
ls -l /opt/airflow
ls -l /opt/airflow/dags
both the dags
folder and the files inside dags/
should have airflow root
permission. if this is not the case simply run the below command
chown -R airflow:root /opt/airflow/dags
Broken DAGs can't load config file: Permission Denied
You might need to change the permission on the /opt/airflow/dag_generated_config
folder. SSH into the ingestion container and check the permission on the folder running the below commands
ls -l /opt/airflow
ls -l /opt/airflow/dag_generated_config
both the dags
folder and the files inside dags/
should have airflow root
permission. if this is not the case simply run the below command
chown -R airflow:root /opt/airflow/dag_generated_config