Guide: Upgrading DSMR-reader v4.x to v5.x

DSMR-reader v5.x is backwards incompatible with 4.x. You will have to manually upgrade to make sure it will run properly.

List of changes

Danger

See the changelog, for v5.x releases and higher. Check them before updating!

  • ✋ Do not upgrade if you run PostgreSQL 9.x or below. Upgrade PostgreSQL first.

  • ✋ Do not upgrade if you run InfluxDB 1.x. Upgrade to InfluxDB 2.x first.

  • ✋ Do not upgrade if you run MySQL 5.6 or below. Upgrade MySQL first.

  • ℹ️ This upgrade will require you to run (or upgrade to) Python 3.7 or higher. Upgrade steps later below.

Docker

Attention

Docker users, see the changelog for env var changes!

If you’re using Docker, you can probably just install the v5.x version of the Docker container without having to perform any of the steps below.

1. Update to the latest v4.x version

Update to v4.20 to ensure you have the latest v4.x version.

2. Relocate to github.com/dsmrreader/

Over a year ago the DSMR-reader project was moved to https://github.com/dsmrreader.

Execute the following:

sudo su - dsmr
git remote -v

It should point to:

origin      https://github.com/dsmrreader/dsmr-reader.git (fetch)
origin      https://github.com/dsmrreader/dsmr-reader.git (push)

If not, update it and check again:

git remote set-url origin https://github.com/dsmrreader/dsmr-reader.git
git remote -v

Execute the following:

logout

3. Python version check

DSMR-reader 5.x requires Python 3.7 or higher.

Execute the following:

sudo su - dsmr
python3 --version

It should display the Python version. If you’re already running Python 3.7 (or higher), you can ignore the next section.

Execute the following:

logout

4. Python version upgrade (when running Python 3.6 or lower)

Warning

Only execute this section if you’re running DSMR-reader with Python 3.6 or lower!

There are several guides, depending on your OS. We assume Raspbian OS here.

Tip

You may consider upgrading to a higher Python version, e.g. Python 3.11, if possible for your OS.

Execute the following:

# Credits to Jeroen Peters @ issue #624
sudo apt-get install python3-dev libffi-dev libssl-dev -y
wget https://www.python.org/ftp/python/3.11.2/Python-3.11.2.tar.xz
tar xJf Python-3.11.2.tar.xz
cd Python-3.11.2
./configure --enable-optimizations --with-lto
make
sudo make install
sudo pip3 install --upgrade pip

Attention

Try running the command python3 --version to see if things worked out. If you’re getting any errors, do not continue with the upgrade.

The next thing you’ll absolutely need to do, is create a fresh database backup and store it somewhere safe.

Execute the following:

sudo su - dsmr
./manage.py dsmr_backup_create --full

If things went well, you should see a message like:

Created full backup: /home/dsmr/dsmr-reader/backups/manually/dsmrreader-postgresql-backup-Wednesday.sql.gz

Execute the following (your file name may differ!):

ls -lh /home/dsmr/dsmr-reader/backups/manually/dsmrreader-postgresql-backup-Wednesday.sql.gz

Make sure the file is of some (reasonable) size:

-rw-rw-r-- 1 dsmr dsmr 7.5M Dec 18 20:59 /home/dsmr/dsmr-reader/backups/manually/dsmrreader-postgresql-backup-Wednesday.sql.gz

Execute the following (your file name may differ!):

zcat /home/dsmr/dsmr-reader/backups/manually/dsmrreader-postgresql-backup-Wednesday.sql.gz | tail

Make sure the output ends with:

--
-- PostgreSQL database dump complete
--

Execute the following:

logout

5. Upgrade to DSMR-reader v5

Danger

Reminder

See the changelog, for v5.x releases and higher. Check them before updating!

  • ✋ Do not upgrade if you run PostgreSQL 9.x or below. Upgrade PostgreSQL first.

  • ✋ Do not upgrade if you run InfluxDB 1.x. Upgrade to InfluxDB 2.x first.

  • ✋ Do not upgrade if you run MySQL 5.6 or below. Upgrade MySQL first.

Install Python venv:

sudo apt-get install python3-venv

Install libopenjp2-7-dev as well, to prevent as possible error later:

# "ImportError: libopenjp2.so.7: cannot open shared object file: No such file or directory"

sudo apt-get install libopenjp2-7-dev

Stop DSMR-reader:

sudo supervisorctl stop all

Disable v4.x virtualenv:

sudo su - dsmr
deactivate
mv ~/.virtualenvs/ ~/.old-v4-virtualenvs

Create new v5.x virtualenv:

python3 -m venv ~/dsmr-reader/.venv/

Remove the following line from /home/dsmr/.bashrc:

# (feel free to use "nano" instead or whatever you'd like)
vi ~/.bashrc

Remove this line (if set):

# Probably on the bottom of the file.
source ~/.virtualenvs/dsmrreader/bin/activate

Add this line instead:

source ~/dsmr-reader/.venv/bin/activate

Update DSMR-reader codebase:

git fetch
git checkout -b v5 origin/v5

# Make sure you're at v5 now:
git branch

git pull

Install dependencies:

source ~/dsmr-reader/.venv/bin/activate

pip3 install pip --upgrade
pip3 install -r ~/dsmr-reader/dsmrreader/provisioning/requirements/base.txt

Tip

Does it fail with:

The headers or library files could not be found for jpeg,
a required dependency when compiling Pillow from source.

Try installing libjpeg-dev:

logout
sudo apt-get install libjpeg-dev

sudo su - dsmr
pip3 install -r ~/dsmr-reader/dsmrreader/provisioning/requirements/base.txt

Rename any legacy setting names in .env you find (see below):

# (feel free to use "nano" instead or whatever you'd like)
vi ~/dsmr-reader/.env

If you find any listed on the left hand side, rename them to the one on the right hand side:

# Core env vars/settings
SECRET_KEY       ️                 ➡️   DJANGO_SECRET_KEY
DB_ENGINE        ️                 ➡️   DJANGO_DATABASE_ENGINE
DB_NAME          ️                 ➡️   DJANGO_DATABASE_NAME
DB_USER          ️                 ➡️   DJANGO_DATABASE_USER
DB_PASS          ️                 ➡️   DJANGO_DATABASE_PASSWORD
DB_HOST          ️                 ➡️   DJANGO_DATABASE_HOST
DB_PORT          ️                 ➡️   DJANGO_DATABASE_PORT
CONN_MAX_AGE     ️                 ➡️   DJANGO_DATABASE_CONN_MAX_AGE
TZ               ️                 ➡️   DJANGO_TIME_ZONE
DSMR_USER        ️                 ➡️   DSMRREADER_ADMIN_USER
DSMR_PASSWORD    ️                 ➡️   DSMRREADER_ADMIN_PASSWORD

Check DSMR-reader:

./manage.py check

It should output something similar to: “System check identified no issues (0 silenced).”

Tip

Does it fail with:

ImportError: libopenjp2.so.7: cannot open shared object file: No such file or directory

Make sure you’ve installed libopenjp2-7-dev above:

logout
sudo apt-get install libopenjp2-7-dev

sudo su - dsmr
./manage.py check

Execute:

./manage.py migrate

Execute:

logout

Attention

Note: This may revert any customizations you’ve done yourself, such as HTTP Basic Auth configuration.

Update Nginx config:

sudo cp /home/dsmr/dsmr-reader/dsmrreader/provisioning/nginx/dsmr-webinterface /etc/nginx/sites-available/
sudo ln -s -f /etc/nginx/sites-available/dsmr-webinterface /etc/nginx/sites-enabled/

Reload Nginx:

sudo nginx -t
sudo systemctl reload nginx.service

Update Supervisor configs:

sudo cp /home/dsmr/dsmr-reader/dsmrreader/provisioning/supervisor/dsmr_datalogger.conf /etc/supervisor/conf.d/
sudo cp /home/dsmr/dsmr-reader/dsmrreader/provisioning/supervisor/dsmr_backend.conf /etc/supervisor/conf.d/
sudo cp /home/dsmr/dsmr-reader/dsmrreader/provisioning/supervisor/dsmr_webinterface.conf /etc/supervisor/conf.d/

Reload Supervisor configs:

sudo supervisorctl reread
sudo supervisorctl update

Start DSMR-reader:

sudo supervisorctl start all

6. Deploy

Finally, execute the deploy script:

sudo su - dsmr
./deploy.sh

Great. You should now be on v5.x!

7. Situational: Reconfigure InfluxDB

If you happened to use DSMR-reader export to InfluxDB previously, you must reconfigure it accordingly in order to get it working again. It has been disabled automatically as well.

Hint

Where the previous version utilized usernames, passwords and databases, it now connects using organizations, API tokens and buckets.

8. Situational: Reconfigure Dropbox

If you happened to use DSMR-reader’s Dropbox sync for backups, you must reconfigure it accordingly in order to get it working again.

9. Situational: Remote datalogger env vars

The following remote datalogger script settings were renamed as well, but you’ll only need to change them if you use and update the remote datalogger script as well. E.g. when running it in Docker:

DATALOGGER_INPUT_METHOD            ️➡️   DSMRREADER_REMOTE_DATALOGGER_INPUT_METHOD
DATALOGGER_SERIAL_PORT             ️➡️   DSMRREADER_REMOTE_DATALOGGER_SERIAL_PORT
DATALOGGER_SERIAL_BAUDRATE         ➡️   DSMRREADER_REMOTE_DATALOGGER_SERIAL_BAUDRATE
DATALOGGER_API_HOSTS    ️           ➡️   DSMRREADER_REMOTE_DATALOGGER_API_HOSTS
DATALOGGER_API_KEYS    ️            ➡️   DSMRREADER_REMOTE_DATALOGGER_API_KEYS
DATALOGGER_TIMEOUT    ️             ➡️   DSMRREADER_REMOTE_DATALOGGER_TIMEOUT
DATALOGGER_SLEEP    ️               ➡️   DSMRREADER_REMOTE_DATALOGGER_SLEEP
DATALOGGER_MIN_SLEEP_FOR_RECONNECT ➡️   DSMRREADER_REMOTE_DATALOGGER_MIN_SLEEP_FOR_RECONNECT