##### home | writing | TIL | now | about

# Reorganising a Consul key-value store

by David Winterbottom on Monday, 27 March 2017
by David Winterbottom
on Monday, 27 March 2017

If your Consul key-value store is structured as:

    /
        A/
            X = 1
            Z = 2
            Y = 3
        C
        D

but you now realise you should have namespaced everything within `WEBSERVER/` (or something like that):

    /
        WEBSERVER/
            A/
                X = 1
                Z = 2
                Y = 3
            C
            D

then this Bash script will help you migrate:

    #!/bin/bash

    set -e  # Exit on error

    # Emit "key value" lines for all keys in Consul's KV store
    keys_and_values() {
        # Recursively fetch all values from Consul but exclude:
        # (a) those that end in / (as these are folders)
        # (b) those that start with WEBSERVER/ (as that's where we are migrating
        #     to).
        curl -s "localhost/v1/kv/?recurse" | jq -r '
            .[] |
            select(
                (.Key | endswith("/") | not) and
                (.Key | startswith("WEBSERVER") | not)
            ) |
            [.Key, " ", .Value] |
            add' | while read key b64value
        do
            # Consul's REST API returns values base64-encoded so we decode here.
            echo $key `echo "$b64value" | base64 -d`
        done
    }

    # Set a new value in Consul's KV store
    set_key() {
        key=$1
        value=$2
        curl -s -X PUT -d "$value" "localhost/v1/kv/$key" > /dev/null
    }

    migrate_to_webserver_namespace() {
        keys_and_values | while read key value
        do
            set_key "WEBSERVER/$key" "$value"
        done
    }

    migrate_to_webserver_namespace

This script uses Consul’s REST API and filters the results using `jq` 1 . It’s easily adapted to migrate key-value pairs between different namespaces.

1. I can never remember jq’s `select` syntax so this post is intended largely as a personal reference on how to do this. ↩︎

——————

Something wrong? Suggest an improvement or add a comment (see article history )
Tagged with: consul , bash , jq
Filed in: tips

Previous: Lies you're told on a software project
Next: Podcast.__init__ on Oscar

Copyright © 2005-2023 David Winterbottom
Content licensed under CC BY-NC-SA 4.0 .