Skip to content

Latest commit

 

History

History
300 lines (279 loc) · 8.48 KB

FeedHQ.md

File metadata and controls

300 lines (279 loc) · 8.48 KB

FeedHQ

FeedHQ is a web-based feed reader

GitHub

feedhq-pg-app-century

Migrating Feedhq app from PG to YB with Voyager Refer to repo for local setup in mac m1. As we have to change some requirements for python to run on local M1. https://github.com/yugabyte/feedhq-pg-app-century

Tech & versions used

  • Docker
  • Postgres docker image postgres:13
  • YugabyteDB docker image yugabytedb/yugabyte:2.23.0.0-b710
  • Yugabyte Voyager docker image yugabytedb/yb-voyager:1.8.2-rc2
  • ElasticSearch docker image elasticsearch:7.10.0
  • Redis
  • Django Setup- docker file is present Dockerfile

1. Run FeedHq on PG

  1. Getting the code:
git clone git@github.com:yugabyte/feedhq-pg-app-century.git
cd feedhq-pg-app-century
mkdir backup-dir
mkdir assess-dir
mkdir -p data/yb_data
mkdir -p postgres/data
  1. Build the docker image for django app:
docker build -t django_app:0.1 .
  1. Start services using docker-compose
docker-compose up -d
  1. Check all services are up and running fine
docker ps -a
feedhq dashboard: http://localhost:8001/login/
  1. Make sure databases setting in settings.py will look like this:
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': "feedhq",
        'USER': "feedhq",
        'PASSWORD': "feedhq",
        'HOST': "feedhq_postgres", # write postgres container name here
        'PORT': "5432",
        'OPTIONS': {
            'options': '-c timezone=UTC',
        },
    }
}
  1. Go to feedhq_postgres service and check feedhq database is created or not
docker exec -it feedhq-feedhq_postgres-1 bash
psql -U feedhq -d feedhq
#(if not created then create database feedhq)
  1. Migrate the db in django app
docker exec -it django_app bash
python -V 
# This should come as python3.6
python manage.py migrate
  1. Check ElasticSearch is running fine. Go to Go to UI http://localhost:9200/. Response will look like this:
    {
        "name": "1038ae0e61e5",
        "cluster_name": "docker-cluster",
        "cluster_uuid": "ekDZCVf8SUm0bL_XC6wLIQ",
        "version": {
        "number": "7.10.0",
        "build_flavor": "default",
        "build_type": "docker",
        "build_hash": "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
        "build_date": "2020-11-10T02:06:27.342163Z",
        "build_snapshot": false,
        "lucene_version": "8.7.0",
        "minimum_wire_compatibility_version": "6.8.0",
        "minimum_index_compatibility_version": "6.0.0-beta1"
        },
        "tagline": "You Know, for Search"
}
  1. Create Index for ElasticSearch
curl -X PUT "http://localhost:9200/feedhq" -H 'Content-Type: application/json' -d '{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  }
}'
  1. Add mapping in above feedhq index:
curl -X PUT "http://localhost:9200/feedhq/_mapping" -H 'Content-Type: application/json' -d '{
  "properties": {
    "timestamp": {
      "type": "date"
    },
    "id": {
      "type": "integer"
    },
    "title": {
      "type": "text"
    },
    "raw_title": {
      "type": "text"
    },
    "author": {
      "type": "text"
    },
    "content": {
      "type": "text"
    },
    "link": {
      "type": "keyword"
    },
    "guid": {
      "type": "keyword"
    },
    "read": {
      "type": "boolean"
    },
    "read_later_url": {
      "type": "keyword"
    },
    "starred": {
      "type": "boolean"
    },
    "broadcast": {
      "type": "boolean"
    },
    "tags": {
      "type": "keyword"
    },
    "user": {
      "type": "integer"
    },
    "feed": {
      "type": "integer"
    },
    "category": {
      "type": "integer"
    }
  }
}'
  1. Create Superuser for Django Application
docker exec -it django_app bash
python manage.py createsuperuser
  1. Add permission to above createdsuperuser
docker exec -it feedhq-feedhq_postgres-1 bash
psql -U feedhq -d feedhq
update auth_user set is_active=true where id=1;
  1. Go to admin page and do some entries in feedhq db.
http://localhost:8001/admin/
# Add entries in all the tables

2. Run FeedHq on YB

  1. Go to yugabyte container and check feedhq database is created or not
docker exec -it feedhq-feedhq_yugabyte-1 bash

ysqlsh --host $(hostname) 
#password= yugabyte
# Change database to feedhq. If it is not created then create db feedhq.
\c feedhq;
  1. Change settings.py database block to point to YB database.
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': "feedhq",
        'USER': "feedhq",
        'PASSWORD': "feedhq",
        'HOST': "feedhq_yugabyte", #yugabyte container host name
        'PORT': "5433",
        'OPTIONS': {
            'options': '-c timezone=UTC',
        },
    }
}
  1. Migrate the data from django app
docker exec -it django_app bash
python -V 
# This should come as python3.6
python manage.py migrate
# now all tables created in yb db also
  1. Follow the same step to add entries now in yb db also.
# Create super user
docker exec -it django_app bash
python manage.py createsuperuser
docker exec -it feedhq-feedhq_yugabyte-1 bash
ysqlsh --host $(hostname) 
update auth_user set is_active=true where id=1;
# Add entries from admin page now- http://localhost:8001/admin/

3. Migrate feedhq app from PG to YB using yb-voyager

  1. install yb-voyager using docker
docker pull yugabytedb/yb-voyager:1.8.2-rc2

wget -O ./yb-voyager https://raw.githubusercontent.com/yugabyte/yb-voyager/main/docker/yb-voyager-docker && chmod +x ./yb-voyager && sudo mv yb-voyager /usr/local/bin/yb-voyager

yb-voyager version
  1. Clean the yb db (if there is any table in feedhq db or data)
cd feedhq-pg-app-century
rm -rf data
mkdir backup-dir
mkdir assess-dir
mkdir -p data/yb_data
# go to yb db service and clear all table and sequence if exists.
# Database feedhq should be clean and does not contain anything
# it should have user as feedhq, password as feedhq
docker exec -it feedhq-feedhq_yugabyte-1 bash
ysqlsh --host $(hostname) -d feedhq -U feedhq
# Enter password: feedhq 
# Check \dt, \d should be "no relations found"

Migration Steps

  1. run assessment
yb-voyager assess-migration --source-db-host localhost --source-db-port 5432 --source-db-type postgresql --source-db-user feedhq --source-db-name feedhq --source-db-password feedhq --source-db-schema "public" --send-diagnostics false --export-dir assess-dir --start-clean=true
  1. export schema
yb-voyager export schema --export-dir assess-dir --source-db-host localhost --source-db-port 5432 --source-db-type postgresql --source-db-user feedhq --source-db-name feedhq --source-db-password feedhq --source-db-schema "public" --send-diagnostics false
  1. analyze-schema
yb-voyager analyze-schema --export-dir assess-dir --send-diagnostics false
  1. export data
yb-voyager export data --export-dir assess-dir --source-db-host localhost --source-db-port 5432 --source-db-type postgresql --source-db-user feedhq --source-db-name feedhq --source-db-password feedhq --source-db-schema "public" --send-diagnostics false
  1. export data status
yb-voyager export data status --export-dir assess-dir
  1. import schema. ( Make sure you dont ahve any relations in feedhq db. Otherwise it will fail)
yb-voyager import schema --export-dir assess-dir --target-db-host localhost --target-db-user feedhq --target-db-password feedhq --target-db-name feedhq
  1. import data
yb-voyager import data --export-dir assess-dir --target-db-host localhost --target-db-user feedhq --target-db-password feedhq --target-db-name feedhq
  1. import data status
yb-voyager import data status --export-dir assess-dir
  1. end migration
yb-voyager end migration --export-dir assess-dir --backup-log-files true --backup-data-files true --backup-schema-files true --save-migration-reports true --backup-dir backup-dir/

Important Info- Appendix

Container name can be change when you deploy the services, so change the name of container according to your container. Log in to container name:

docker exec -it <container_name> bash

Author