Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix installation instructions #495

Closed
pchtsp opened this issue Jan 3, 2024 · 2 comments · Fixed by #506
Closed

Fix installation instructions #495

pchtsp opened this issue Jan 3, 2024 · 2 comments · Fixed by #506
Assignees
Labels
documentation Improvements or additions to documentation
Milestone

Comments

@pchtsp
Copy link
Collaborator

pchtsp commented Jan 3, 2024

I used the installation documentation on the following places:

Following these instructions I failed to install any of the parts (cornflow or ariflow) correctly.

I had to go look for the unit tests of cornflow client to understand the current steps and adapt them to a normal (i.e., outside of github) environment.

here are a few of the issues I found in the two guides above:

  • apache airflow version is wrong (2.7.1 vs 2.1)
  • there's no instruction to do cd cornflow-server before installing virtual environments and deploying.
  • the flask db upgrade instruction lacks an argument.
  • cornflow-dags dags and requirements need to be copied to the airflow_config directory
  • Unconsistent references to afvenv and cfvenv (venv) in all documentation/ guied.
  • A command to install apache airflow providers google was missing and lead to an error. Solved with pip install apache-airflow-providers-google
  • References to python instead of python3 are not portable to Ubuntu (replacing with python3 worked)
  • A reference to a cornflow port on launch (5050 vs 5000) was inconsistent.

I'm sure I'm still lacking a few things. I leave below the installation instructions (commands) that worked for me, merging from memory, unit tests and current docs.


Cornflow server installation and running

cd cornflow-server
python -m venv cfvenv
source cfvenv/bin/activate
python3 -m pip install -U -r requirements-dev.txt
python3 -m pip install -U -e ../libs/client

export FLASK_APP=cornflow.app
export FLASK_ENV=development
export DATABASE_URL=sqlite:///cornflow.db
export SECRET_KEY=THISNEEDSTOBECHANGED
export AIRFLOW_URL=http://localhost:8080
export AIRFLOW_USER=admin
export AIRFLOW_PWD=admin

flask db upgrade -d cornflow/migrations/
flask access_init
flask register_deployed_dags -r http://127.0.0.1:8080 -u admin -p admin
flask create_admin_user -u admin -e admin@cornflow.org -p Adminpassword1!
flask create_service_user -u airflow -e airflow@cornflow.org -p Airflow_test_password1
flask create_base_user -u user -e user@cornflow.org -p UserPassword1!
flask register_dag_permissions -o 1
flask run -p 5050 &


Airflow server installation

cp -r cornflow-dags/DAG/* cornflow-server/airflow_config/dags/
cp cornflow-dags/requirements.txt cornflow-server/airflow_config/
cd cornflow-server
python3 -m venv afvenv
source afvenv/bin/activate
AIRFLOW_VERSION=2.7.1
PYTHON_VERSION="$(python3 --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
CLIENT_BRANCH="${{ github.head_ref || github.ref_name }}"
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
python3 -m pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
python3 -m pip install -U orloge pulp
python3 -m pip install -U -e ../libs/client
python3 -m pip install -U -r airflow_config/requirements.txt
pip install apache-airflow-providers-google
airflow db init
airflow users create -u admin -f admin -l admin -r Admin -p admin -e admin@example.org

Airflow server running

export AIRFLOW_HOME="$PWD/airflow_config"
export AIRFLOW__CORE__LOAD_EXAMPLES=0
export AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=0
export AIRFLOW__API__AUTH_BACKEND=airflow.api.auth.backend.basic_auth
export AIRFLOW__WEBSERVER__SECRET_KEY=e9adafa751fd35adfc1fdd3285019be15eea0758f76e38e1e37a1154fb36
export AIRFLOW_CONN_CF_URI=http://airflow:Airflow_test_password1@localhost:5050
airflow webserver -p 8080 &
airflow scheduler &

@ggsdc ggsdc self-assigned this Jan 3, 2024
@ggsdc ggsdc added the documentation Improvements or additions to documentation label Jan 3, 2024
@ggsdc
Copy link
Member

ggsdc commented Jan 3, 2024

Thank you @pchtsp we will work to fix the documentation issues asap.

@ggsdc ggsdc added this to the v1.1.0 milestone Jan 11, 2024
@ggsdc ggsdc mentioned this issue Jan 11, 2024
@ggsdc ggsdc linked a pull request Jan 11, 2024 that will close this issue
@ggsdc
Copy link
Member

ggsdc commented May 15, 2024

Fixed by #506

@ggsdc ggsdc closed this as completed May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants