Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor/db #749

Merged
merged 73 commits into from
Jan 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
c42bd25
add migration file for new db tables (only for text-line and number c…
Dec 4, 2023
308538e
small adjustments
Dec 4, 2023
cf70afb
new models and mappers
Dec 4, 2023
f86cb79
basic integration of the new models (only for text-line and number co…
Dec 4, 2023
b217de2
WIP psalm and super classes
Dec 5, 2023
52337ee
WIP try to adjust types to pass the psalm tests
Dec 8, 2023
1a45682
WIP try to adjust types to pass the psalm tests - part 2
Dec 8, 2023
cf9fd45
fix: add the id to the new rows
Dec 15, 2023
abf262c
code cleanup
Dec 15, 2023
420da4c
remove data from db if a column gets deleted
Dec 15, 2023
3346054
fix: even get results if the rows don't have data
Dec 15, 2023
f58f352
adjust and fix types to make psalm happy
Dec 18, 2023
a61e2f1
code clean up
Dec 19, 2023
8be2541
update rowService to use Row2 everywhere
Dec 19, 2023
c8b1402
add a helper class to define all the existing column-types
Dec 20, 2023
07a5235
add command to makefile for psalm check errors only
Dec 20, 2023
bc662cd
add all missing column types
Dec 20, 2023
98a4449
fix: formatting and types
Dec 20, 2023
7b7818d
fix(sqlite): remove unnecessary brackets
Dec 20, 2023
42eb83b
fix(psalm): add template für dynamic types
Dec 21, 2023
1ec223c
Update lib/Helper/ColumnsHelper.php
Jan 2, 2024
d876459
code cleanup
Jan 2, 2024
9799b0c
db migration - add combined index
Jan 2, 2024
12cc779
Update lib/Db/Row2Mapper.php
Jan 3, 2024
b01cbaf
Update lib/Db/Row2Mapper.php
Jan 3, 2024
526d944
remove unnecessary interface
Jan 3, 2024
31b62f4
add index for new db table
Jan 3, 2024
0f3aee2
Renaming rows
Jan 3, 2024
99548ef
Code cleanup
Jan 3, 2024
ba7a9db
Repair step to migrate rows data into the new db structure
Jan 4, 2024
3fb06a6
fix info output
Jan 4, 2024
dbd4e53
add command to migrate data to new db structure
Jan 5, 2024
5a20018
make the column definition not dynamic to avoid runtime problems duri…
Jan 5, 2024
05b1a6a
Set legacy transfer to app config to run it only once
Jan 5, 2024
3ef403c
split columns mapper find method into find() and findAll()
Jan 11, 2024
02d0b7a
Update lib/Db/RowCellNumberMapper.php
Jan 11, 2024
c8862c7
wrap row deletion in a transaction
Jan 11, 2024
98f013a
code cleanup
Jan 11, 2024
14da9ff
update names to avoid misunderstandings with the word "magic"
Jan 11, 2024
9806641
Update lib/Db/RowMapper.php
Jan 11, 2024
8cb2b38
Revert "Update lib/Db/RowMapper.php"
Jan 11, 2024
d41e717
rename row to row2 to avoid overlapping namespaces specially within t…
Jan 11, 2024
4f75694
code cleanup
Jan 11, 2024
2455d51
Update lib/Command/TransferLegacyRows.php
Jan 11, 2024
35e8ad5
Update lib/Command/TransferLegacyRows.php
Jan 11, 2024
8b6699e
Update lib/Command/TransferLegacyRows.php
Jan 11, 2024
46a045a
Update lib/Command/TransferLegacyRows.php
Jan 11, 2024
4e3e848
use new row mapper in clean command
Jan 11, 2024
9c4a182
adjust the clean command to use the new db structure and code base
Jan 12, 2024
2951d07
Update lib/Service/RowService.php
Jan 12, 2024
e785a9a
only add the where clause if needed
Jan 12, 2024
46be5b4
code cleanup
Jan 12, 2024
2b6945e
added some error details
Jan 12, 2024
32fb5a2
Revert "added some error details"
Jan 12, 2024
b08ed0b
adjust psalm return types
Jan 12, 2024
6dd64f6
add CI upgrade test
Jan 12, 2024
baf54ca
add unit test for data migration to new db modal
Jan 12, 2024
e84c102
simplify the columns definition helper
Jan 16, 2024
152ed10
only trigger CI upgrade test on PR with destination branch main
Jan 16, 2024
1bdc6da
remove unnecessary method
Jan 16, 2024
bb74f19
fix permission check
Jan 16, 2024
1cdcbac
Add command for clean up legacy rows
Jan 18, 2024
aeae31a
cleanup
Jan 18, 2024
85f2e5b
add handling for view meta-column filtering
Jan 18, 2024
bead04e
Update lib/Migration/NewDbStructureRepairStep.php
Jan 22, 2024
41f2174
Check for column existence
Jan 22, 2024
ce72ade
Adjust the unit tests
Jan 22, 2024
3c9d453
fix: replace placeholder for selection and check columns
Jan 23, 2024
89ca33d
Adjust no-delete option to delete option to avoid data lost
Jan 23, 2024
57e42b7
fix: Print error during commands and catch more general errors during…
juliushaertl Jan 24, 2024
72b9ef3
fix: use all rows for querying row candidates if a filter on a not-ac…
juliushaertl Jan 24, 2024
02a58d0
enh: add filtering in views for multi selection
Jan 24, 2024
55e9c29
test: add end2end tests for views filtering for selection columns
Jan 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 150 additions & 0 deletions .github/workflows/app-upgrade-mysql.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
name: app upgrade mysql

on:
pull_request:
branches: main

permissions:
contents: read

concurrency:
group: app-upgrade-mysql-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

jobs:
changes:
runs-on: ubuntu-latest

outputs:
src: ${{ steps.changes.outputs.src}}

steps:
- uses: dorny/paths-filter@4512585405083f25c027a35db413c2b3b9006d50 # v2.11.1
id: changes
continue-on-error: true
with:
filters: |
src:
- '.github/workflows/**'
- 'appinfo/**'
- 'lib/**'
- 'templates/**'
- 'tests/**'
- 'vendor/**'
- 'vendor-bin/**'
- '.php-cs-fixer.dist.php'
- 'composer.json'
- 'composer.lock'

app-upgrade-mysql:
runs-on: ubuntu-latest

needs: changes
if: needs.changes.outputs.src != 'false'

strategy:
matrix:
php-versions: ['8.2']
server-versions: ['master']

services:
mysql:
image: ghcr.io/nextcloud/continuous-integration-mysql-8.0:latest
ports:
- 4444:3306/tcp
env:
MYSQL_ROOT_PASSWORD: rootpassword
options: --health-cmd="mysqladmin ping" --health-interval 5s --health-timeout 2s --health-retries 5

steps:
- name: Set app env
run: |
# Split and keep last
echo "APP_NAME=${GITHUB_REPOSITORY##*/}" >> $GITHUB_ENV

- name: Checkout server
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
submodules: true
repository: nextcloud/server
ref: ${{ matrix.server-versions }}

- name: Register text Git reference
run: |
text_app_ref="$(if [ "${{ matrix.server-versions }}" = "master" ]; then echo -n "main"; else echo -n "${{ matrix.server-versions }}"; fi)"
echo "text_app_ref=$text_app_ref" >> $GITHUB_ENV

- name: Checkout text app
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
repository: nextcloud/text
path: apps/text
ref: ${{ env.text_app_ref }}

- name: Checkout viewer app
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
repository: nextcloud/viewer
path: apps/viewer
ref: ${{ matrix.server-versions }}

- name: Set up php ${{ matrix.php-versions }}
uses: shivammathur/setup-php@e6f75134d35752277f093989e72e140eaa222f35 # v2
with:
php-version: ${{ matrix.php-versions }}
# https://docs.nextcloud.com/server/stable/admin_manual/installation/source_installation.html#prerequisites-for-manual-installation
extensions: bz2, ctype, curl, dom, fileinfo, gd, iconv, intl, json, libxml, mbstring, openssl, pcntl, posix, session, simplexml, xmlreader, xmlwriter, zip, zlib, mysql, pdo_mysql, sqlite, pdo_sqlite
coverage: none
ini-file: development
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Enable ONLY_FULL_GROUP_BY MySQL option
run: |
echo "SET GLOBAL sql_mode=(SELECT CONCAT(@@sql_mode,',ONLY_FULL_GROUP_BY'));" | mysql -h 127.0.0.1 -P 4444 -u root -prootpassword
echo "SELECT @@sql_mode;" | mysql -h 127.0.0.1 -P 4444 -u root -prootpassword

- name: Set up Nextcloud, enable tables from app store
env:
DB_PORT: 4444
run: |
mkdir data
./occ maintenance:install --verbose --database=mysql --database-name=nextcloud --database-host=127.0.0.1 --database-port=$DB_PORT --database-user=root --database-pass=rootpassword --admin-user admin --admin-pass admin
./occ app:enable --force ${{ env.APP_NAME }}
datenangebot marked this conversation as resolved.
Show resolved Hide resolved

- name: Checkout app
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
path: apps/${{ env.APP_NAME }}

- name: Set up dependencies
working-directory: apps/${{ env.APP_NAME }}
run: composer i --no-dev

- name: Upgrade Nextcloud and see whether the app still works
run: |
./occ upgrade
./occ app:list

- name: Upload nextcloud logs
uses: actions/upload-artifact@v4
if: always()
with:
name: nextcloud.log
path: data/nextcloud.log
retention-days: 5


summary:
permissions:
contents: none
runs-on: ubuntu-latest
needs: [changes, app-upgrade-mysql]

if: always()

name: app-upgrade-mysql-summary

steps:
- name: Summary status
run: if ${{ needs.changes.outputs.src != 'false' && needs.app-upgrade-mysql.result != 'success' }}; then exit 1; fi
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,9 @@ lint: lint-php lint-js lint-css lint-xml

lint-php: lint-php-lint lint-php-cs-fixer lint-php-psalm

lint-fast:
composer run psalm -- --show-info=false

lint-php-lint:
# Check PHP syntax errors
@! find $(php_dirs) -name "*.php" | xargs -I{} php -l '{}' | grep -v "No syntax errors detected"
Expand Down
9 changes: 8 additions & 1 deletion appinfo/info.xml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Share your tables and views with users and groups within your cloud.
Have a good time and manage whatever you want.

]]></description>
<version>0.6.5</version>
<version>0.7.0-dev.1</version>
<licence>agpl</licence>
<author mail="florian.steffens@nextcloud.com" >Florian Steffens</author>
<namespace>Tables</namespace>
Expand All @@ -46,13 +46,20 @@ Have a good time and manage whatever you want.
<database>sqlite</database>
<nextcloud min-version="25" max-version="29"/>
</dependencies>
<repair-steps>
<post-migration>
<step>OCA\Tables\Migration\NewDbStructureRepairStep</step>
</post-migration>
</repair-steps>
<commands>
<command>OCA\Tables\Command\ListTables</command>
<command>OCA\Tables\Command\AddTable</command>
<command>OCA\Tables\Command\RemoveTable</command>
<command>OCA\Tables\Command\RenameTable</command>
<command>OCA\Tables\Command\ChangeOwnershipTable</command>
<command>OCA\Tables\Command\Clean</command>
<command>OCA\Tables\Command\CleanLegacy</command>
<command>OCA\Tables\Command\TransferLegacyRows</command>
</commands>
<navigations>
<navigation>
Expand Down
Loading
Loading