Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GitHub Discussions Search Collator #1744

Open
wants to merge 55 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
ef464fb
Create github-discussions workspace
minkimcello Oct 2, 2024
77c6476
Instantiate new backend-module
minkimcello Oct 2, 2024
350ca18
Initialize new frontend plugin
minkimcello Oct 15, 2024
780cc9d
Remove unnecessary files from default template
minkimcello Oct 16, 2024
a684d7c
Create bare collator and yield example documents
minkimcello Oct 16, 2024
4ca7a27
Export collator factory
minkimcello Oct 16, 2024
a7c222d
Upgrade yarn to 4.5.0
minkimcello Oct 17, 2024
77cc3af
Upgrade backstage to 1.32.1
minkimcello Oct 17, 2024
1b82ee2
Install backstage app and backend to workspace
minkimcello Oct 17, 2024
7664d01
Strip away all irrelevant plugins from app/backend
minkimcello Oct 18, 2024
c97aaa1
Install collator and result item to example app
minkimcello Oct 18, 2024
50fbe2a
Add another example document to the collator
minkimcello Oct 18, 2024
284e80a
Display title and text in result item
minkimcello Oct 18, 2024
24ed620
Create a common plugin
minkimcello Oct 18, 2024
ff495b5
Add Github Discussion Fetcher integration
minkimcello Oct 18, 2024
40a4d22
Move document type to types file
minkimcello Oct 18, 2024
d882d28
WIP
minkimcello Oct 18, 2024
860d975
Move document type to common plugin
minkimcello Oct 18, 2024
1455a83
Add category chip to search result item
minkimcello Oct 18, 2024
831f0c1
Integrate github-discussions-fetcher in collator
minkimcello Oct 19, 2024
6e98853
Add clickable github user chip and discussions label chips
minkimcello Oct 19, 2024
1b77ed1
Upgrade fetcher to 0.2.3 and fix integration config
minkimcello Oct 20, 2024
e88ea1e
Provide something that looks like a logger
taras Oct 20, 2024
a866d92
Fix chip label and conditional logic
minkimcello Oct 20, 2024
73cb3ee
Avoid passing unneeded integrations
taras Oct 20, 2024
04cc648
Bump fetcher to 0.3.0
minkimcello Oct 20, 2024
1c1636a
Require users to provide a cache path for discussion queries
minkimcello Oct 21, 2024
91f4cc0
Allow collator schedule to be set in app-config
minkimcello Oct 21, 2024
1afbf9a
Move other collator config next to schedule
minkimcello Oct 21, 2024
4fc73ea
Parse and pass timeout of collator to fetcher
minkimcello Oct 21, 2024
65466a1
Bump fetcher to 0.4.0
minkimcello Oct 21, 2024
cd4ba23
Move config out of collator
minkimcello Oct 21, 2024
46dbce1
Allow cache dir to be optional and add clear boolean config
minkimcello Oct 21, 2024
1a35dc6
Bump fetcher to 0.5.0
minkimcello Oct 21, 2024
f5cc127
Add .cache to .gitignore
minkimcello Oct 22, 2024
396cba7
Make batch sizes configurable via app-config
minkimcello Oct 22, 2024
ed12f33
Cache arg is optional
minkimcello Oct 22, 2024
0898b83
WIP app-config type validation
minkimcello Oct 22, 2024
b71ee85
Bump fetcher to 0.6.0
minkimcello Oct 23, 2024
b0309af
Add READMEs
minkimcello Oct 23, 2024
29ad091
Add comments for config.d.ts
minkimcello Oct 23, 2024
5fdf802
Setup first collator test
minkimcello Oct 30, 2024
51a94d2
Remove @microsoft/api-extractor resolution
minkimcello Oct 30, 2024
67c86c4
Add comments and re-generate report
minkimcello Oct 30, 2024
01f4c07
WIP Write more tests
minkimcello Oct 30, 2024
d765a55
Bump fetcher to 0.7.2
minkimcello Nov 5, 2024
f68913f
Upgrade Effection and refactor collator
minkimcello Nov 5, 2024
2115d41
Bump fetcher to 0.7.3
minkimcello Nov 6, 2024
1dcde2f
Write unit tests with mock data
minkimcello Nov 8, 2024
1127bee
Bump fetcher to 0.7.4
minkimcello Nov 12, 2024
5ecbcd1
Fix prettier issue and dedupe lockfile
minkimcello Nov 12, 2024
083999b
Merge pull request #1 from guidanti/mk/collator-tests
minkimcello Nov 12, 2024
f37ef0f
Run repo fix to fill in missing info
minkimcello Nov 12, 2024
f1adcdb
Pass root config for ScmIntegrations
minkimcello Nov 13, 2024
00ae9ae
Remove .yarn
minkimcello Nov 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions workspaces/github-discussions/.changeset/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Changesets

Hello and welcome! This folder has been automatically generated by `@changesets/cli`, a build tool that works
with multi-package repos, or single-package repos to help you version and publish your code. You can
find the full documentation for it [in our repository](https://github.com/changesets/changesets)

We have a quick list of common questions to get you started engaging with this project in
[our documentation](https://github.com/changesets/changesets/blob/main/docs/common-questions.md)
10 changes: 10 additions & 0 deletions workspaces/github-discussions/.changeset/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"$schema": "https://unpkg.com/@changesets/config@3.0.0/schema.json",
"changelog": "@changesets/cli/changelog",
"commit": false,
"fixed": [],
"linked": [],
"access": "public",
"baseBranch": "main",
"updateInternalDependencies": "patch"
}
8 changes: 8 additions & 0 deletions workspaces/github-discussions/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
.git
.yarn/cache
.yarn/install-state.gz
node_modules
packages/*/src
packages/*/node_modules
plugins
*.local.yaml
1 change: 1 addition & 0 deletions workspaces/github-discussions/.eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
playwright.config.ts
1 change: 1 addition & 0 deletions workspaces/github-discussions/.eslintrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
module.exports = require('../../.eslintrc.cjs');
56 changes: 56 additions & 0 deletions workspaces/github-discussions/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
.cache

# macOS
.DS_Store

# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*

# Coverage directory generated when running tests with coverage
coverage

# Dependencies
node_modules/

# Yarn 3 files
.pnp.*
.yarn/
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions

# Node version directives
.nvmrc

# dotenv environment variables file
.env
.env.test

# Build output
dist
dist-types

# Temporary change files created by Vim
*.swp

# MkDocs build output
site

# Local configuration files
*.local.yaml

# Sensitive credentials
*-credentials.yaml

# vscode database functionality support files
*.session.sql

# E2E test reports
e2e-test-report/
5 changes: 5 additions & 0 deletions workspaces/github-discussions/.prettierignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
dist
dist-types
coverage
.vscode
.eslintrc.js
5 changes: 5 additions & 0 deletions workspaces/github-discussions/.yarnrc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
compressionLevel: mixed
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can skip this file and the .yarn folder


nodeLinker: node-modules

yarnPath: .yarn/releases/yarn-4.5.0.cjs
16 changes: 16 additions & 0 deletions workspaces/github-discussions/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# [Backstage](https://backstage.io)

This is your newly scaffolded Backstage App, Good Luck!

To start the app, run:

```sh
yarn install
yarn dev
```

To generate knip reports for this app, run:

```sh
yarn backstage-repo-tools knip-reports
```
69 changes: 69 additions & 0 deletions workspaces/github-discussions/app-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
app:
title: Scaffolded Backstage App
baseUrl: http://localhost:3000

organization:
name: My Company

search:
collators:
githubDiscussions:
url: https://github.com/guidanti/github-discussions-fetcher
schedule:
initialDelay: { seconds: 10 }
timeout: { minutes: 10 }
frequency: { minutes: 30 }
cacheBase: file:///example/absolute/path
clearCacheOnSuccess: true
discussionsBatchSize: 65
commentsBatchSize: 65
repliesBatchSize: 65

backend:
# Used for enabling authentication, secret is shared by all backend plugins
# See https://backstage.io/docs/auth/service-to-service-auth for
# information on the format
# auth:
# keys:
# - secret: ${BACKEND_SECRET}
baseUrl: http://localhost:7007
listen:
port: 7007
# Uncomment the following host directive to bind to specific interfaces
# host: 127.0.0.1
csp:
connect-src: ["'self'", 'http:', 'https:']
# Content-Security-Policy directives follow the Helmet format: https://helmetjs.github.io/#reference
# Default Helmet Content-Security-Policy values can be removed by setting the key to false
cors:
origin: http://localhost:3000
methods: [GET, HEAD, PATCH, POST, PUT, DELETE]
credentials: true
# This is for local development only, it is not recommended to use this in production
# The production database configuration is stored in app-config.production.yaml
database:
client: better-sqlite3
connection: ':memory:'
# workingDirectory: /tmp # Use this to configure a working directory for the scaffolder, defaults to the OS temp-dir

integrations:
github:
- host: github.com
# This is a Personal Access Token or PAT from GitHub. You can find out how to generate this token, and more information
# about setting up the GitHub integration here: https://backstage.io/docs/integrations/github/locations#configuration
token: ${GITHUB_TOKEN}
### Example for how to add your GitHub Enterprise instance using the API:
# - host: ghe.example.net
# apiBaseUrl: https://ghe.example.net/api/v3
# token: ${GHE_TOKEN}

auth:
# see https://backstage.io/docs/auth/ to learn about auth providers
providers:
# See https://backstage.io/docs/auth/guest/provider
guest: {}

# see https://backstage.io/docs/permissions/getting-started for more on the permission framework
permission:
# setting this to `false` will disable permissions
enabled: true
3 changes: 3 additions & 0 deletions workspaces/github-discussions/backstage.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"version": "1.32.1"
}
13 changes: 13 additions & 0 deletions workspaces/github-discussions/catalog-info.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: github-discussions
description: An example of a Backstage application.
# Example for optional annotations
# annotations:
# github.com/project-slug: backstage/backstage
# backstage.io/techdocs-ref: dir:.
spec:
type: website
owner: john@example.com
lifecycle: experimental
66 changes: 66 additions & 0 deletions workspaces/github-discussions/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
{
"name": "@internal/github-discussions",
"version": "1.0.0",
"private": true,
"engines": {
"node": "18 || 20"
},
"scripts": {
"dev": "yarn workspaces foreach -A --include backend --include app --parallel -v -i run start",
"start": "yarn workspace app start",
"start-backend": "yarn workspace backend start",
"tsc": "tsc",
"tsc:full": "tsc --skipLibCheck false --incremental false",
"build:all": "backstage-cli repo build --all",
"build:api-reports": "yarn build:api-reports:only --tsc",
"build:api-reports:only": "backstage-repo-tools api-reports -o --allow-all-warnings ae-wrong-input-file-type --validate-release-tags",
"clean": "backstage-cli repo clean",
"test": "backstage-cli repo test",
"test:all": "backstage-cli repo test --coverage",
"fix": "backstage-cli repo fix",
"lint": "backstage-cli repo lint --since origin/main",
"lint:all": "backstage-cli repo lint",
"prettier:check": "prettier --check .",
"new": "backstage-cli new --scope @backstage-community",
"postinstall": "cd ../../ && yarn install"
},
"workspaces": {
"packages": [
"packages/*",
"plugins/*"
]
},
"repository": {
"type": "git",
"url": "https://github.com/backstage/community-plugins",
"directory": "workspaces/github-discussions"
},
"devDependencies": {
"@backstage/cli": "^0.28.0",
"@backstage/e2e-test-utils": "^0.1.1",
"@backstage/repo-tools": "^0.10.0",
"@changesets/cli": "^2.27.1",
"@spotify/prettier-config": "^12.0.0",
"node-gyp": "^9.0.0",
"prettier": "^2.3.2",
"typescript": "~5.3.0"
},
"dependencies": {
"@backstage/plugin-search-react": "^1.8.1",
"knip": "^5.27.4"
},
"resolutions": {
"@types/react": "^18",
"@types/react-dom": "^18"
},
"prettier": "@spotify/prettier-config",
"lint-staged": {
"*.{js,jsx,ts,tsx,mjs,cjs}": [
"eslint --fix",
"prettier --write"
],
"*.{json,md}": [
"prettier --write"
]
}
}
9 changes: 9 additions & 0 deletions workspaces/github-discussions/packages/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# The Packages Folder

This is where your own applications and centrally managed libraries live, each
in a separate folder of its own.

From the start there's an `app` folder (for the frontend) and a `backend` folder
(for the Node backend), but you can also add more modules in here that house
your core additions and adaptations, such as themes, common React component
libraries, utilities, and similar.
1 change: 1 addition & 0 deletions workspaces/github-discussions/packages/app/.eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
public
1 change: 1 addition & 0 deletions workspaces/github-discussions/packages/app/.eslintrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
module.exports = require('@backstage/cli/config/eslint-factory')(__dirname);
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
/*
* Copyright 2020 The Backstage Authors
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import { test, expect } from '@playwright/test';

test('App should render the welcome page', async ({ page }) => {
await page.goto('/');

const enterButton = page.getByRole('button', { name: 'Enter' });
await expect(enterButton).toBeVisible();
await enterButton.click();

await expect(page.getByText('My Company Catalog')).toBeVisible();
});
64 changes: 64 additions & 0 deletions workspaces/github-discussions/packages/app/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
{
"name": "app",
"version": "0.0.0",
"private": true,
"bundled": true,
"repository": {
"type": "git",
"url": "https://github.com/backstage/community-plugins",
"directory": "workspaces/github-discussions/packages/app"
},
"backstage": {
"role": "frontend"
},
"scripts": {
"start": "backstage-cli package start",
"build": "backstage-cli package build",
"clean": "backstage-cli package clean",
"test": "backstage-cli package test",
"lint": "backstage-cli package lint"
},
"dependencies": {
"@backstage-community/plugin-github-discussions": "workspace:^",
"@backstage/app-defaults": "^1.5.12",
"@backstage/cli": "^0.28.0",
"@backstage/core-app-api": "^1.15.1",
"@backstage/core-components": "^0.15.1",
"@backstage/core-plugin-api": "^1.10.0",
"@backstage/integration-react": "^1.2.0",
"@backstage/plugin-search": "^1.4.18",
"@backstage/plugin-search-react": "^1.8.1",
"@backstage/theme": "^0.6.0",
"@material-ui/core": "^4.12.2",
"@material-ui/icons": "^4.9.1",
"react": "^18.0.2",
"react-dom": "^18.0.2",
"react-router": "^6.3.0",
"react-router-dom": "^6.3.0"
},
"devDependencies": {
"@backstage/test-utils": "^1.7.0",
"@playwright/test": "^1.32.3",
"@testing-library/dom": "^9.0.0",
"@testing-library/jest-dom": "^6.0.0",
"@testing-library/react": "^14.0.0",
"@testing-library/user-event": "^14.0.0",
"@types/react-dom": "*",
"cross-env": "^7.0.0"
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
},
"files": [
"dist"
]
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Loading
Loading