diff --git a/README.md b/README.md index 0705a2f6..faf350aa 100644 --- a/README.md +++ b/README.md @@ -7,11 +7,11 @@ This repository is host to: * *`docs-builder`* command line tool to generate single doc-sets (13mb native code, no dependencies) * *`docs-assembler`* command line tool to assemble all the doc sets. (IN PROGRESS) * `elastic/docs-builder@main` Github Action to build and validate a repositories documentation - +* *`docs-generator`* command line tool to deterministically generate a docset and incremental updates to generated content ## Command line interface -``` +```bash $ docs-builder --help Usage: [command] [options...] [-h|--help] [--version] @@ -26,7 +26,7 @@ Options: Commands: generate Converts a source markdown folder or file to an output folder serve Continuously serve a documentation folder at http://localhost:5000. - File systems changes will be reflected without having to restart the server. + File systems changes will be reflected without having to restart the server. ``` In the near term native code will be published by CI for the following platforms @@ -132,22 +132,19 @@ https://github.com/elastic/{your-repository}/settings/pages ## Run without docker You can use the .NET CLI to publish a self-contained `docs-builder` native code -binary. (On my M2 Pro mac the binary is currently 13mb) +binary. (On my M2 Pro mac the binary is currently 16mb) Install [.NET 9.0](https://dotnet.microsoft.com/en-us/download/dotnet/9.0), then run: ```bash -dotnet publish "src/docs-builder/docs-builder.csproj" -c Release -o .artifacts/publish \ - --self-contained true /p:PublishTrimmed=true /p:PublishSingleFile=false /p:PublishAot=true -a arm64 +dotnet publish "src/docs-builder/docs-builder.csproj" ``` -**Note**: `-a` should be the machine's CPU architecture - -The resulting binary `./.artifacts/publish/docs-builder` will run on machines without .NET installed +The resulting binary `./.artifacts/publish/docs-builder/release/docs-builder` will run on machines without .NET installed. # Performance To test performance it's best to build the binary and run outside of docker: -For refence here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to +For reference here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to existing surveyed tools diff --git a/docs/source/configure/content-set/attributes.md b/docs/source/configure/content-set/attributes.md new file mode 100644 index 00000000..f348d59e --- /dev/null +++ b/docs/source/configure/content-set/attributes.md @@ -0,0 +1,5 @@ +--- +title: Attributes +--- + +tbd \ No newline at end of file diff --git a/docs/source/configure/content-set/file-structure.md b/docs/source/configure/content-set/file-structure.md new file mode 100644 index 00000000..a04f108b --- /dev/null +++ b/docs/source/configure/content-set/file-structure.md @@ -0,0 +1,5 @@ +--- +title: File structure +--- + +tbd \ No newline at end of file diff --git a/docs/source/configure/content-set/index.md b/docs/source/configure/content-set/index.md new file mode 100644 index 00000000..a76e717d --- /dev/null +++ b/docs/source/configure/content-set/index.md @@ -0,0 +1,11 @@ +--- +title: Content set configuration +navigation_title: Content set +--- + +Now we'll zoom into the AsciiDoc book level, and explore the V3 equivalent: content sets. At the book level, the system consists of: + +| System property | Asciidoc | V3 | +| -------------------- | -------------------- | -------------------- | +| **Content source files** --> A whole bunch of markup files as well as any other assets used in the docs (for example, images, videos, and diagrams). | **Markup**: AsciiDoc files **Assets**: Images, videos, and diagrams | **Markup**: MD files **Assets**: Images, videos, and diagrams | +| **Information architecture** --> A way to specify the order in which these text-based files should appear in the information architecture of the book. | `index.asciidoc` file (this can be spread across several AsciiDoc files, but generally starts with the index file specified in the `conf.yaml` file)) | TBD | diff --git a/docs/source/configure/content-set/navigation.md b/docs/source/configure/content-set/navigation.md new file mode 100644 index 00000000..9253fa13 --- /dev/null +++ b/docs/source/configure/content-set/navigation.md @@ -0,0 +1,5 @@ +--- +title: Navigation +--- + +tbd \ No newline at end of file diff --git a/docs/source/configure/index.md b/docs/source/configure/index.md new file mode 100644 index 00000000..f8993b0f --- /dev/null +++ b/docs/source/configure/index.md @@ -0,0 +1,4 @@ +--- +title: Configure Elastic Docs +navigation_title: Configuration reference +--- \ No newline at end of file diff --git a/docs/source/configure/page.md b/docs/source/configure/page.md new file mode 100644 index 00000000..21a35ea9 --- /dev/null +++ b/docs/source/configure/page.md @@ -0,0 +1,10 @@ +--- +title: Page configuration +navigation_title: Page +--- + +tbd + +go into frontmatter here? + +for now, see [frontmatter](../syntax/frontmatter.md). \ No newline at end of file diff --git a/docs/source/configure/site/content.md b/docs/source/configure/site/content.md new file mode 100644 index 00000000..0b96f049 --- /dev/null +++ b/docs/source/configure/site/content.md @@ -0,0 +1,43 @@ +--- +title: Content config +--- + +In both the AsciiDoctor- and V3-based system, there is site-wide configuration where you list all content sources, where to find those sources, and in what order they should be added to the site. + +In the AsciiDoctor system, this all happens in one YAML file in the /docs repo. In the V3 system, this ... + +## AsciiDoctor conf.yml + +In the AsciiDoctor-powered site, content configuration at the site level is done in the [`conf.yaml`](https://github.com/elastic/docs/blob/master/conf.yaml) file in the elastic/docs repo. In the `conf.yml` file, the configuration information for all books are listed in this one file. Here's the example of what it looks like to configure a single book: + +```yaml +- title: Machine Learning + prefix: en/machine-learning + current: *stackcurrent + index: docs/en/stack/ml/index.asciidoc + branches: [ {main: master}, 8.9, 8.8, 8.7, 8.6, 8.5, 8.4, 8.3, 8.2, 8.1, 8.0, 7.17, 7.16, 7.15, 7.14, 7.13, 7.12, 7.11, 7.10, 7.9, 7.8, 7.7, 7.6, 7.5, 7.4, 7.3, 7.2, 7.1, 7.0, 6.8, 6.7, 6.6, 6.5, 6.4, 6.3 ] + live: *stacklive + chunk: 1 + tags: Elastic Stack/Machine Learning + subject: Machine Learning + sources: + - + repo: stack-docs + path: docs/en/stack + - + repo: elasticsearch + path: docs + - + repo: docs + path: shared/versions/stack/{version}.asciidoc + - + repo: docs + path: shared/attributes.asciidoc + - + repo: docs + path: shared/settings.asciidoc +``` + +## V3 configuration + +TO DO \ No newline at end of file diff --git a/docs/source/configure/site/img/site-level-asciidoctor.png b/docs/source/configure/site/img/site-level-asciidoctor.png new file mode 100644 index 00000000..9bf17267 Binary files /dev/null and b/docs/source/configure/site/img/site-level-asciidoctor.png differ diff --git a/docs/source/configure/site/index.md b/docs/source/configure/site/index.md new file mode 100644 index 00000000..ecdf1260 --- /dev/null +++ b/docs/source/configure/site/index.md @@ -0,0 +1,24 @@ +--- +title: Site configuration +navigation_title: Site +--- + +Start by understanding how the new V3 system works at the site level compared to how our custom AsciiDoctor system works. The system consists of: + + +| System property | Asciidoc | V3 | +| -------------------- | -------------------- | -------------------- | +| **Content sources** --> Collections of markup files containing doc content. These are split up across many docs repos. | _Books_ | _Content sets_ | +| **Content configuration** --> A way to specify where to find those content sources, and in what order they should be added to the site. | Configuration file ([`conf.yml`](https://github.com/elastic/docs/blob/master/conf.yaml) in elastic/docs) | Config file location TBD | +| **Cross-site values** --> Key-value pairs that should be substituted across all sources as web pages are built. | Shared attributes ([`shared/`](https://github.com/elastic/docs/tree/master/shared) in elastic/docs) | Shared attrs file TBD | +| **Docs build tool** --> An engine used to take the markup in the content sources and transform it into web pages. | Customized version of AsciiDoctor (lives in [**elastic/docs**](https://github.com/elastic/docs)) | Customized doc builder using open source tools (lives in [**elastic/docs-builder**](https://github.com/elastic/docs-builder)) | + +Where these pieces live and what format they are in varies between the two systems, but they generally achieve the same goal. + +## Asciidoc + +![site-level config in the asciidoc system](./img/site-level-asciidoctor.png) + +## V3 + +DIAGRAM NEEDED \ No newline at end of file diff --git a/docs/source/configure/site/landing-page.md b/docs/source/configure/site/landing-page.md new file mode 100644 index 00000000..cff57fde --- /dev/null +++ b/docs/source/configure/site/landing-page.md @@ -0,0 +1,5 @@ +--- +title: Landing page +--- + +tbd \ No newline at end of file diff --git a/docs/source/configure/site/redirects.md b/docs/source/configure/site/redirects.md new file mode 100644 index 00000000..eca65cf5 --- /dev/null +++ b/docs/source/configure/site/redirects.md @@ -0,0 +1,26 @@ +--- +title: Redirects +--- + +## From an elastic.co/guide page + +The web team owns redirects for `elastic.co`. Writers do not have direct access to the system where www.elastic.co redirects are managed. + +### Permanent redirect + +Currently the only way to implement true redirects from a page on `elastic.co/guide` is to open an issue with the web team, add your request to a Google Sheet, and wait for the web team to address your request. + +### Stopgap + +Writers have developed a stopgap to communicate to users that content has been removed or moved to another location. We create a page at the existing URL that links to the new location of the information. This is better than removing a page without redirecting at all, but is not the best user experience and doesn't align with SEO best practices. + +## From an elastic.co/docs page + +STRATEGY TBD + +## Migration redirects + +Docs inventory script: https://github.com/elastic/docs-helpers/tree/main/docs-inventory +Docs inventory sheet: https://docs.google.com/spreadsheets/d/1LfPI3TZqdpONGxOmL8B8V-Feo1flLwObz9_ibCEMkIQ/edit?gid=605983744#gid=605983744 + +This sheet will be used to request redirects after migration from 8.current to 9.current. \ No newline at end of file diff --git a/docs/source/configure/site/shared-attributes.md b/docs/source/configure/site/shared-attributes.md new file mode 100644 index 00000000..f76ba872 --- /dev/null +++ b/docs/source/configure/site/shared-attributes.md @@ -0,0 +1,29 @@ +--- +title: Shared attributes +--- + +To promote consistency across documentation, AsciiDoc uses shared attributes for common terms, URLs, and versions. + +In the AsciiDoctor-based system, shared attributes live in the [`shared/` directory](https://github.com/elastic/docs/blob/master/shared) in the elastic/docs repo. The most used files in this directory are: + +The [`attributes.asciidoc` file](https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc), which contains URLs, common words and phrases, and more. +The files in the [`versions/stack` directory](https://github.com/elastic/docs/tree/master/shared/versions/stack), which contain the latest versions for various products for a given Stack version. + +## Versions + +### Asciidoc + +File: [`shared/versions/stack/8.8.asciidoc`](https://github.com/elastic/docs/blob/master/shared/versions/stack/8.8.asciidoc) in the elastic/docs repo + +### V3 + +See [Product availability](../../syntax/applies.md). + + +## URLS + +File: [`shared/attributes.asciidoc`](https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc) in the elastic/docs repo + +## Common terms + +File: [`shared/attributes.asciidoc`](https://github.com/elastic/docs/blob/master/shared/attributes.asciidoc) in the elastic/docs repo \ No newline at end of file diff --git a/docs/source/contribute/change-browser.md b/docs/source/contribute/change-browser.md new file mode 100644 index 00000000..106b0cb3 --- /dev/null +++ b/docs/source/contribute/change-browser.md @@ -0,0 +1,11 @@ +--- +title: Update the docs locally +--- + +1. Install dependencies +2. Clone repositories +3. Make changes +4. Open a Pull Request +5. Work with CI +6. Get approvals and merge +7. View your changes live on elastic.co \ No newline at end of file diff --git a/docs/source/contribute/change-local.md b/docs/source/contribute/change-local.md new file mode 100644 index 00000000..d1ab75c1 --- /dev/null +++ b/docs/source/contribute/change-local.md @@ -0,0 +1,3 @@ +--- +title: Update the docs in your web browser +--- diff --git a/docs/source/contribute/index.md b/docs/source/contribute/index.md new file mode 100644 index 00000000..345042a7 --- /dev/null +++ b/docs/source/contribute/index.md @@ -0,0 +1,10 @@ +--- +title: Elastic Docs contribution guide +navigation_title: Contribution guide +--- + +Want to contribute to the Elastic documentation? You're in the right spot! Select an option below to get started: + +* I just want to suggest a change --> [Open a docs issue](https://github.com/elastic/docs-content/issues/new?template=internal-request.yaml) +* I want to make a small change to a single page --> [Update the docs in your _web browser_](change-browser.md). +* I want to make a larger change or a change to multiple pages --> [Update the docs in your _code editor_](change-local.md). diff --git a/docs/source/docset.yml b/docs/source/docset.yml index 040842bf..2ca53699 100644 --- a/docs/source/docset.yml +++ b/docs/source/docset.yml @@ -19,6 +19,32 @@ exclude: - '_*.md' toc: - file: index.md + - folder: migration + children: + - file: index.md + - file: file-structure.md + - folder: contribute + children: + - file: index.md + - file: change-browser.md + - file: change-local.md + - folder: configure + children: + - file: index.md + - folder: site + children: + - file: index.md + - file: content.md + - file: shared-attributes.md + - file: landing-page.md + - file: redirects.md + - folder: content-set + children: + - file: index.md + - file: file-structure.md + - file: attributes.md + - file: navigation.md + - file: page.md - folder: syntax children: - file: index.md diff --git a/docs/source/index.md b/docs/source/index.md index bc23f968..39256164 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -28,7 +28,7 @@ Options: Commands: generate Converts a source markdown folder or file to an output folder serve Continuously serve a documentation folder at http://localhost:5000. - File systems changes will be reflected without having to restart the server. + File systems changes will be reflected without having to restart the server. ``` In the near term native code will be published by CI for the following platforms @@ -44,7 +44,8 @@ And we'll invest time in making sure these are easily obtainable (`brew`, `winge For now you can run the tool locally through `docker run` ```bash -docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" ghcr.io/elastic/docs-builder:edge +docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \ + ghcr.io/elastic/docs-builder:edge ``` This ensures `.git`/`docs` and `.artifacts` (the default output directory) are mounted. @@ -54,7 +55,8 @@ Only the changed files on subsequent runs will be compiled unless you pass `--fo to force a new compilation. ```bash -docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" ghcr.io/elastic/docs-builder:edge --force +docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \ + ghcr.io/elastic/docs-builder:edge --force ``` #### Live mode @@ -62,7 +64,8 @@ docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.ar Through the `serve` command you can continuously and partially compile your documentation. ```bash -docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" --expose 8080 ghcr.io/elastic/docs-builder:edge serve +docker run -v "./.git:/app/.git" -v "./docs:/app/docs" -v "./.artifacts:/app/.artifacts" \ + -p 8080:8080 ghcr.io/elastic/docs-builder:edge serve ``` Each page is compiled on demand as you browse http://localhost:8080 and is never cached so changes to files and @@ -130,18 +133,20 @@ https://github.com/elastic/{your-repository}/settings/pages ## Run without docker -If you have dotnet 8 installed you can use its CLI to publish a self-contained `docs-builder` native code +You can use the .NET CLI to publish a self-contained `docs-builder` native code binary. (On my M2 Pro mac the binary is currently 16mb) +Install [.NET 9.0](https://dotnet.microsoft.com/en-us/download/dotnet/9.0), then run: + ```bash -$ dotnet publish "src/docs-builder/docs-builder.csproj" +dotnet publish "src/docs-builder/docs-builder.csproj" ``` -The resulting binary `./.artifacts/publish/docs-builder/release/docs-builder` will run on machines without .NET installed +The resulting binary `./.artifacts/publish/docs-builder/release/docs-builder` will run on machines without .NET installed. # Performance To test performance it's best to build the binary and run outside of docker: -For refence here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to +For reference here's the `markitpy-doc` docset (50k markdown files) currently takes `14s` vs `several minutes` compared to existing surveyed tools diff --git a/docs/source/migration/file-structure.md b/docs/source/migration/file-structure.md new file mode 100644 index 00000000..675ffeab --- /dev/null +++ b/docs/source/migration/file-structure.md @@ -0,0 +1,61 @@ +--- +title: File structure +--- + +In both the AsciiDoc- and V3-based systems, file structure is largely independent from the resulting site navigation. + +## File and directory structure + +AsciiDoc files and assets (like images and videos) can be stored anywhere within the directories provided as `sources` for a given book in the central [`config.yaml` file in the /docs repo](https://github.com/elastic/docs/blob/master/conf.yaml). + +## Migration tool output + +The migration tool will add all MD files in a single directory. There will be one MD file for each web page in the migrated book. The name of each MD file and the URL in the new docs system are both derived from the URL of the AsciiDoc-built page. + +**Example:** Here's what happens with the +[Spans](https://www.elastic.co/guide/en/apm/guide/current/data-model-spans.html) page +in the APM docs: + +* **Old URL**: The URL for the page in the old system is: + `https://www.elastic.co/guide/en/apm/guide/current/data-model-spans.html`. + From the URL, we can determine the: + * _base URL_: `https://www.elastic.co/guide` + * _book ID_: `en/apm/guide` + * _version_: `current` + * _page ID_: `data-model-spans` +* **New filename**: The page ID determines the filename of the migrated MD file: +% `data-model-spans.mdx`. This file will be in the root directory of the directory containing the content for the `en/apm/guide` book. +* **New URL**: The new URL for this page in the new docs system will be `xxxx`. +* +Because a single AsciiDoc file can contain the content for multiple pages (or content +displayed on a single page could be spread across multiple AsciiDoc files), the `.asciidoc` +filename can be different than the `.md` filename. However, you should be able to locate +the source content if you know which web page it lives on. + +In the output from the migration tool the slug (for example, `en/apm/guide/data-model-spans`) +and the MD filename (for example, `data-model-spans.md`) are both derived from +the page ID, they don't _have_ to be the same. + +### Assets + +The migration tool creates an `images/` directory in the base directory for the doc set. +Inside the `images/` directory, there is subdirectory for each page ID that contains images. + +For example, the images that are used on the [Entity Analytics dashboard](https://www.elastic.co/guide/en/security/current/detection-entity-dashboard.html) page in the AsciiDoc Security book would be copied to +the following location in the migrated docs: + +``` +images + └─ detection-entity-dashboard + ├─ dashboards-anomalies-table.png + ├─ dashboards-entity-dashboard.png + ├─ dashboards-host-score-data.png + ├─ dashboards-run-job.png + └─ dashboards-user-score-data.png +``` + +These can be reorganized post-migration. + +### Reusable content + +Reusable content is lost during migration. \ No newline at end of file diff --git a/docs/source/migration/index.md b/docs/source/migration/index.md new file mode 100644 index 00000000..6ad0f1c8 --- /dev/null +++ b/docs/source/migration/index.md @@ -0,0 +1,13 @@ +--- +title: Migrate from Asciidoc to V3 +navigation_title: Migration guide +--- + +How to migrate content from Asciidoc to V3. + +1. Clone https://github.com/elastic/adoc-to-md +2. Run the migration tool +3. Copy the output to the appropriate directory in https://github.com/elastic/docs-content +4. Update the `docset.yml` file to reflect the new IA of the content set +5. Build your changes with https://github.com/elastic/docs-builder +6. Open a PR and merge your changes \ No newline at end of file diff --git a/docs/source/syntax/tables.md b/docs/source/syntax/tables.md index 0d10e117..ba8c6a7a 100644 --- a/docs/source/syntax/tables.md +++ b/docs/source/syntax/tables.md @@ -29,41 +29,3 @@ You can use the GFM table syntax to create a table: | ------- | --------------- | | USA | Washington D.C. | | Canada | Ottawa | - -% Neither of these work -% ## `{table}` Directive -% -% You can use the `table` directive which gives you configuration captions such as caption and alignment. -% -% ```{table} Countries and their capitals -% :widths: auto -% :align: right -% -% | Country | Capital | -% | ------- | --------------- | -% | USA | Washington D.C. | -% | Canada | Ottawa | -% ``` --> -% -%