diff --git a/.github/workflows/documentation.yml b/.github/workflows/documentation.yml index 40eb44b..8b24943 100644 --- a/.github/workflows/documentation.yml +++ b/.github/workflows/documentation.yml @@ -5,23 +5,32 @@ on: push permissions: contents: write + + jobs: build: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@master + - name: Set up Python 3.10 + uses: actions/setup-python@v2 + with: + python-version: '3.10' + - name: Build HTML uses: ammaraskar/sphinx-action@master + with: + docs-folder: "docs" - name: Upload artifacts - uses: actions/upload-artifact@v3 + uses: actions/upload-artifact@master with: name: html-docs path: docs/_build/html/ - name: Deploy - uses: peaceiris/actions-gh-pages@v3 + uses: peaceiris/actions-gh-pages@v4 if: github.ref == 'refs/heads/main' with: github_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.gitignore b/.gitignore index 11e7f85..5d56408 100644 --- a/.gitignore +++ b/.gitignore @@ -9,7 +9,9 @@ build/ .DS_Store */*/result.html styled_source.md +extra_styled_source.md *.icloud _build/ -make.bat \ No newline at end of file +make.bat + diff --git a/README.md b/README.md index d6e9423..26f76be 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ A library for containerizing local markdown content to be published into the Canvas learning management system. -:warning: This library is under active development and the interface is NOT stable. +[Documentation on Github Pages](https://ofloveandhate.github.io/markdown2canvas/) --- @@ -8,6 +8,10 @@ A library for containerizing local markdown content to be published into the Can The particular problem this library solves is that of putting Canvas content under version control, and also using Markdown for that content. Canvas pages are not well-suited to version control *per se*, because they live on the LMS. I wanted local files, with repos I can share with other designers and instrutors. +Further, I want to be able to find-and-replace across many pieces of Canvas content at once. Local files with my editor of choice is the way to do that; it's impossible on Canvas. Hence, this library. + +Additionally, uniform appearance and ability to change much with little effort. I wrote a "style" system that puts headers and footers around my content, eliminating repetitive and error-prone work. Want emester-specific text at the top of all your content? Trivial with `markdown2canvas`: just change the header file, re-publish, and do something better with your time than wait for Canvas pages to load. + A secondary problem this library solves is that of images. Images on Canvas are bare, and it's easy to end up with duplicate versions, as well as not have alt text. By using markdown/html under version control, I can write my alt text directly into page source, instead of using the crappy click-heavy interface on Canvas. --- @@ -20,138 +24,26 @@ Containerization is accomplished by making pages/assignments live in folders. T - `meta.json` -- a json file containing attributes. keys should be compliant with the expectations of `canvasapi`. - `source.md` -- a markdown file. Can contain latex math, html, references to local images, emoji using double-colon notation and shortcodes, and of course online images. - - ---- - -# Testing - -This library comes with several test pages/assignments: -- pages: - - `plain_text` -- source is just plain old text. start simple, ya know? - - `uses_latex` -- a page that contains latex math - - `has_remote_images` -- a page that has remote images embedded - - `has_local_images` -- a page that uses local images - - `uses_droplets` -- a page using [the Droplets framework from UWEX](https://media.uwex.edu/app/droplets/index.html). - - `uses_droplets_via_style` -- a page using [the Droplets framework from UWEX](https://media.uwex.edu/app/droplets/index.html). The code enabling Droplets comes from a header/footer contained in a style folder. The main purpose of this test page is the header/footer style thing. -- assignments: -- `programming_assignment` -- an assignment that has a local image - -The above list is not exhaustive. +You can also put needed files, images, etc in the folder for a piece of content. `markdown2canvas` aims to automate as much of the process as possible. --- # Installation -1. Clone the repo / pull from the repo -2. Move to repo location in terminal -3. `pip install .` If you already had it installed, then use `pip install . --upgrade` to make sure you get the newer version. - -## Critical setup step, do not skip this - -You must also define an environment variable called `CANVAS_CREDENTIAL_FILE`, which is the location of a `.py` file containing two variables: -1. `API_URL` -- a string, the url of how to access your Canvas install. - - At UW Eau Claire, it's `https://uweau.instructure.com/`. - - I cannot possibly tell you your url, but your local Canvas admin can. -2. `API_KEY` -- a string, the key you can get from Canvas. Here's [a link to a guide on how to generate yours](https://community.canvaslms.com/t5/Admin-Guide/How-do-I-obtain-an-API-access-token-in-the-Canvas-Data-Portal/ta-p/157). Do not share it with anyone -- having only this one piece of data, anyone can act as you. Protect it at least as much as you would any other password or sensitive information. - - -# Alternate-ish Installation for Windows Users - -These instructions are tested on Windows 11 on February 26, 2024. - -## Get Canvas Credentials and Make Canvas Credential File - this is the same step as above. - -Your first step will need to be to get a Canvas API Key. - -1. On Canvas, navigate Accounts -> Settings -2. Scroll to the button labeled `+ New Acces Token` -3. Add a description for yourself to know, later, what the access token is for and optionally add an expiration date. (I like to make a new one every semester, for safety.) -4. Copy the text of the token (you won't get to see this again) to a file that we will name `canvas_credential_file.py`. -5. Create a variable in `canvas_credential_file.py` named `API_KEY`, whose value is the string that we just copied from canvas. - - Additionally, add a second variable `API_URL` whose value is the string that is the general Canvas URL you use. For UWEC, this is `'https://uweau.instructure.com/'`. - - Ultimately, your `canvas_credential_file.py` will contain the lines: +Please see [the documentation](https://ofloveandhate.github.io/markdown2canvas/) -- we have two tutorials, one for Mac/Linux and one for Windows. - ``` - API_KEY = "stringofrandomcharacters" - API_URL = "https://uweau.instructure.com/" - ``` - -## Initial Setup - Using VS Code - -1. Install python via the Microsoft Store -2. Install VS code and GitBash - at UW Eau Claire this is done via the software center, you might use the Microsoft Store for this step as well. -3. Clone the markdown2canvas repo from github -4. Open VS code, open GitBash terminal and run the command - - ``` - pip install /path/to/markdown2canvas - ```` - - Then also run the command - - ``` - pip install lxml beautifulsoup4 - ``` - - Note that the default terminal that VSCode opens will be the Windows powershell, don't use that. - -## Generate necessary global variables - -1. Run the following command to make a file called `.bashrc` and save the location of your canvas credential file in your home directory. - - ``` - echo 'CANVAS_CREDENTIAL_FILE=h:\\path\\to\\canvas_credential_file.py' >> ~/.bashrc - ``` - - Note that you should be using `\\` here as directory separators because you are using Windows. If you use `/` you run the risk of the operating system not understanding the path. - -2. Open a new git bash terminal and see if the following works: - - ``` - echo $CANVAS_CREDENTIAL_FILE - ``` - - if not, you might need to run the following command in your bash terminal: - - ``` - source ~/.bashrc - ``` --- -# Some quick examples - -This library is under active development. I suggest checking out the `test_*.py` files in the `test` folder for example code. - -Assuming you did my setup step, defining the environment variable and creating that file. Do that first. - -### Download all pages, with a filter on the name of the pages - -``` -import markdown2canvas as mc -course_id = 127210000000003099 # silviana's sandbox for development - -canvas = mc.make_canvas_api_obj() # gets link and api key via environment variable -course = canvas.get_course(course_id) - -destination = 'downloaded_pages' +# Some things you can do with this library -my_filter = lambda title: '📖' in title # pages about readings have an open book in their names. -mc.download_pages(destination, course, even_if_exists=True, name_filter=my_filter) -``` - ---- - -# Things you can do with this library +See the [the documentation](https://ofloveandhate.github.io/markdown2canvas/). This is just highlights in a root readme. ## Replacements during translation The purpose of this library is to increase modularity and flexibility, while reducing duplication in source code and allowing version control. I implemented a simple text replacement feature as part of this, so that I can create uniform appearances in my content without duplicate code. -That is, you can specify a set of string replacements using a .json file, and during translation from markdown to html, before uploading, each substitution happens. +That is, you can specify a set of string replacements using a `.json` file, and during translation from markdown to html, before uploading, each substitution happens. For example, you can create a `replacements.json` file in a folder at root level (relative to the folder for the course) called `_course_metadata`, and in this file put the content: @@ -164,13 +56,6 @@ For example, you can create a `replacements.json` file in a folder at root level } ``` -I'm using simple python `.replace` to do the replacements. There are consequences: -* It will replace strings exactly, there are no implemented efforts to allow patterns or functions. -* It's case sensitive, and includes exact spacing. -* The dollar signs above are NOT special. They're just a nice way to indicate the text will be replaced. - -⚠ī¸ Furthermore, I'm not sure what order the replacements will be done in, so if the target of one replacement includes the source of another, I can't guarantee you at this time that it will actually happen in a deterministic order. If you want this feature, please add it and submit a PR to this repo. - You can specify a default set of replacements to happen for every file (except those with overridden replacements). To do this, make a file `_course_metadata/defaults.json`, and create a record `"replacements": "relative/path/to/replacements_filename.json"`. The name of the replacements file is arbitrary, and it's relative to root of the course folder. To override the default replacements, put a record in the `meta.json` file for the content (page / assignment) of the form `"replacements": "relative/path/to/replacements_filename.json"`. @@ -180,8 +65,19 @@ Examples of content using replacements can be found in the `test/` folder of thi If a replacements file doesn't exist where you say it should, an exception will be raised at `publish` time for the `CanvasObject` (`Page` or `Assignment`). (You can construct a thing with a bad replacements file and not know it until you try to publish!) -## Referencing existing Canvas assignments, pages, and files +ℹī¸ I'm using simple Python `str.replace` to do the replacements. There are consequences: + +* It will replace strings exactly, there are no implemented efforts to allow patterns or functions. +* It's case sensitive, and includes exact spacing. +* The dollar signs above are NOT special. They're just a nice way to indicate the text will be replaced. + +⚠ī¸ Furthermore, I'm not sure what order the replacements will be done in, so if the target of one replacement includes the source of another, I can't guarantee you at this time that it will actually happen in a deterministic order. If you want this feature, please add it and submit a PR to this repo. + + + +## Reference existing Canvas assignments, pages, and files +Whereas I find it to be a pain to link to other content on Canvas using their editor, it's easy using `markdown2canvas`. To link to an existing Canvas assignment, use a link of the form @@ -197,7 +93,8 @@ To link to an existing Canvas file, use a link of the form Link to file called DavidenkoDiffEqn.pdf ``` -If the "existing" content doesn't yet exist when the content is published, a broken link will be made. This is ok. Think of the publishing process using Markdown2Canvas similar to the compilation of a TeX document, which is done in multiple passes. Once the page / assignment / file exists, the link will resolve correctly to it. Publish all content about twice to get links to resolve. +ℹī¸ If the "existing" content doesn't yet exist when the content is published, a broken link will be made. This is ok. Think of the publishing process using Markdown2Canvas similar to the compilation of a TeX document, which is done in multiple passes. Once the page / assignment / file exists, the link will resolve correctly to it. Publish all content about twice to get links to resolve. + ## Emoji conversion from shortcodes @@ -205,9 +102,13 @@ This library supports the automatic conversion of shortcodes to emoji. For exam Right now, emoji shortcodes can only be used in content, not in names of things -- shortcodes in names will not be emojized. + + ## Automatic uploading and warehousing of images and embedded content -List your images relative the folder containing the `source.md` for the content. +List your images relative the folder containing the `source.md` for the content and they'll automatically be uploaded when you publish. If the image is already uploaded, a link to the existing image will be generated instead of uploading. + + ## "Styling" -- Automatic inclusion of uniform headers and footers @@ -216,37 +117,11 @@ This library attempts to provide a way to uniformly style pages across sections * I have content in my course in four blocks, and want a different header for each block. But, copypasta for that header content sucks (avoid repitition is a key tenet of programming). So, I'd rather specify a "style" for the four blocks, and make the pages refer to the styles. * I use Droplets from UWEX, and don't want to have to put that code in *every single page*. I'd rather put it one place (or, at least, only a few places). So the html code that brings in Droplets lives in a header/footer html code file. -### Style basics - -Put your "style" folders in a folder in your course. In my DS150 course, I have the following structure: - -* `_styles/` - * `/generic.style` - * `/assignments.style` - -And in the `meta.json` file for the pages / assignments, I simply have to put the record `"style":"_styles/generic.style"` or whatever. - -As of July 2022, there is no default style -- if a page doesn't list a style, it gets no style. - -### Additional notes about styles: - -The folder for each style should have the following four files: -* `header.html` -* `header.md` -* `footer.md` -* `footer.html` - -They'll get concatenated around `source.md` in that order. HTML around markdown, and header/footer around source. - -If you want to use images in your header/footer, put them in the markdown part (even if they appear in html tags), and use the text `$PATHTOMD2CANVASSTYLEFILE` before typing the name of the file, so that its filepath gets listed correctly. (This happens via a simple string replacement) - - ## Assignments -### Possible Upload Types -In the `meta.json` file for an assignment, the submission type is encoded by a line that looks like the following. +Reduce your mental load by specify possible upload types in the `meta.json` file for an assignment. The submission type is encoded by a line that looks like the following. ``` "submission_types":['online_text_entry', 'online_url', 'media_recording', 'online_upload'] diff --git a/docs/canvas_objects.rst b/docs/canvas_objects.rst new file mode 100644 index 0000000..6529ff4 --- /dev/null +++ b/docs/canvas_objects.rst @@ -0,0 +1,50 @@ +Concrete Classes for Canvas Objects +-------------------------------------- + + + +.. autoclass:: markdown2canvas.Page + :members: + :undoc-members: + + +.. autoclass:: markdown2canvas.Assignment + :members: + :undoc-members: + +.. autoclass:: markdown2canvas.Image + :members: + :undoc-members: + + +.. autoclass:: markdown2canvas.Link + :members: + :undoc-members: + + +.. autoclass:: markdown2canvas.File + :members: + :undoc-members: + + +.. autoclass:: markdown2canvas.BareFile + :members: + :undoc-members: + + + + + + + +Base Classes +-------------- + +.. autoclass:: markdown2canvas.canvas_objects.CanvasObject + :members: + :undoc-members: + + +.. autoclass:: markdown2canvas.canvas_objects.Document + :members: + :undoc-members: \ No newline at end of file diff --git a/docs/conf.py b/docs/conf.py index 6a694b9..14eb543 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -14,7 +14,7 @@ # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration extensions = [ - 'sphinx.ext.autodoc', + 'sphinx.ext.autodoc','sphinx.ext.autosectionlabel' ] templates_path = ['_templates'] @@ -28,7 +28,20 @@ html_theme = 'bizstyle' html_static_path = ['_static'] - import os import sys -sys.path.insert(0, os.path.abspath('../markdown2canvas/')) \ No newline at end of file + +_HERE = os.path.dirname(__file__) +_ROOT_DIR = os.path.abspath(os.path.join(_HERE, '..')) +_PACKAGE_DIR = os.path.abspath(os.path.join(_HERE, '../markdown2canvas')) + +sys.path.insert(0, _ROOT_DIR) +sys.path.insert(0, _PACKAGE_DIR) + +# test the path; not strictly needed +import markdown2canvas + + +rst_prolog = """ +.. |markdowndefaults| replace:: :attr:`markdown2canvas.translation_functions.default_markdown_extensions` +""" diff --git a/docs/emoji.rst b/docs/emoji.rst new file mode 100644 index 0000000..390173a --- /dev/null +++ b/docs/emoji.rst @@ -0,0 +1,16 @@ +🙂 Emojification +================== + + +This library supports the automatic conversion of shortcodes to emoji. During translation from local file to Canvas content, `markdown2canvas` will attempt to emojize your text from "shortcodes". + +* You may also just use emoji directly without using short codes. +* We felt that also allowing short codes would be helpful. + +For example, `:open_book:` goes to 📖. + +* We use the `emoji` library to do this. Here's `a link to their documentation `_. +* `Shortcodes can be found here `_. + +Right now, emoji shortcodes can only be used in content, not in names of things -- shortcodes in names will not be emojized. + diff --git a/docs/exceptions.rst b/docs/exceptions.rst new file mode 100644 index 0000000..1cb36b9 --- /dev/null +++ b/docs/exceptions.rst @@ -0,0 +1,14 @@ +Exceptions +------------- + +.. autoclass:: markdown2canvas.exception.AlreadyExists + :members: + :undoc-members: + +.. autoclass:: markdown2canvas.exception.SetupError + :members: + :undoc-members: + +.. autoclass:: markdown2canvas.exception.DoesntExist + :members: + :undoc-members: \ No newline at end of file diff --git a/docs/free_functions.rst b/docs/free_functions.rst new file mode 100644 index 0000000..0c0c3de --- /dev/null +++ b/docs/free_functions.rst @@ -0,0 +1,30 @@ +Free functions +===================== + + +Setup functions +------------------- + +.. automodule:: markdown2canvas.setup_functions + :members: + + + +Functions for interacting with a course on Canvas +--------------------------------------------------- + +.. automodule:: markdown2canvas.course_interaction_functions + :members: + + +Functions markdown2canvas uses to translate from markdown to Canvas-html +-------------------------------------------------------------------------- + +.. automodule:: markdown2canvas.translation_functions + :members: + +.. autofunction:: markdown2canvas.canvas_objects.find_local_images + +.. autofunction:: markdown2canvas.canvas_objects.find_local_files + + diff --git a/docs/gotchas.rst b/docs/gotchas.rst new file mode 100644 index 0000000..4a15026 --- /dev/null +++ b/docs/gotchas.rst @@ -0,0 +1,12 @@ +đŸ’Ĩ Gotchas +=========== + + +There are some idiosyncrasies in `markdown2canvas`. This page is me doing my best to share them with you. + + + +Markdown lists and enumerations +---------------------------------- + +Markdown lists and enumerations MUST have a newline above them. If your list / enumeration doesn't render correctly on Canvas, then it probably needs a newline above it. \ No newline at end of file diff --git a/docs/history.rst b/docs/history.rst new file mode 100644 index 0000000..19f01a9 --- /dev/null +++ b/docs/history.rst @@ -0,0 +1,47 @@ +Why this library exists +======================== + +(This is silviana amethyst writing.) + +Why I wrote/maintain this library +------------------------------------- + +The particular problem this library solves is that of putting Canvas content under version control, and also using Markdown for that content. Canvas pages are not well-suited to version control *per se*, because they live on the LMS. I wanted local files, with repos I can share with other designers and instrutors. + +Further, I wanted to be able to find-and-replace across many pieces of Canvas content at once. Local files with my editor of choice is the way to do that; it's impossible on Canvas. Hence, this library. + +Additionally, uniform appearance and ability to change much with little effort. I wrote a "style" system that puts headers and footers around my content, eliminating repetitive and error-prone work. Want emester-specific text at the top of all your content? Trivial with `markdown2canvas`: just change the header file, re-publish, and do something better with your time than wait for the stupid Canvas editor to load. + +A secondary problem this library solves is that of images. Images on Canvas are bare, and it's easy to end up with duplicate versions, as well as not have alt text. By using markdown/html under version control, I can write my alt text directly into page source, instead of using the crappy click-heavy interface on Canvas. + + +I've successfully automated many mundane and error-prone tasks using this library, including: + +* Automating creation of Canvas assignments for the Webwork homework system +* Consistent and flexible styling and beautification of content across an entire course + +The cost of development and maintenance has paid itself off many times over, both in terms of mental load and time savings. + + +History 2021-2024 +----------------------- + +I (silviana) started writing this library in 2021 to meet the needs of a new course at UWEC, DS150: Computing in Python: Fundamentals and Procedural Programming. I also anticipated using it for the upcoming 2022 re-design of DS710, Programming for Data Science at UWEC/UW Extended Campus. + +Mckenzie started contributing to `markdown2canvas` in 2022 during that DS710 re-design, and she really ran with it as she applied it to her math courses. Mckenzie added significant features to this library, including + +* links to existing pages, assignments, quizzes, and files +* warnings when some content doesn't exist +* clarifications and consistency across parts of the library +* bugfixes + + +Allison contributing to the library in summer 2024 as silviana prepared to move to MPI, contributing: + +* improved unit testing +* documentation, particularly tutorials and notes +* finding more bugs and gotchas + +As I move to new things in August 2024, I wish you well. + + diff --git a/docs/index.rst b/docs/index.rst index c146db2..d60be44 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -12,7 +12,7 @@ This is the documentation for `markdown2canvas`, a Python library for containeri :maxdepth: 2 :caption: Contents: -Tutorials +Tutorials 👩‍đŸĢ ================ .. toctree:: @@ -24,36 +24,55 @@ Tutorials tutorials/writing_content tutorials/styling_content tutorials/text_replacements + tutorials/uploading_files tutorials/publishing_content + tutorials/markdown_python_extensions -Notes -======== +Usage notes +================================= .. toctree:: - :maxdepth: 1 + :maxdepth: 2 :caption: Some useful notes on aspects of using the library + gotchas + emoji on_meta_dot_json + markdown_1 making_links_to_existing_content - wishlist -Useful links -============== + + + + +Useful links 🔗 +================ * `Canvas API documentation at root level `_ * `Canvas API class documentation `_ I use this when chasing down how to set additional properties for various content types. +Implementation notes +======================= + +.. toctree:: + :maxdepth: 1 + :caption: Notes about library implementation + + wishlist + unit_tests + history + + Details 📝 ================ .. toctree:: - :maxdepth: 0 - :caption: Contents: + :maxdepth: 1 markdown2canvas diff --git a/docs/making_links_to_existing_content.rst b/docs/making_links_to_existing_content.rst index 1707168..227dcf6 100644 --- a/docs/making_links_to_existing_content.rst +++ b/docs/making_links_to_existing_content.rst @@ -1,9 +1,15 @@ 🔗 Making links to existing content =================================== -* You can use either markdown style links, or html style links. -* Specify the type of content to which you are linking by preceding the name of the content with that type. -* + +Notes for links to all content types +---------------------------------------- + +* You can use either markdown style links, or html style links. Below, I'll give examples using both syntaxes. +* Specify the type of content to which you are linking by preceding the name of the content with that type. + + * For example, `assignment:Test Assignment` will make a link to the assignment with name `Test Assignment`. + Link to an assignment ----------------------- @@ -14,7 +20,8 @@ To link to an existing Canvas assignment, use a link of the form .. code-block:: link to Test Assignment - [Test Assignment](assignment:Test Assignment) + + [link to Test Assignment](assignment:Test Assignment) The name must match exactly, including case. This is the name on Canvas, not the name of the containerized content on your local computer. That is, the thing after `assignment:` is the `name` field from `meta.json`. @@ -26,6 +33,7 @@ To link to an existing Canvas page, use a link of the form .. code-block:: Link to page titled Test Page + [Link to page titled Test Page](page:Test Page) The name must match exactly, including case. This is the name on Canvas, not the name of the containerized content on your local computer. That is, the thing after `page:` is the `name` field from `meta.json`. @@ -38,14 +46,15 @@ To link to an existing Canvas file, use a link of the form .. code-block:: Link to file called DavidenkoDiffEqn.pdf + [Link to file called DavidenkoDiffEqn.pdf](file:DavidenkoDiffEqn.pdf) The name must match exactly, including case. This is the name of the file on Canvas. There is currently no way to refer to multiple files of the same name in different folders on Canvas. If you want this, make an issue, or implement it yourself and make a PR. -Notes ------- +Notes about making links +-------------------------- What if the content doesn't (yet) exist? ****************************************** diff --git a/docs/markdown2canvas.rst b/docs/markdown2canvas.rst index 8eef4e6..49b26ba 100644 --- a/docs/markdown2canvas.rst +++ b/docs/markdown2canvas.rst @@ -8,5 +8,18 @@ This is the detailed list of all functions and types available in the library. Python libraries used: `canvasapi` -.. automodule:: markdown2canvas - :members: + +.. toctree:: + :maxdepth: 2 + + canvas_objects + exceptions + free_functions + + + + + + + + diff --git a/docs/markdown_1.rst b/docs/markdown_1.rst new file mode 100644 index 0000000..4b4de8e --- /dev/null +++ b/docs/markdown_1.rst @@ -0,0 +1,5 @@ +Making sure markdown renders when there is a
+=================================================== + +Any time you have a `
` (in a header, footer, body, etc.) make sure that the option `markdown="1"` is included. +Otherwise, markdown will not render properly on Canvas. diff --git a/docs/on_meta_dot_json.rst b/docs/on_meta_dot_json.rst index 06b2487..7850ed6 100644 --- a/docs/on_meta_dot_json.rst +++ b/docs/on_meta_dot_json.rst @@ -2,30 +2,133 @@ On the `meta.json` file ========================= -The `meta.json` file should be present in every containerized content folder. +The `meta.json` file must be present in every containerized content folder. -Valid properties ------------------ +meta.json for ALL content types +********************************* -Assignments -***************** +Required: +* `name` -- the name of the thing as it will appear in Canvas. -Pages -****** +Optional. If not used, will not be done. +* `modules` -- a list of strings, the names of the modules to put the content in. +* `indent` -- the depth of indentation within the module -Links -******* -Files -******* +meta.json for Document types +********************************** +Both `Assignment` and `Page` are documents, in that they have a body. + +These are optional. If you provide a default in `_course_metadata/defaults.json`, then those will be used unless overridden per-object. + +* `style` -- filepath relative to course root. The name of the folder containing the headers/footers. +* `replacements` -- filepath relative to course root. The name of the .json file containing the list of replacements. + + + +meta.json for Assignments +**************************** + +Submission type +################### + +In the `meta.json` file for an assignment, the submission type is encoded by a line that looks like the following. + +.. code-block:: + + "submission_types":['online_text_entry', 'online_url', 'media_recording', 'online_upload'] + +These are four of the five upload types available with Canvas. The other is an `annotation`, but I've never used those. You may any sublist of this list. + + +If `online_upload`, allowed extensions +######################################## + +If you choose to allow the `online_upload` submission type, you may also specify the allowable file types by including an allowed extensions list in your `meta.json` file for the assignment. + +.. code-block:: + + "allowed_extensions": ["pdf","docx"] + +Assignment group name +####################### + +I put my assignments in groups according to my Syllabus. `markdown2canvas` lets me express this programmatically by setting the `assignment_group_name` property in `meta.json`. For example, in the `meta.json` for all parts of my semester project, I use: + +.. code-block:: + + "assignment_group_name": "Project" + +At this time, it is not built-in to use `markdown2canvas` to set the weight of these groups in the gradebook -- I've just been doing that in Canvas. If you want a programmatic way, I suggest you use `canvasapi`, because `mc` already depends on that. You could just write a script in your `_tools` folder to do it and put some new .json file in your `_course_metadata` folder. There are always ways. + +Due dates +########### + +Use these key-value pairs to set due dates for assignments: + +.. code-block:: + + unlock_at + lock_at + due_at + +The formatting of these strings is likely to be a pain in the ass. The place I've most used this was in `webwork2canvas` -- it's in `the tools folder (NOT MODULE) of the markdown2canvas repo `_ + +More direct Canvas properties +################################ + +These are plucked from `meta.json` and set into the properties at publish-time. + +* `external_tool_tag_attributes` +* `omit_from_final_grade` +* `grading_type` + +If you want additional properties, you'll need to modify `mc.Assignment._set_from_metadata` to pass them through. Full generic passthrough seems difficult to achieve (more difficult than I felt was worth it at this moment), since `canvasapi` might complain about invalid keys and the list might change. I would need a list to populate from, and this probably means webcrawling. You do it. + + + + + +meta.json for Pages +*********************** + +Nothing beyond that for `Documents` of any type. + + + +meta.json for Links +************************ + +Required: + +* `external_url` -- the url to map to. +* `name` -- of course. This is the string of text which will appear in Canvas, on which students will click. + + + +meta.json for Files +************************ + +Required: + +* `filename` -- the name of the file, relative to the folder containing the `meta.json`. This must be a strict match. + +Optional: + +* `title` -- the name of the item inside a Canvas module in which the `File` is included. What happens if I specify a property / key that's not used or is invalid? ------------------------------------------------------------------------------ +***************************************************************************** + +* Extra keys are ignored with no message. +* Missing required keys hopefully WILL generate a problem!!! + +The `meta.json` includes some things for Canvas, some things for `markdown2canvas`, and could, if you wish and write the code, some things for your creative uses, too. diff --git a/docs/requirements.txt b/docs/requirements.txt index a411efc..c1f96fb 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1 +1,3 @@ -canvasapi \ No newline at end of file +canvasapi +Pygments +markdown \ No newline at end of file diff --git a/docs/snippets/publication_process_document.rst b/docs/snippets/publication_process_document.rst new file mode 100644 index 0000000..944656e --- /dev/null +++ b/docs/snippets/publication_process_document.rst @@ -0,0 +1,12 @@ +#. :doc:`The style is applied <../tutorials/styling_content>`. + + #. The markdown header and footer from the used style are assembled around your `source.md`. + #. The html header and footer from the used style are assembled around that. + +#. :doc:`Replacements are done <../tutorials/text_replacements>`. +#. :doc:`Emoji shortcodes are emojified <../emoji>` +#. The markdown is translated to HTML. +#. :doc:`Links to existing content are implemented <../making_links_to_existing_content>`, further modifying the HTML. +#. Local files and images which are linked-to in the source are uploaded (if necessary), and the HTML is adjusted to use the Canvas links to that content. +#. Content properties are set from `meta.json`. +#. The content is placed in modules as described in `meta.json`. \ No newline at end of file diff --git a/docs/tutorials/making_your_first_course.rst b/docs/tutorials/making_your_first_course.rst index 1566b77..db30bf3 100644 --- a/docs/tutorials/making_your_first_course.rst +++ b/docs/tutorials/making_your_first_course.rst @@ -192,7 +192,7 @@ I use a script to help me publish my content. đŸŽ¯ Let's add it: course_folder/_tools/content_all # a txt file with names of content folders -Here's a script I use in DS710. đŸŽ¯ Copy-paste it. +Here's a script I use in DS710. đŸŽ¯ Copy-paste it into `_tools/publish_ready_content.py`. .. literalinclude:: ../../example/starter_course/_tools/publish_ready_content.py @@ -210,7 +210,12 @@ Note that you just list the folder, and `markdown2canvas` does all the work with Publish the content!!!!! -------------------------- -Now, assuming you've completed the setup steps (Mac/Linux: saving your API key and URL in a .py file, and specifying the name of the file via an environment variable called `CANVAS_CREDENTIALS_FILE`), you should be able to publish the content to your course. +Now, assuming you've completed the setup steps + +* :doc:`setup_mac_linux` +* :doc:`setup_windows` + +you should be able to publish the content to your course. đŸŽ¯ Be sure you copied in your Canvas course number to the `_tools/publish_ready_content.py` script! diff --git a/docs/tutorials/markdown_python_extensions.rst b/docs/tutorials/markdown_python_extensions.rst new file mode 100644 index 0000000..69bd0fa --- /dev/null +++ b/docs/tutorials/markdown_python_extensions.rst @@ -0,0 +1,63 @@ + + +Customize translation from Markdown to HTML via extensions to `markdown` +============================================================================ + + +This library, `markdown2canvas`, essentially acts as a wrapper around a translation function `markdown2canvas.translation_functions.markdown2html`, or just `markdown2html` for short. The `markdown2html` function uses the Python library `markdown` (`link to library `_) to translate. You can customize the behaviour of this translation using "extensions" to the `markdown` library. + + +Background +------------ + +Before I tell you how to use your own custom set of extensions to `markdown`, here's a brief rundown of how `markdown2canvas` works when you publish a page or assignment to Canvas: + +.. include:: ../snippets/publication_process_document.rst + +This tutorial is about customizing the first translation to HTML, by using `markdown` extensions. + + +Provided default `markdown` extensions +-------------------------------------------- + +This library provides a decent set of default extensions, chosen for + +* Syntax highlighting of code +* Translating markdown tables +* Trying to make sure that markdown inside of HTML is translated, too + +The specific default list is |markdowndefaults|. + + + +Specifying your own list of extensions +----------------------------------------- + +You may wish to use your own set of extensions. Easy! Just specify the list in your `defaults.json` file. + +#. Edit `_course_metadata/defaults.json`. +#. Add a key-value pair. + + * Key: `markdown_extensions` + * Value: A list of strings which are names of extensions. + + + + +Where to find more extensions +---------------------------------- + +I have found `the official markdown Extensions documentation `_ to be very helpful. + +Cruise the list, find one that you need, and try it out by: + +* Add it to the list of extensions you are using +* Publish the content to a test or sandbox course on Canvas +* Check it out! Did it do what you needed? + + +Limitations +--------------- + +ℹī¸ Known limitation: This system is missing methods for you writing your own extensions to `markdown`. I do not know what happens if your list of extensions includes references to your own, but I suspect errors because they'll be strings, not Python modules, classes, or functions. + diff --git a/docs/tutorials/setup_mac_linux.rst b/docs/tutorials/setup_mac_linux.rst index 865e07d..9b13be4 100644 --- a/docs/tutorials/setup_mac_linux.rst +++ b/docs/tutorials/setup_mac_linux.rst @@ -19,10 +19,10 @@ Canvas credentials, do not skip this You must define an environment variable called `CANVAS_CREDENTIAL_FILE`, which is the location of a `.py` file containing two variables: #. `API_URL` -- a string, the url of how to access your Canvas install. - * At UW Eau Claire, it's `https://uweau.instructure.com/`. + * At UW Eau Claire, it's `uweau.instructure.com `_. * I cannot possibly tell you your url, but your local Canvas admin can. -#. `API_KEY` -- a string, the key you can get from Canvas. Here's [a link to a guide on how to generate yours](https://community.canvaslms.com/t5/Admin-Guide/How-do-I-obtain-an-API-access-token-in-the-Canvas-Data-Portal/ta-p/157). Do not share it with anyone -- having only this one piece of data, anyone can act as you. Protect it at least as much as you would any other password or sensitive information. +#. `API_KEY` -- a string, the key you can get from Canvas. Here's `a link to a guide on how to generate yours `_. Do not share it with anyone -- having only this one piece of data, anyone can act as you. Protect it at least as much as you would any other password or sensitive information. Thus, you should have a Python script somewhere, like this: diff --git a/docs/tutorials/styling_content.rst b/docs/tutorials/styling_content.rst index 9e2d77c..6ab78f4 100644 --- a/docs/tutorials/styling_content.rst +++ b/docs/tutorials/styling_content.rst @@ -1,3 +1,4 @@ + Styling Pages and Assignments =============================== @@ -39,10 +40,14 @@ I hope they're self-documenting in purpose and content. Here's what's in the `h -
+
+ +Note the `markdown="1"` included in the `
` above. Any time you include a `
`, make sure that this appears. +Otherwise, markdown inside that `div` will not get translated to HTML, and thus will look incorrect. Unless, of course, you want raw markdown in your content, I could see some use-cases! + The footer simply closes the `div` I opened in the header: .. code-block:: html diff --git a/docs/tutorials/text_replacements.rst b/docs/tutorials/text_replacements.rst index 6c656ba..e670aa3 100644 --- a/docs/tutorials/text_replacements.rst +++ b/docs/tutorials/text_replacements.rst @@ -22,15 +22,21 @@ Note that `_course_metadata/replacements.json` is just a regular old JSON file. - Usage -------- +Keep in mind that the order of replacement is unspecified. Thus, it is important to choose keys that will not appear within values, and will not appear within source documents where replacement is undesired. + Custom text replacements per-content -------------------------------------- Content can override which replacements file is used, say in case you want all your assignments of a certain type to use one style, and the pages of another type to use their own. It's easy. -In the `meta.json` for the content you want to use the non-default replacements file, specify the key-value pair `'replacements':'path/to/custom_replacements.json'` +In the `meta.json` for the content you want to use the non-default replacements file, specify the key-value pair + +.. code-block:: + + "replacements":"path/to/custom_replacements.json" +The path is relative to the root of the course folder. \ No newline at end of file diff --git a/docs/tutorials/uploading_files.rst b/docs/tutorials/uploading_files.rst new file mode 100644 index 0000000..93169b0 --- /dev/null +++ b/docs/tutorials/uploading_files.rst @@ -0,0 +1,43 @@ +How to upload a file +-------------------------------------------------------------------------- + +When uploading a file `FILE.XXX`, a `meta.json` file should be created in a folder named `FILE.file`, +specifying where the file is sent on Canvas. + +The `meta.json` file +==================== + +Filenames and titles of files are distinct on Canvas: +the latter is what you will see when the file is placed in a module, while the former is what is shown in the file structure. + +You can place a file in as many modules as you wish by specifying the modules in the `meta.json` file. +The key `module` has a value which is a list of names of modules in the Canvas course. +If no module with the specified name exists, a module will be created to house the file. + +The `destination` key specifies where in the file structure you would like the file to be placed. + +Note that while a file cannot be simultaneously placed in multiple file structure locations using `meta.json`, if `meta.json` is updated, +the file will **not** automatically be deleted from any previous location unless that instance is specifically deleted. + + +Example +======= + +If the `meta.json` file looks like: + +.. code-block:: + + { + "type":"file", + "title":"Syllabus", + "filename":"F24_Math100_syllabus.pdf", + "modules":["Course Information", "Week 1"], + "destination": "course_info/syllabus_schedule" + } + +then the file in question will be named `F24_Math100_syllabus.pdf` and put into two modules: `Course Information` and `Week 1`. +Within these two modules, its title will appear to students as `Syllabus`. The file will be located in `course_info/syllabus_schedule`, +which will be created if it did not already exist. + + + diff --git a/docs/unit_tests.rst b/docs/unit_tests.rst new file mode 100644 index 0000000..55417e6 --- /dev/null +++ b/docs/unit_tests.rst @@ -0,0 +1,18 @@ +Unit tests +============== + +This library comes with several test pages/assignments: + +* pages: + + * `plain_text` -- source is just plain old text. start simple, ya know? + * `uses_latex` -- a page that contains latex math + * `has_remote_images` -- a page that has remote images embedded + * `has_local_images` -- a page that uses local images + * `uses_droplets` -- a page using [the Droplets framework from UWEX](https://media.uwex.edu/app/droplets/index.html). + * `uses_droplets_via_style` -- a page using [the Droplets framework from UWEX](https://media.uwex.edu/app/droplets/index.html). The code enabling Droplets comes from a header/footer contained in a style folder. The main purpose of this test page is the header/footer style thing. + +* assignments: +* `programming_assignment` -- an assignment that has a local image + +The above list is not exhaustive. diff --git a/markdown2canvas/__init__.py b/markdown2canvas/__init__.py index 92b9cf1..78d0ff4 100644 --- a/markdown2canvas/__init__.py +++ b/markdown2canvas/__init__.py @@ -1,1695 +1,65 @@ -import canvasapi -import os.path as path -import os -import requests +""" +`markdown2canvas`, a library for containerizing and publishing Canvas content. +Containerization of content is via filesystem folders with a `meta.json` file specifying type of content. Some content types like Assignment and Page use `source.md`, while others like File and Image are just a `meta.json` plus the files. +Publishing content is via the `.publish` member function for the canvas object, like -import logging +``` +my_assignment.publish(course) +``` -logging.basicConfig(encoding='utf-8') +Documentation may be found at the GitHub pages for this library. Use it. -import datetime -today = datetime.datetime.today().strftime("%Y-%m-%d") -log_level=logging.DEBUG +A more complete example might be -log_dir = path.join(path.normpath(os.getcwd()), '_logs') +``` +import markdown2canvas as mc +canvas_url = "https://uweau.instructure.com/" # đŸŽ¯ REPLACE WITH YOUR URL -if not path.exists(log_dir): - os.mkdir(log_dir) +# get the course. +course_id = 705022 # đŸŽ¯ REPLACE WITH YOUR NUMBER!!!!!!!!!!!!!!!!! +course = canvas.get_course(course_id) -log_filename = path.join(log_dir, f'markdown2canvas_{today}.log') +# make the API object. this is from the `canvasapi` library, NOT something in `markdown2canvas`. +canvas = mc.make_canvas_api_obj(url=canvas_url) +my_assignment = mc.Assignent('path_to_assignment") -log_encoding = 'utf-8' +# finally, publish +my_assignment.publish(course) +``` +""" -root_logger = logging.getLogger() -root_logger.setLevel(log_level) -handler = logging.FileHandler(log_filename, 'a', log_encoding) -root_logger.addHandler(handler) +# the root-level file for `markdown2canvas` -logging.debug(f'starting logging at {datetime.datetime.now()}') +__version__ = '0.' +__author__ = 'silviana amethyst, Mckenzie West, Allison Beemer' +__all__ = ['logging','exception','translation_functions','course_interaction_functions', + 'CanvasObject', 'Document', 'Page', 'Assignment', 'Image', 'File', 'BareFile', 'Link', + 'canvas2markdown','tool'] -logging.debug(f'reducing logging level of `requests` to WARNING') -logging.getLogger('canvasapi.requester').setLevel(logging.WARNING) -logging.getLogger('requests').setLevel(logging.WARNING) +import markdown2canvas.logging as logging +import markdown2canvas.exception as exception +from .setup_functions import get_canvas_key_url, make_canvas_api_obj -def is_file_already_uploaded(filename,course): - """ - returns a boolean, true if there's a file of `filename` already in `course`. - - This function wants the full path to the file. - """ - return ( not find_file_in_course(filename,course) is None ) - - - - -def find_file_in_course(filename,course): - """ - Checks to see of the file at `filename` is already in the "files" part of `course`. - - It tests filename and size as reported on disk. If it finds a match, then it's up. - - This function wants the full path to the file. - """ - import os - - base = path.split(filename)[1] - - files = course.get_files() - for f in files: - if f.filename==base and f.size == path.getsize(filename): - return f - - return None - - - - - -def is_page_already_uploaded(name,course): - """ - returns a boolean indicating whether a page of the given `name` is already in the `course`. - """ - return ( not find_page_in_course(name,course) is None ) - - -def find_page_in_course(name,course): - """ - Checks to see if there's already a page named `name` as part of `course`. - - tests merely based on the name. assumes assignments are uniquely named. - """ - import os - pages = course.get_pages() - for p in pages: - if p.title == name: - return p - - return None - - - -def is_assignment_already_uploaded(name,course): - """ - returns a boolean indicating whether an assignment of the given `name` is already in the `course`. - """ - return ( not find_assignment_in_course(name,course) is None ) - - -def find_assignment_in_course(name,course): - """ - Checks to see if there's already an assignment named `name` as part of `course`. - - tests merely based on the name. assumes assingments are uniquely named. - """ - import os - assignments = course.get_assignments() - for a in assignments: - - if a.name == name: - return a - - return None - - - - -def get_canvas_key_url(): - """ - reads a file using an environment variable, namely the file specified in `CANVAS_CREDENTIAL_FILE`. - - We need the - - * API_KEY - * API_URL - - variables from that file. - """ - from os import environ - - cred_loc = environ.get('CANVAS_CREDENTIAL_FILE') - if cred_loc is None: - raise SetupError('`get_canvas_key_url()` needs an environment variable `CANVAS_CREDENTIAL_FILE`, containing the full path of the file containing your Canvas API_KEY, *including the file name*') - - # yes, this is scary. it was also low-hanging fruit, and doing it another way was going to be too much work - with open(path.join(cred_loc),encoding='utf-8') as cred_file: - exec(cred_file.read(),locals()) - - if isinstance(locals()['API_KEY'], str): - logging.info(f'using canvas with API_KEY as defined in {cred_loc}') - else: - raise SetupError(f'failing to use canvas. Make sure that file {cred_loc} contains a line of code defining a string variable `API_KEY="keyhere"`') - - return locals()['API_KEY'],locals()['API_URL'] - - -def make_canvas_api_obj(url=None): - """ - - reads the key from a python file, path to which must be in environment variable CANVAS_CREDENTIAL_FILE. - - optionally, pass in a url to use, in case you don't want the default one you put in your CANVAS_CREDENTIAL_FILE. - """ - - key, default_url = get_canvas_key_url() - - if not url: - url = default_url - - return canvasapi.Canvas(url, key) - - - -def generate_course_link(type,name,all_of_type,courseid=None): - ''' - Given a type (assignment or page) and the name of said object, generate a link - within course to that object. - ''' - if type in ['page','quiz']: - the_item = next( (p for p in all_of_type if p.title == name) , None) - elif type == 'assignment': - the_item = next( (a for a in all_of_type if a.name == name) , None) - elif type == 'file': - the_item = next( (a for a in all_of_type if a.display_name == name) , None) - if the_item is None: # Separate case to allow change of filenames on Canvas to names that did exist - the_item = next( (a for a in all_of_type if a.filename == name) , None) - # Canvas retains the name of the file uploaded and calls it `filename`. - # To access the name of the document seen in the Course Files, we use `display_name`. - else: - the_item = None - - - if the_item is None: - print(f"ℹī¸ No content of type `{type}` named `{name}` exists in this Canvas course. Either you have the name incorrect, the content is not yet uploaded, or you used incorrect type before the colon") - elif type == 'file' and not courseid is None: - # Construct the url with reference to the coruse its coming from - file_id = the_item.id - full_url = the_item.url - stopper = full_url.find("files") - - html_url = full_url[:stopper] + "courses/" + str(courseid) + "/files/" + str(file_id) - - return html_url - elif type == 'file': - # Construct the url - removing the "download" portion - full_url = the_item.url - stopper = full_url.find("download") - return full_url[:stopper] - else: - return the_item.html_url - - - -def find_in_containing_directory_path(target): - import pathlib - - target = pathlib.Path(target) - - here = pathlib.Path('.').absolute() - - testme = here / target - - found = testme.exists() - - while (not found) and here.parent!=here: - here = here.parent - testme = here / target - found = testme.exists() - - - if not found: - raise FileNotFoundError('unable to find {} in a containing folder of {}'.format(target, pathlib.Path('.').absolute())) - - return here / target - - - -def preprocess_replacements(contents, replacements_path): - """ - attempts to read in a file containing substitutions to make, and then makes those substitutions - """ - - if replacements_path is None: - return contents - with open(replacements_path,'r',encoding='utf-8') as f: - import json - replacements = json.loads(f.read()) - - for source, target in replacements.items(): - contents = contents.replace(source, target) - - return contents - - - - -def preprocess_markdown_images(contents,style_path): - - rel_style_path = find_in_containing_directory_path(style_path) - - contents = contents.replace('$PATHTOMD2CANVASSTYLEFILE',str(rel_style_path)) - - return contents - - -def get_default_property(key, helpstr): - - defaults_name = find_in_containing_directory_path(path.join("_course_metadata","defaults.json")) - - try: - logging.info(f'trying to use defaults from {defaults_name}') - with open(defaults_name,'r',encoding='utf-8') as f: - import json - defaults = json.loads(f.read()) - - if key in defaults: - return defaults[key] - else: - print(f'no default `{key}` specified in {defaults_name}. add an entry with key `{key}`, being {helpstr}') - return None - - except Exception as e: - print(f'WARNING: failed to load defaults from `{defaults_name}`. either you are not at the correct location to be doing this, or you need to create a json file at {defaults_name}.') - return None - - -def get_default_style_name(): - return get_default_property(key='style', helpstr='a path to a file relative to the top course folder') - -def get_default_replacements_name(): - return get_default_property(key='replacements', helpstr='a path to a json file containing key:value pairs of text-to-replace. this path should be expressed relative to the top course folder') - - - - -def apply_style_markdown(sourcename, style_path, outname): - from os.path import join - - # need to add header and footer. assume they're called `header.md` and `footer.md`. we're just going to concatenate them and dump to file. - - with open(sourcename,'r',encoding='utf-8') as f: - body = f.read() - - with open(join(style_path,'header.md'),'r',encoding='utf-8') as f: - header = f.read() - - with open(join(style_path,'footer.md'),'r',encoding='utf-8') as f: - footer = f.read() - - - contents = f'{header}\n{body}\n{footer}' - contents = preprocess_markdown_images(contents, style_path) - - with open(outname,'w',encoding='utf-8') as f: - f.write(contents) - - - - -def apply_style_html(translated_html_without_hf, style_path, outname): - from os.path import join - - # need to add header and footer. assume they're called `header.html` and `footer.html`. we're just going to concatenate them and dump to file. - - with open(join(style_path,'header.html'),'r',encoding='utf-8') as f: - header = f.read() - - with open(join(style_path,'footer.html'),'r',encoding='utf-8') as f: - footer = f.read() - - - return f'{header}\n{translated_html_without_hf}\n{footer}' - - - - - -def markdown2html(filename, course, replacements_path): - """ - This is the main routine in the library. - - This function returns a string of html code. - - It does replacements, emojizes, converts markdown-->html via `markdown.markdown`, and does page, assignment, and file reference link adjustments. - - If `course` is None, then you won't get some of the functionality. In particular, you won't get link replacements for references to other content on Canvas. - - If `replacements_path` is None, then no replacements, duh. Otherwise it should be a string or Path object to an existing json file containing key-value pairs of strings to replace with other strings. - """ - if course is None: - courseid = None - else: - courseid = course.id - - root = path.split(filename)[0] - - import emoji - import markdown - from bs4 import BeautifulSoup - - - with open(filename,'r',encoding='utf-8') as file: - markdown_source = file.read() - - markdown_source = preprocess_replacements(markdown_source, replacements_path) - - emojified = emoji.emojize(markdown_source) - - - html = markdown.markdown(emojified, extensions=['codehilite','fenced_code','md_in_html','tables','nl2br']) # see https://python-markdown.github.io/extensions/ - soup = BeautifulSoup(html,features="lxml") - - all_imgs = soup.findAll("img") - for img in all_imgs: - src = img["src"] - if ('http://' not in src) and ('https://' not in src): - img["src"] = path.join(root,src) - - all_links = soup.findAll("a") - course_page_and_assignments = {} - if any(l['href'].startswith("page:") for l in all_links) and course: - course_page_and_assignments['page'] = course.get_pages() - if any(l['href'].startswith("assignment:") for l in all_links) and course: - course_page_and_assignments['assignment'] = course.get_assignments() - if any(l['href'].startswith("quiz:") for l in all_links) and course: - course_page_and_assignments['quiz'] = course.get_quizzes() - if any(l['href'].startswith("file:") for l in all_links) and course: - course_page_and_assignments['file'] = course.get_files() - for f in all_links: - href = f["href"] - root_href = path.join(root,href) - split_at_colon = href.split(":",1) - if path.exists(path.abspath(root_href)): - f["href"] = root_href - elif course and split_at_colon[0] in ['assignment','page','quiz','file']: - type = split_at_colon[0] - name = split_at_colon[1].strip() - get_link = generate_course_link(type,name,course_page_and_assignments[type],courseid) - if get_link: - f["href"] = get_link - - - return str(soup) - - - - - -def find_local_images(html): - """ - constructs a map of local url's : Images - """ - from bs4 import BeautifulSoup - - soup = BeautifulSoup(html,features="lxml") - - local_images = {} - - all_imgs = soup.findAll("img") - - if all_imgs: - for img in all_imgs: - src = img["src"] - if src[:7] not in ['https:/','http://']: - local_images[src] = Image(path.abspath(src)) - - return local_images - - - - - -def adjust_html_for_images(html, published_images, courseid): - """ - - published_images: a dict of Image objects, which should have been published (so we have their canvas objects stored into them) - - this function edits the html source, replacing local url's - with url's to images on Canvas. - """ - from bs4 import BeautifulSoup - - soup = BeautifulSoup(html,features="lxml") - - all_imgs = soup.findAll("img") - if all_imgs: - for img in all_imgs: - src = img["src"] - if src[:7] not in ['https:/','http://']: - # find the image in the list of published images, replace url, do more stuff. - local_img = published_images[src] - img['src'] = local_img.make_src_url(courseid) - img['class'] = "instructure_file_link inline_disabled" - img['data-api-endpoint'] = local_img.make_api_endpoint_url(courseid) - img['data-api-returntype'] = 'File' - - return str(soup) - - #

- # hauser_menagerie.jpg - #

- - - - -def find_local_files(html): - """ - constructs a list of BareFiles, so that they can later be replaced with a url to a canvas thing - """ - from bs4 import BeautifulSoup - - soup = BeautifulSoup(html,features="lxml") - - local_files = {} - - all_links = soup.findAll("a") - - if all_links: - for file in all_links: - href = file["href"] - if path.exists(path.abspath(href)): - local_files[href] = BareFile(path.abspath(href)) - - return local_files - - - -def adjust_html_for_files(html, published_files, courseid): - - - # need to write a url like this : - # Download - - - from bs4 import BeautifulSoup - - soup = BeautifulSoup(html,features="lxml") - - all_files = soup.findAll("a") - - if all_files: - for file in all_files: - href = file["href"] - if path.exists(path.abspath(href)): - # find the image in the list of published images, replace url, do more stuff. - local_file = published_files[href] - file['href'] = local_file.make_href_url(courseid) - file['class'] = "instructure_file_link instructure_scribd_file" - file['title'] = local_file.name # what it's called when you download it??? - file['data-api-endpoint'] = local_file.make_api_endpoint_url(courseid) - file['data-api-returntype'] = 'File' - - return str(soup) - - - -def get_root_folder(course): - for f in course.get_folders(): - if f.full_name == 'course files': - return f - - - - - - - -class AlreadyExists(Exception): - - def __init__(self, message, errors=""): - # Call the base class constructor with the parameters it needs - super().__init__(message) - - self.errors = errors - -class SetupError(Exception): - - def __init__(self, message, errors=""): - # Call the base class constructor with the parameters it needs - super().__init__(message) - - self.errors = errors - - - - -class DoesntExist(Exception): - """ - Used when getting a thing, but it doesn't exist - """ - - def __init__(self, message, errors=""): - # Call the base class constructor with the parameters it needs - super().__init__(message) - - self.errors = errors - - - - - - -def get_assignment_group_id(assignment_group_name, course, create_if_necessary=False): - - existing_groups = course.get_assignment_groups() - - if not isinstance(assignment_group_name,str): - raise RuntimeError(f'assignment_group_name must be a string, but I got {assignment_group_name} of type {type(assignment_group_name)}') - - - for g in existing_groups: - if g.name == assignment_group_name: - return g.id - - - - if create_if_necessary: - msg = f'making new assignment group `{assignment_group_name}`' - logging.info(msg) - - group = course.create_assignment_group(name=assignment_group_name) - group.edit(name=assignment_group_name) # this feels stupid. didn't i just request its name be this? - - return group.id - else: - raise DoesntExist(f'cannot get assignment group id because an assignment group of name {assignment_group_name} does not already exist, and `create_if_necessary` is set to False') - - - - - -def create_or_get_assignment(name, course, even_if_exists = False): - - if is_assignment_already_uploaded(name,course): - if even_if_exists: - return find_assignment_in_course(name,course) - else: - raise AlreadyExists(f"assignment {name} already exists") - else: - # make new assignment of name in course. - return course.create_assignment(assignment={'name':name}) - - - -def create_or_get_page(name, course, even_if_exists): - if is_page_already_uploaded(name,course): - - if even_if_exists: - return find_page_in_course(name,course) - else: - raise AlreadyExists(f"page {name} already exists") - else: - # make new assignment of name in course. - result = course.create_page(wiki_page={'body':"empty page",'title':name}) - return result - - - - -def create_or_get_module(module_name, course): - - try: - return get_module(module_name, course) - except DoesntExist as e: - return course.create_module(module={'name':module_name}) - - - - -def get_module(module_name, course): - """ - returns - * Module if such a module exists, - * raises if not - """ - modules = course.get_modules() - - for m in modules: - if m.name == module_name: - return m - - raise DoesntExist(f"tried to get module {module_name}, but it doesn't exist in the course") - - -def get_subfolder_named(folder, subfolder_name): - - assert '/' not in subfolder_name, "this is likely broken if subfolder has a / in its name, / gets converted to something else by Canvas. don't use / in subfolder names, that's not allowed" - - current_subfolders = folder.get_folders() - for f in current_subfolders: - if f.name == subfolder_name: - return f - - raise DoesntExist(f'a subfolder of {folder.name} named {subfolder_name} does not currently exist') - - -def delete_module(module_name, course, even_if_exists): - - if even_if_exists: - try: - m = get_module(module_name, course) - m.delete() - except DoesntExist as e: - return - - else: - # this path is expected to raise if the module doesn't exist - m = get_module(module_name, course) - m.delete() +import markdown2canvas.translation_functions as translation_functions +import markdown2canvas.course_interaction_functions as course_interaction_functions ################## classes - -class CanvasObject(object): - """ - A base class for wrapping canvas objects. - """ - - def __init__(self,canvas_obj=None): - - super(object, self).__init__() - - self.canvas_obj = canvas_obj - - - - -class Document(CanvasObject): - """ - A base class which handles common pieces of interface for things like Pages and Assignments - - This type is abstract. Assignments and Pages both derive from this. - - At least two files are required in the folder for a Document: - - 1. `meta.json` - 2. `source.md` - - You may have additional files in the folder for a Document, such as images and files to include in the content on Canvas. This library will automatically upload those for you! - """ - - def __init__(self,folder,course=None): - """ - Construct a Document. - Reads the meta.json file and source.md files - from the specified folder. - """ - - super(Document,self).__init__(folder) - import json, os - from os.path import join - - self.folder = folder - - # try to open, and see if the meta and source files exist. - # if not, raise - self.metaname = path.join(folder,'meta.json') - with open(self.metaname,'r',encoding='utf-8') as f: - self.metadata = json.load(f) - - self.sourcename = path.join(folder,'source.md') - - - # variables populated from the metadata. should these even exist? IDK - self.name = None - self.style_path = None - self.replacements_path = None - - # populate the above variables from the meta.json file - self._set_from_metadata() - - - # these internally-used variables are used to carry state between functions - self._local_images = None - self._local_files = None - self._translated_html = None - - - def _set_from_metadata(self): - """ - this function is called during `__init__`. - """ - - self.name = self.metadata['name'] - - if 'modules' in self.metadata: - self.modules = self.metadata['modules'] - else: - self.modules = [] - - if 'indent' in self.metadata: - self.indent = self.metadata['indent'] - else: - self.indent = 0 - - if 'style' in self.metadata: - self.style_path = find_in_containing_directory_path(self.metadata['style']) - else: - self.style_path = get_default_style_name() # could be None if doesn't exist - if self.style_path: - self.style_path = find_in_containing_directory_path(self.style_path) - - if 'replacements' in self.metadata: - self.replacements_path = find_in_containing_directory_path(self.metadata['replacements']) - else: - self.replacements_path = get_default_replacements_name() # could be None if doesn't exist - if self.replacements_path: - self.replacements_path = find_in_containing_directory_path(self.replacements_path) - - - - - def translate_to_html(self,course): - """ - populates the internal variables with the results of translating from markdown to html. - - This step requires the `course` since this library allows for referencing of content already on canvas (or to be later published on Canvas) - - The main result of translation is held in self._translated_html. The local content (on YOUR computer, NOT Canvas) is parsed out and held in `self._local_images` and `self._local_files`. - - * This function does NOT make content appear on Canvas. - * It DOES leave behind a temporary file: `{folder}/styled_source.md`. Be sure to add `*/styled_source.md` to your .gitignore for your course! - """ - from os.path import join - - if self.style_path: - outname = join(self.folder,"styled_source.md") - apply_style_markdown(self.sourcename, self.style_path, outname) - - translated_html_without_hf = markdown2html(outname,course, self.replacements_path) - - self._translated_html = apply_style_html(translated_html_without_hf, self.style_path, outname) - else: - self._translated_html = markdown2html(self.sourcename,course, self.replacements_path) - - - self._local_images = find_local_images(self._translated_html) - self._local_files = find_local_files(self._translated_html) - - - - - - def publish_linked_content_and_adjust_html(self,course,overwrite=False): - """ - this function should be called *after* `translate_to_html`, since it requires the internal variables that other function populates - - the result of this function is written to `folder/result.html` - - * This function does NOT make content appear on Canvas. - * It DOES leave behind a temporary file: `{folder}/result.html`. Be sure to add `*/result.html` to your .gitignore for your course! - """ - - # first, publish the local images. - for im in self._local_images.values(): - im.publish(course,'images', overwrite=overwrite) - - for file in self._local_files.values(): - file.publish(course,'automatically_uploaded_files', overwrite=overwrite) - - - # then, deal with the urls - self._translated_html = adjust_html_for_images(self._translated_html, self._local_images, course.id) - self._translated_html = adjust_html_for_files(self._translated_html, self._local_files, course.id) - - save_location = path.join(self.folder,'result.html') - with open(save_location,'w',encoding='utf-8') as result: - result.write(self._translated_html) - - - - - - - - - - def _construct_dict_of_props(self): - """ - construct a dictionary of properties, such that it can be used to `edit` a canvas object. - """ - d = {} - return d - - - def ensure_in_modules(self, course): - """ - makes sure this item is listed in the Module on Canvas. If it's not, it's added to the bottom. There's not currently any way to control order. - - If the item doesn't already exist, this function will raise. Be sure to actually publish the content first. - """ - - if not self.canvas_obj: - raise DoesntExist(f"trying to make sure an object is in its modules, but this item ({self.name}) doesn't exist on canvas yet. publish it first.") - - for module_name in self.modules: - module = create_or_get_module(module_name, course) - - if not self.is_in_module(module_name, course): - - if self.metadata['type'] == 'page': - content_id = self.canvas_obj.page_id - elif self.metadata['type'] == 'assignment': - content_id = self.canvas_obj.id - - - module.create_module_item(module_item={'type':self.metadata['type'], 'content_id':content_id, 'indent':self.indent}) - - - def is_in_module(self, module_name, course): - """ - checks whether this content is an item in the listed module, where `module_name` is a string. It's case sensitive and exact. - - passthrough raise if the module doesn't exist - """ - - module = get_module(module_name,course) - - for item in module.get_module_items(): - - if item.type=='Page': - if self.metadata['type']=='page': - - if course.get_page(item.page_url).title == self.name: - return True - - else: - continue - - - if item.type=='Assignment': - if self.metadata['type']=='assignment': - - if course.get_assignment(assignment=item.content_id).name == self.name: - return True - else: - continue - - return False - - -class Page(Document): - """ - a Page is an abstraction around content for plain old canvas pages, which facilitates uploading to Canvas. - - folder -- a string, the name of the folder we're going to read data from. - """ - def __init__(self, folder): - super(Page, self).__init__(folder) - - - def _set_from_metadata(self): - super(Page,self)._set_from_metadata() - - - def publish(self, course, overwrite=False): - """ - if `overwrite` is False, then if an assignment is found with the same name already, the function will decline to make any edits. - - That is, if overwrite==False, then this function will only succeed if there's no existing assignment of the same name. - - This base-class function will handle things like the html, images, etc. - - Other derived-class `publish` functions will handle things like due-dates for assignments, etc. - """ - - logging.info(f'starting translate and upload process for Page `{self.name}`') - - - try: - page = create_or_get_page(self.name, course, even_if_exists=overwrite) - except AlreadyExists as e: - if not overwrite: - raise e - - self.canvas_obj = page - - self.translate_to_html(course) - - self.publish_linked_content_and_adjust_html(course, overwrite=overwrite) - - d = self._construct_dict_of_props() - page.edit(wiki_page=d) - - self.ensure_in_modules(course) - - logging.info(f'done uploading {self.name}') - - - - def _construct_dict_of_props(self): - - d = super(Page,self)._construct_dict_of_props() - - d['body'] = self._translated_html - d['title'] = self.name - - return d - - def __str__(self): - result = f"Page({self.folder})" - return result - - -class Assignment(Document): - """docstring for Assignment""" - def __init__(self, folder): - super(Assignment, self).__init__(folder) - - # self._set_from_metadata() # <-- this is called from the base __init__ - - def __str__(self): - result = f"Assignment({self.folder})" - return result - - def _get_list_of_canvas_properties_(self): - doc_url = "https://canvas.instructure.com/doc/api/assignments.html#method.assignments_api.update" - thing = "Request Parameters:" - raise NotImplementedError(f"this function is not implemented, but is intended to provide a programmatic way to determine the validity of a property name. see `{doc_url}`") - - - def _set_from_metadata(self): - super(Assignment,self)._set_from_metadata() - - default_to_none = lambda propname: self.metadata[propname] if propname in self.metadata else None - - self.allowed_extensions = default_to_none('allowed_extensions') - - self.points_possible = default_to_none('points_possible') - - self.unlock_at = default_to_none('unlock_at') - self.lock_at = default_to_none('lock_at') - self.due_at = default_to_none('due_at') - - self.published = default_to_none('published') - - self.submission_types = default_to_none('submission_types') - - self.external_tool_tag_attributes = default_to_none('external_tool_tag_attributes') - self.omit_from_final_grade = default_to_none('omit_from_final_grade') - - self.grading_type = default_to_none('grading_type') - self.assignment_group_name = default_to_none('assignment_group_name') - - self._validate_props() - - def _validate_props(self): - - - if self.allowed_extensions is not None and self.submission_types is None: - print('warning: using allowed_extensions but submission_types is not specified in the meta.json file for this assignment. you should probably use / include ["online_upload"]. valid submission_types can be found at https://canvas.instructure.com/doc/api/assignments.html#method.assignments_api.update') - - if self.allowed_extensions is not None and not isinstance(self.allowed_extensions,list): - print('warning: allowed_extensions must be a list') - - if self.submission_types is not None and not isinstance(self.submission_types,list): - print('warning: submission_types must be a list. Valid submission_types can be found at https://canvas.instructure.com/doc/api/assignments.html#method.assignments_api.update') - - if self.allowed_extensions is not None and isinstance(self.submission_types,list): - if 'online_upload' not in self.submission_types: - print('warning: using allowed_extensions, but "online_upload" is not in your list of submission_types. you should probably add it.') - - def _construct_dict_of_props(self): - - d = super(Assignment,self)._construct_dict_of_props() - d['name'] = self.name - d['description'] = self._translated_html - - if not self.allowed_extensions is None: - d['allowed_extensions'] = self.allowed_extensions - - if not self.points_possible is None: - d['points_possible'] = self.points_possible - - if not self.unlock_at is None: - d['unlock_at'] = self.unlock_at - if not self.due_at is None: - d['due_at'] = self.due_at - if not self.lock_at is None: - d['lock_at'] = self.lock_at - - if not self.published is None: - d['published'] = self.published - - if not self.submission_types is None: - d['submission_types'] = self.submission_types - - if not self.external_tool_tag_attributes is None: - d['external_tool_tag_attributes'] = self.external_tool_tag_attributes - - if not self.omit_from_final_grade is None: - d['omit_from_final_grade'] = self.omit_from_final_grade - - if not self.grading_type is None: - d['grading_type'] = self.grading_type - - return d - - - - - def ensure_in_assignment_groups(self, course, create_if_necessary=False): - - if self.assignment_group_name is None: - logging.info(f'when putting assignment {self.name} into group, taking no action because no assignment group specified') - return - - assignment_group_id = get_assignment_group_id(self.assignment_group_name, course, create_if_necessary) # todo: change this to try/except, instead of passing `create_if_necessary` to the get function. getting gets. it shouldn't create. - self.canvas_obj.edit(assignment={'assignment_group_id':assignment_group_id}) - - - - def publish(self, course, overwrite=False, create_modules_if_necessary=False, create_assignment_group_if_necessary=False): - """ - if `overwrite` is False, then if an assignment is found with the same name already, the function will decline to make any edits. - - That is, if overwrite==False, then this function will only succeed if there's no existing assignment of the same name. - """ - - logging.info(f'starting translate and upload process for Assignment `{self.name}`') - - - # need a remote object to work with - assignment = None - try: - assignment = create_or_get_assignment(self.name, course, overwrite) - except AlreadyExists as e: - if not overwrite: - raise e - - self.canvas_obj = assignment - - self.translate_to_html(course) - - self.publish_linked_content_and_adjust_html(course, overwrite=overwrite) - - # now that we have the assignment, we'll update its content. - - new_props=self._construct_dict_of_props() - - # for example, - # ass[0].edit(assignment={'lock_at':datetime.datetime(2021, 8, 17, 4, 59, 59),'due_at':datetime.datetime(2021, 8, 17, 4, 59, 59)}) - # we construct the dict of values in the _construct_dict_of_props() function. - - assignment.edit(assignment=new_props) - - self.ensure_in_modules(course) - self.ensure_in_assignment_groups(course,create_if_necessary=create_assignment_group_if_necessary) - - logging.info(f'done uploading {self.name} to Canvas') - - return True - - - - - - - -class Image(CanvasObject): - """ - A wrapper class for images on Canvas - """ - - - def __init__(self, filename, alttext = ''): - super(Image, self).__init__() - - self.givenpath = filename - self.filename = filename - # self.name = path.basename(filename) - # self.folder = path.abspath(filename) - - self.name = path.split(filename)[1] - self.folder = path.split(filename)[0] - - self.alttext = alttext - - - #

- # hauser_menagerie.jpg - #

- - def publish(self, course, dest, overwrite=False, raise_if_already_uploaded = False): - """ - - - see also https://canvas.instructure.com/doc/api/file.file_uploads.html - """ - - if overwrite: - on_duplicate = 'overwrite' - else: - on_duplicate = 'rename' - - - # this still needs to be adjusted to capture the Canvas image, in case it exists - if overwrite: - logging.debug('uploading {} to {}'.format(self.givenpath, dest)) - success_code, json_response = course.upload(self.givenpath, parent_folder_path=dest,on_duplicate=on_duplicate) - logging.debug('success_code from uploading was {}'.format(success_code)) - logging.debug('json response from uploading was {}'.format(json_response)) - - if not success_code: - print(f'failed to upload... {self.givenpath}') - - self.canvas_obj = course.get_file(json_response['id']) - return self.canvas_obj - - else: - if is_file_already_uploaded(self.givenpath,course): - if raise_if_already_uploaded: - raise AlreadyExists(f'image {self.name} already exists in course {course.name}, but you don\'t want to overwrite.') - else: - img_on_canvas = find_file_in_course(self.givenpath,course) - else: - # get the remote image - print(f'file not already uploaded, uploading {self.name}') - - success_code, json_response = course.upload(self.givenpath, parent_folder_path=dest,on_duplicate=on_duplicate) - img_on_canvas = course.get_file(json_response['id']) - if not success_code: - print(f'failed to upload... {self.givenpath}') - - - self.canvas_obj = img_on_canvas - - return img_on_canvas - - def make_src_url(self,courseid): - """ - constructs a string which can be used to embed the image in a Canvas page. - - sadly, the JSON back from Canvas doesn't just produce this for us. lame. - - """ - import canvasapi - im = self.canvas_obj - assert(isinstance(self.canvas_obj, canvasapi.file.File)) - - n = im.url.find('/files') - - url = im.url[:n]+'/courses/'+str(courseid)+'/files/'+str(im.id)+'/preview' - - return url - - def make_api_endpoint_url(self,courseid): - import canvasapi - im = self.canvas_obj - assert(isinstance(self.canvas_obj, canvasapi.file.File)) - - n = im.url.find('/files') - - url = im.url[:n] + '/api/v1/courses/' + str(courseid) + '/files/' + str(im.id) - return url - # data-api-endpoint="https://uws-td.instructure.com/api/v1/courses/3099/files/219835" - - - def __str__(self): - result = "\n" - result = result + f'givenpath: {self.givenpath}\n' - result = result + f'name: {self.name}\n' - result = result + f'folder: {self.folder}\n' - result = result + f'alttext: {self.alttext}\n' - result = result + f'canvas_obj: {self.canvas_obj}\n' - url = self.make_src_url('fakecoursenumber') - result = result + f'constructed canvas url: {url}\n' - - return result+'\n' - - def __repr__(self): - return str(self) - - - - -class BareFile(CanvasObject): - """ - A wrapper class for bare, unwrapped files on Canvas, for link to inline. - """ - - - def __init__(self, filename): - super(BareFile, self).__init__() - - self.givenpath = filename - self.filename = filename - self.name = path.basename(filename) - self.folder = path.abspath(filename) - - # self.name = path.split(filename)[1] - # self.folder = path.split(filename)[0] - - - - def publish(self, course, dest, overwrite=False, raise_if_already_uploaded = False): - """ - - - see also https://canvas.instructure.com/doc/api/file.file_uploads.html - """ - - if overwrite: - on_duplicate = 'overwrite' - else: - on_duplicate = 'rename' - - - - # this still needs to be adjusted to capture the Canvas file, in case it exists - if overwrite: - success_code, json_response = course.upload(self.givenpath, parent_folder_path=dest,on_duplicate=on_duplicate) - if not success_code: - print(f'failed to upload... {self.givenpath}') - else: - print(f'overwrote {self.name}') - - self.canvas_obj = course.get_file(json_response['id']) - return self.canvas_obj - - else: - if is_file_already_uploaded(self.givenpath,course): - if raise_if_already_uploaded: - raise AlreadyExists(f'file {self.name} already exists in course {course.name}, but you don\'t want to overwrite.') - else: - file_on_canvas = find_file_in_course(self.givenpath,course) - else: - # get the remote file - print(f'file not already uploaded, uploading {self.name}') - - success_code, json_response = course.upload(self.givenpath, parent_folder_path=dest,on_duplicate=on_duplicate) - file_on_canvas = course.get_file(json_response['id']) - if not success_code: - print(f'failed to upload... {self.givenpath}') - - - self.canvas_obj = file_on_canvas - - return file_on_canvas - - - - - def make_href_url(self,courseid): - """ - constructs a string which can be used to reference the file in a Canvas page. - - sadly, the JSON back from Canvas doesn't just produce this for us. lame. - - """ - import canvasapi - file = self.canvas_obj - assert(isinstance(self.canvas_obj, canvasapi.file.File)) - - n = file.url.find('/files') - - url = file.url[:n]+'/courses/'+str(courseid)+'/files/'+str(file.id)+'/download?wrap=1' - - return url - - - def make_api_endpoint_url(self,courseid): - import canvasapi - file = self.canvas_obj - assert(isinstance(self.canvas_obj, canvasapi.file.File)) - - n = file.url.find('/files') - - url = file.url[:n] + '/api/v1/courses/' + str(courseid) + '/files/' + str(file.id) - return url - # data-api-endpoint="https://uws-td.instructure.com/api/v1/courses/3099/files/219835" - - - def __str__(self): - result = "\n" - result = result + f'givenpath: {self.givenpath}\n' - result = result + f'name: {self.name}\n' - result = result + f'folder: {self.folder}\n' - result = result + f'alttext: {self.alttext}\n' - result = result + f'canvas_obj: {self.canvas_obj}\n' - url = self.make_href_url('fakecoursenumber') - result = result + f'constructed canvas url: {url}\n' - - return result+'\n' - - def __repr__(self): - return str(self) - - - - - - - - - - - - - - - - - - - - - -class Link(CanvasObject): - """ - a containerization of url's, for uploading to Canvas modules - """ - def __init__(self, folder): - super(Link, self).__init__() - self.folder = folder - - import json, os - from os.path import join - - self.metaname = path.join(folder,'meta.json') - with open(self.metaname,'r',encoding='utf-8') as f: - self.metadata = json.load(f) - - if 'indent' in self.metadata: - self.indent = self.metadata['indent'] - else: - self.indent = 0 - - def __str__(self): - result = f"Link({self.metadata['external_url']})" - return result - - def __repr__(self): - return str(self) - - - def publish(self, course, overwrite=False): - - for m in self.metadata['modules']: - if link_on_canvas:= self.is_in_module(course, m): - if not overwrite: - n = self.metadata['external_url'] - raise AlreadyExists(f'trying to upload {self}, but is already on Canvas in module {m}') - else: - link_on_canvas.edit(module_item={'external_url':self.metadata['external_url'],'title':self.metadata['name'], 'new_tab':bool(self.metadata['new_tab'])}) - - else: - mod = create_or_get_module(m, course) - mod.create_module_item(module_item={'type':'ExternalUrl','external_url':self.metadata['external_url'],'title':self.metadata['name'], 'new_tab':bool(self.metadata['new_tab']), 'indent':self.indent}) - - - def is_already_uploaded(self, course): - for m in self.metadata['modules']: - if not self.is_in_module(course, m): - return False - - return True - - - - def is_in_module(self, course, module_name): - try: - module = get_module(module_name,course) - except DoesntExist as e: - return None - - - for item in module.get_module_items(): - - if item.type=='ExternalUrl' and item.external_url==self.metadata['external_url']: - return item - else: - continue - - return None - - - - -class File(CanvasObject): - """ - a containerization of arbitrary files, for uploading to Canvas - """ - def __init__(self, folder): - super(File, self).__init__(folder) - - import json, os - from os.path import join - - self.folder = folder - - self.metaname = path.join(folder,'meta.json') - with open(self.metaname,'r',encoding='utf-8') as f: - self.metadata = json.load(f) - - try: - self.title = self.metadata['title'] - except: - self.title = self.metadata['filename'] - - - if 'indent' in self.metadata: - self.indent = self.metadata['indent'] - else: - self.indent = 0 - - - def __str__(self): - result = f"File({self.metadata})" - return result - - def __repr__(self): - return str(self) - - - def _upload_(self, course): - pass - - - def publish(self, course, overwrite=False): - """ - publishes a file to Canvas in a particular folder - """ - - on_duplicate='overwrite' - if (file_on_canvas:= self.is_already_uploaded(course)) and not overwrite: - # on_duplicate='rename' - n = self.metadata['filename'] - # content_id = file_on_canvas.id - - raise AlreadyExists(f'The file {n} is already on Canvas and `not overwrite`.') - else: - root = get_root_folder(course) - - d = self.metadata['destination'] - d = d.split('/') - - curr_dir = root - for subd in d: - try: - curr_dir = get_subfolder_named(curr_dir, subd) - except DoesntExist as e: - curr_dir = curr_dir.create_folder(subd) - - filepath_to_upload = path.join(self.folder,self.metadata['filename']) - reply = curr_dir.upload(file=filepath_to_upload,on_duplicate=on_duplicate) - - if not reply[0]: - raise RuntimeError(f'something went wrong uploading {filepath_to_upload}') - - file_on_canvas = reply[1] - content_id = file_on_canvas['id'] - - - # now to make sure it's in the right modules - for module_name in self.metadata['modules']: - module = create_or_get_module(module_name, course) - - items = module.get_module_items() - is_in = False - for item in items: - if item.type=='File' and item.content_id==content_id: - is_in = True - break - - if not is_in: - module.create_module_item(module_item={'type':'File', 'content_id':content_id, 'title':self.title, 'indent':self.indent}) - # if the title doesn't match, update it - elif item.title != self.title: - item.edit(module_item={'type':'File', 'content_id':content_id, 'title':self.title},module=module) - - - def is_in_module(self, course, module_name): - file_on_canvas = self.is_already_uploaded(course) - - if not file_on_canvas: - return False - - module = get_module(module_name,course) - - for item in module.get_module_items(): - - if item.type=='File' and item.content_id==file_on_canvas.id: - return True - else: - continue - - return False - - - def is_already_uploaded(self,course, require_same_path=True): - files = course.get_files() - - for f in files: - if f.filename == self.metadata['filename']: - - if not require_same_path: - return f - else: - containing_folder = course.get_folder(f.folder_id) - if containing_folder.full_name.startswith('course files') and containing_folder.full_name.endswith(self.metadata['destination']): - return f - - - return None - - - - - - -def page2markdown(destination, page, even_if_exists=False): - """ - takes a Page from Canvas, and saves it to a folder inside `destination` - into a markdown2canvas compatible format. - - the folder is automatically named, at your own peril. - """ - - import os - - assert(isinstance(page,canvasapi.page.Page)) - - if (path.exists(destination)) and not path.isdir(destination): - raise AlreadyExists(f'you want to save a page into directory {destination}, but it exists and is not a directory') - - - - - r = page.show_latest_revision() - body = r.body # this is the content of the page, in html. - title = r.title - - dir_name = title.replace(":","").replace(" ","_") - destdir = path.join(destination,dir_name) - if (not even_if_exists) and path.exists(destdir): - raise AlreadyExists(f'trying to save page {title} to folder {destdir}, but that already exists. If you want to force, use `even_if_exists=True`.') - - if not path.exists(destdir): - os.makedirs(destdir) - - logging.info(f'downloading page {title}, saving to folder {destdir}') - - with open(path.join(destdir,'source.md'),'w',encoding='utf-8') as file: - file.write(body) - - - d = {} - - d['name'] = title - d['type'] = 'page' - with open(path.join(destdir,'meta.json'),'w',encoding='utf-8') as file: - import json - json.dump(d, file) - - - - -def download_pages(destination, course, even_if_exists=False, name_filter=None): - """ - downloads the regular pages from a course, saving them - into a markdown2canvas compatible format. that is, as - a folder with markdown source and json metadata. - """ - - if name_filter is None: - name_filter = lambda x: True - - logging.info(f'downloading all pages from course {course.name}, saving to folder {destination}') - pages = course.get_pages() - for p in pages: - if name_filter(p.show_latest_revision().title): - page2markdown(destination,p,even_if_exists) - - -def assignment2markdown(destination, assignment, even_if_exists=False): - """ - takes a Page from Canvas, and saves it to a folder inside `destination` - into a markdown2canvas compatible format. - - the folder is automatically named, at your own peril. - """ - - import os - - assert(isinstance(assignment,canvasapi.assignment.Assignment)) - - if (path.exists(destination)) and not path.isdir(destination): - raise AlreadyExists(f'you want to save a page into directory {destination}, but it exists and is not a directory') - - - - - body = assignment.description # this is the content of the page, in html. - title = assignment.name - - destdir = path.join(destination,title) - if (not even_if_exists) and path.exists(destdir): - raise AlreadyExists(f'trying to save page {title} to folder {destdir}, but that already exists. If you want to force, use `even_if_exists=True`.') - - if not path.exists(destdir): - os.makedirs(destdir) - - logging.info(f'downloading page {title}, saving to folder {destdir}') - - with open(path.join(destdir,'source.md'),'w',encoding='utf-8') as file: - file.write(body) - - - d = {} - - d['name'] = title - d['type'] = 'assignment' - with open(path.join(destdir,'meta.json'),'w',encoding='utf-8') as file: - import json - json.dump(d, file) - -def download_assignments(destination, course, even_if_exists=False, name_filter=None): - """ - downloads the regular pages from a course, saving them - into a markdown2canvas compatible format. that is, as - a folder with markdown source and json metadata. - """ - - if name_filter is None: - name_filter = lambda x: True - - logging.info(f'downloading all pages from course {course.name}, saving to folder {destination}') - assignments = course.get_assignments() - for a in assignments: - if name_filter(a.name): - assignment2markdown(destination,a,even_if_exists) - +from .canvas_objects import CanvasObject, Document, Page, Assignment, Image, File, BareFile, Link +import markdown2canvas.canvas2markdown as canvas2markdown -import markdown2canvas.tool +import markdown2canvas.tool as tool diff --git a/markdown2canvas/canvas2markdown.py b/markdown2canvas/canvas2markdown.py new file mode 100644 index 0000000..d1c53fb --- /dev/null +++ b/markdown2canvas/canvas2markdown.py @@ -0,0 +1,171 @@ +''' +Functions for grabbing non-containerized content from Canvas and saving it to disk. Useful for making a markdown2canvas repo from an existing course. + +The two main functions you should use are `download_pages` and `download_assignments`. + +The resulting folder will have name equal to the name of the content on Canvas, for better or for worse. + +I can see some situations where folder names are invalid -- feel free to improve this functionality. PR's welcome. + +''' + +__all__ = [ + 'download_pages','download_assignments' + 'page2markdown','assignment2markdown'] + + + +import canvasapi + +import os.path as path + +import logging +logger = logging.getLogger() + + + + +def download_pages(destination, course, even_if_exists=False, name_filter=None): + """ + downloads the regular pages from a course, saving them + into a markdown2canvas compatible format. that is, as + a folder with markdown source and json metadata. + + You can provide a predicate `name_filter` to filter on the name of the content! The function should return True/False. + + The flag `even_if_exists` is to overwrite the local content. If `even_if_exists` is True, the remote content will be written to disk EVEN IF IT ALREADY EXISTED LOCALLY. Thus, this may involve data loss. Use version control. + """ + + if name_filter is None: + name_filter = lambda x: True + + logger.info(f'downloading all pages from course {course.name}, saving to folder {destination}') + pages = course.get_pages() + for p in pages: + if name_filter(p.show_latest_revision().title): + page2markdown(destination,p,even_if_exists) + + +def download_assignments(destination, course, even_if_exists=False, name_filter=None): + """ + downloads the assignments from a course, saving them + into a markdown2canvas compatible format. that is, as + a folder with markdown source and json metadata. + + `destination` is the path you want to write the content to. This function will make sub-folders of `destination`. + + You can provide a predicate `name_filter` to filter on the name of the content! The function should return True/False. + + The flag `even_if_exists` is to overwrite the local content. If `even_if_exists` is True, the remote content will be written to disk EVEN IF IT ALREADY EXISTED LOCALLY. Thus, this may involve data loss. Use version control. + """ + + if name_filter is None: + name_filter = lambda x: True + + logger.info(f'downloading all pages from course {course.name}, saving to folder {destination}') + assignments = course.get_assignments() + for a in assignments: + if name_filter(a.name): + assignment2markdown(destination,a,even_if_exists) + + + +def page2markdown(destination, page, even_if_exists=False): + """ + takes a Page from Canvas, and saves it to a folder inside `destination` + into a markdown2canvas compatible format. + + the folder is automatically named, at your own peril. Colons are removed, and spaces are replaced by underscores. + + The flag `even_if_exists` is to overwrite the local content. If `even_if_exists` is True, the remote content will be written to disk EVEN IF IT ALREADY EXISTED LOCALLY. Thus, this may involve data loss. Use version control. + + """ + + import os + + assert(isinstance(page,canvasapi.page.Page)) + + if (path.exists(destination)) and not path.isdir(destination): + raise AlreadyExists(f'you want to save a page into directory {destination}, but it exists and is not a directory') + + + + + r = page.show_latest_revision() + body = r.body # this is the content of the page, in html. + title = r.title + + dir_name = title.replace(":","").replace(" ","_") + destdir = path.join(destination,dir_name) + if (not even_if_exists) and path.exists(destdir): + raise AlreadyExists(f'trying to save page {title} to folder {destdir}, but that already exists. If you want to force, use `even_if_exists=True`.') + + if not path.exists(destdir): + os.makedirs(destdir) + + logger.info(f'downloading page {title}, saving to folder {destdir}') + + with open(path.join(destdir,'source.md'),'w',encoding='utf-8') as file: + file.write(body) + + + d = {} + + d['name'] = title + d['type'] = 'page' + with open(path.join(destdir,'meta.json'),'w',encoding='utf-8') as file: + import json + json.dump(d, file) + + + + + + +def assignment2markdown(destination, assignment, even_if_exists=False): + """ + takes a Page from Canvas, and saves it to a folder inside `destination` + into a markdown2canvas compatible format. + + the folder is automatically named, at your own peril. Colons are removed, and spaces are replaced by underscores. + + The flag `even_if_exists` is to overwrite the local content. If `even_if_exists` is True, the remote content will be written to disk EVEN IF IT ALREADY EXISTED LOCALLY. Thus, this may involve data loss. Use version control. + """ + + import os + + assert(isinstance(assignment,canvasapi.assignment.Assignment)) + + if (path.exists(destination)) and not path.isdir(destination): + raise AlreadyExists(f'you want to save a page into directory {destination}, but it exists and is not a directory') + + + + + body = assignment.description # this is the content of the page, in html. + title = assignment.name + + destdir = path.join(destination,title) + if (not even_if_exists) and path.exists(destdir): + raise AlreadyExists(f'trying to save page {title} to folder {destdir}, but that already exists. If you want to force, use `even_if_exists=True`.') + + if not path.exists(destdir): + os.makedirs(destdir) + + logger.info(f'downloading page {title}, saving to folder {destdir}') + + with open(path.join(destdir,'source.md'),'w',encoding='utf-8') as file: + file.write(body) + + + d = {} + + d['name'] = title + d['type'] = 'assignment' + with open(path.join(destdir,'meta.json'),'w',encoding='utf-8') as file: + import json + json.dump(d, file) + + + + diff --git a/markdown2canvas/canvas_objects.py b/markdown2canvas/canvas_objects.py new file mode 100644 index 0000000..38ed0b9 --- /dev/null +++ b/markdown2canvas/canvas_objects.py @@ -0,0 +1,979 @@ +""" +Main content types for markdown2canvas + +Page and Assignment both require meta.json and source.md + +Image, File, BareFile each require meta.json and the file they containerize. Importantly, the file does NOT need to be contained in the folder!!! I'm trying to make your life easy. + +Link requires meta.json and that's it +""" + + +__all__ = [ + 'Page', + 'Assignment', + 'Image', + 'BareFile', + 'Link', + 'File' +] + +import os.path as path +import os + +import canvasapi +from markdown2canvas.logging import * +from markdown2canvas.setup_functions import * +from markdown2canvas.translation_functions import * +from markdown2canvas.course_interaction_functions import * +from markdown2canvas.exception import AlreadyExists, SetupError, DoesntExist + + + + + +def find_local_images(html): + """ + returns a dictionary of local url's to markdown2canvas.Images + + works by assuming that local url's don't start with http or https, as those should be online. + """ + from bs4 import BeautifulSoup + + soup = BeautifulSoup(html,features="lxml") + + local_images = {} + + all_imgs = soup.findAll("img") + + if all_imgs: + for img in all_imgs: + src = img["src"] + if src[:7] not in ['https:/','http://']: + local_images[src] = Image(path.abspath(src)) + + return local_images + + + +def find_local_files(html): + """ + constructs a dictionary of markdown2canvas.BareFiles, so that they can later be replaced with a url to a canvas thing after upload + """ + from bs4 import BeautifulSoup + + soup = BeautifulSoup(html,features="lxml") + + local_files = {} + + all_links = soup.findAll("a") + + if all_links: + for file in all_links: + href = file["href"] + if path.exists(path.abspath(href)): + local_files[href] = BareFile(path.abspath(href)) + + return local_files + + +class CanvasObject(object): + """ + A base class for wrapping canvas objects. + """ + + def __init__(self,canvas_obj=None): + + super(object, self).__init__() + + self.canvas_obj = canvas_obj + + + + +class Document(CanvasObject): + """ + A base class which handles common pieces of interface for things like Pages and Assignments + + This type is abstract. Assignments and Pages both derive from this. + + At least two files are required in the folder for a Document: + + 1. `meta.json` + 2. `source.md` + + You may have additional files in the folder for a Document, such as images and files to include in the content on Canvas. This library will automatically upload those for you! + """ + + def __init__(self,folder,course=None): + """ + Construct a Document. + Reads the meta.json file and source.md files + from the specified folder. + """ + + super(Document,self).__init__(folder) + import json, os + from os.path import join + + self.folder = folder + + # try to open, and see if the meta and source files exist. + # if not, raise + self.metaname = path.join(folder,'meta.json') + with open(self.metaname,'r',encoding='utf-8') as f: + self.metadata = json.load(f) + + self.sourcename = path.join(folder,'source.md') + + + # variables populated from the metadata. should these even exist? IDK + self.name = None + self.style_path = None + self.replacements_path = None + self.markdown_extensions = None + + # populate the above variables from the meta.json file + self._set_from_metadata() + + + # these internally-used variables are used to carry state between functions + self._local_images = None + self._local_files = None + self._translated_html = None + + + def _set_from_metadata(self): + """ + this function is called during `__init__`. + """ + + self.name = self.metadata['name'] + + if 'modules' in self.metadata: + self.modules = self.metadata['modules'] + else: + self.modules = [] + + if 'indent' in self.metadata: + self.indent = self.metadata['indent'] + else: + self.indent = 0 + + if 'style' in self.metadata: + self.style_path = find_in_containing_directory_path(self.metadata['style']) + else: + self.style_path = get_default_style_name() # could be None if doesn't exist + if self.style_path: + self.style_path = find_in_containing_directory_path(self.style_path) + + if 'replacements' in self.metadata: + self.replacements_path = find_in_containing_directory_path(self.metadata['replacements']) + else: + self.replacements_path = get_default_replacements_name() # could be None if doesn't exist + if self.replacements_path: + self.replacements_path = find_in_containing_directory_path(self.replacements_path) + + self.markdown_extensions = get_default_markdown_extensions() + + def translate_to_html(self,course): + """ + populates the internal variables with the results of translating from markdown to html. + + This step requires the `course` since this library allows for referencing of content already on canvas (or to be later published on Canvas) + + The main result of translation is held in self._translated_html. The local content (on YOUR computer, NOT Canvas) is parsed out and held in `self._local_images` and `self._local_files`. + + * This function does NOT make content appear on Canvas. + * It DOES leave behind a temporary file: `{folder}/styled_source.md`. Be sure to add `*/styled_source.md` to your .gitignore for your course! + """ + from os.path import join + + if self.style_path: + outname = join(self.folder,"styled_source.md") + with_header_md = apply_style_markdown(self.sourcename, self.style_path, outname) + + outname = join(self.folder,"extra_styled_source.md") + apply_style_html(with_header_md, self.style_path, outname) + + self._translated_html = markdown2html(outname, course, self.replacements_path, self.markdown_extensions) + + + + # translated_html_without_hf = markdown2html(outname,course, self.replacements_path) + + # self._translated_html = apply_style_html(translated_html_without_hf, self.style_path, outname) + else: + self._translated_html = markdown2html(self.sourcename,course, self.replacements_path, self.markdown_extensions) + + + self._local_images = find_local_images(self._translated_html) + self._local_files = find_local_files(self._translated_html) + + + + + + def publish_linked_content_and_adjust_html(self,course,overwrite=False): + """ + this function should be called *after* `translate_to_html`, since it requires the internal variables that other function populates + + the result of this function is written to `folder/result.html` + + * This function does NOT make content appear on Canvas. + * It DOES leave behind a temporary file: `{folder}/result.html`. Be sure to add `*/result.html` to your .gitignore for your course! + """ + + # first, publish the local images. + for im in self._local_images.values(): + im.publish(course,'images', overwrite=overwrite) + + for file in self._local_files.values(): + file.publish(course,'automatically_uploaded_files', overwrite=overwrite) + + + # then, deal with the urls + self._translated_html = adjust_html_for_images(self._translated_html, self._local_images, course.id) + self._translated_html = adjust_html_for_files(self._translated_html, self._local_files, course.id) + + save_location = path.join(self.folder,'result.html') + with open(save_location,'w',encoding='utf-8') as result: + result.write(self._translated_html) + + + + + + + + + + def _construct_dict_of_props(self): + """ + construct an empty dictionary of properties, such that it can be used to `edit` a canvas object. + """ + d = {} + return d + + + def ensure_in_modules(self, course): + """ + makes sure this item is listed in the Modules listed (by name) in self.modules, on Canvas. If it's not, it's added to the bottom. There's not currently any way to control order. + + If the item doesn't already exist, this function will raise `DoesntExist`. Be sure to actually publish the content first. + """ + + if not self.canvas_obj: + raise DoesntExist(f"trying to make sure an object is in its modules, but this item ({self.name}) doesn't exist on canvas yet. publish it first.") + + for module_name in self.modules: + module = create_or_get_module(module_name, course) + + if not self.is_in_module(module_name, course): + + if self.metadata['type'] == 'page': + content_id = self.canvas_obj.page_id + elif self.metadata['type'] == 'assignment': + content_id = self.canvas_obj.id + + + module.create_module_item(module_item={'type':self.metadata['type'], 'content_id':content_id, 'indent':self.indent}) + + + def is_in_module(self, module_name, course): + """ + checks whether this content is an item in the listed module, where `module_name` is a string. It's case sensitive and exact. + + passthrough raise if the module doesn't exist + """ + + module = get_module(module_name,course) + + for item in module.get_module_items(): + + if item.type=='Page': + if self.metadata['type']=='page': + + if course.get_page(item.page_url).title == self.name: + return True + + else: + continue + + + if item.type=='Assignment': + if self.metadata['type']=='assignment': + + if course.get_assignment(assignment=item.content_id).name == self.name: + return True + else: + continue + + return False + + + + + + + + + + + + + + + + + + + + + + + + +class Page(Document): + """ + a Page is an abstraction around content for plain old canvas pages, which facilitates uploading to Canvas. + + folder -- a string, the name of the folder we're going to read data from. + """ + def __init__(self, folder): + super(Page, self).__init__(folder) + + + def _set_from_metadata(self): + super(Page,self)._set_from_metadata() + + + def publish(self, course, overwrite=False): + """ + if `overwrite` is False, then if an assignment is found with the same name already, the function will decline to make any edits. + + That is, if overwrite==False, then this function will only succeed if there's no existing assignment of the same name. + + This base-class function will handle things like the html, images, etc. + + Other derived-class `publish` functions will handle things like due-dates for assignments, etc. + """ + + logger.info(f'starting translate and upload process for Page `{self.name}`') + + + try: + page = create_or_get_page(self.name, course, even_if_exists=overwrite) + except AlreadyExists as e: + if not overwrite: + raise e + + self.canvas_obj = page + + self.translate_to_html(course) + + self.publish_linked_content_and_adjust_html(course, overwrite=overwrite) + + d = self._construct_dict_of_props() + page.edit(wiki_page=d) + + self.ensure_in_modules(course) + + logger.info(f'done uploading {self.name}') + + + + def _construct_dict_of_props(self): + + d = super(Page,self)._construct_dict_of_props() + + d['body'] = self._translated_html + d['title'] = self.name + + return d + + def __str__(self): + result = f"Page({self.folder})" + return result + + + +class Assignment(Document): + """docstring for Assignment""" + def __init__(self, folder): + super(Assignment, self).__init__(folder) + + # self._set_from_metadata() # <-- this is called from the base __init__ + + def __str__(self): + result = f"Assignment({self.folder})" + return result + + def _get_list_of_canvas_properties_(self): + doc_url = "https://canvas.instructure.com/doc/api/assignments.html#method.assignments_api.update" + thing = "Request Parameters:" + raise NotImplementedError(f"this function is not implemented, but is intended to provide a programmatic way to determine the validity of a property name. see `{doc_url}`") + + + def _set_from_metadata(self): + super(Assignment,self)._set_from_metadata() + + default_to_none = lambda propname: self.metadata[propname] if propname in self.metadata else None + + self.allowed_extensions = default_to_none('allowed_extensions') + + self.points_possible = default_to_none('points_possible') + + self.unlock_at = default_to_none('unlock_at') + self.lock_at = default_to_none('lock_at') + self.due_at = default_to_none('due_at') + + self.published = default_to_none('published') + + self.submission_types = default_to_none('submission_types') + + self.external_tool_tag_attributes = default_to_none('external_tool_tag_attributes') + self.omit_from_final_grade = default_to_none('omit_from_final_grade') + + self.grading_type = default_to_none('grading_type') + self.assignment_group_name = default_to_none('assignment_group_name') + + self._validate_props() + + def _validate_props(self): + + + if self.allowed_extensions is not None and self.submission_types is None: + print('warning: using allowed_extensions but submission_types is not specified in the meta.json file for this assignment. you should probably use / include ["online_upload"]. valid submission_types can be found at https://canvas.instructure.com/doc/api/assignments.html#method.assignments_api.update') + + if self.allowed_extensions is not None and not isinstance(self.allowed_extensions,list): + print('warning: allowed_extensions must be a list') + + if self.submission_types is not None and not isinstance(self.submission_types,list): + print('warning: submission_types must be a list. Valid submission_types can be found at https://canvas.instructure.com/doc/api/assignments.html#method.assignments_api.update') + + if self.allowed_extensions is not None and isinstance(self.submission_types,list): + if 'online_upload' not in self.submission_types: + print('warning: using allowed_extensions, but "online_upload" is not in your list of submission_types. you should probably add it.') + + def _construct_dict_of_props(self): + + d = super(Assignment,self)._construct_dict_of_props() + d['name'] = self.name + d['description'] = self._translated_html + + if not self.allowed_extensions is None: + d['allowed_extensions'] = self.allowed_extensions + + if not self.points_possible is None: + d['points_possible'] = self.points_possible + + if not self.unlock_at is None: + d['unlock_at'] = self.unlock_at + if not self.due_at is None: + d['due_at'] = self.due_at + if not self.lock_at is None: + d['lock_at'] = self.lock_at + + if not self.published is None: + d['published'] = self.published + + if not self.submission_types is None: + d['submission_types'] = self.submission_types + + if not self.external_tool_tag_attributes is None: + d['external_tool_tag_attributes'] = self.external_tool_tag_attributes + + if not self.omit_from_final_grade is None: + d['omit_from_final_grade'] = self.omit_from_final_grade + + if not self.grading_type is None: + d['grading_type'] = self.grading_type + + return d + + + + + def ensure_in_assignment_groups(self, course, create_if_necessary=False): + + if self.assignment_group_name is None: + logger.info(f'when putting assignment {self.name} into group, taking no action because no assignment group specified') + return + + assignment_group_id = get_assignment_group_id(self.assignment_group_name, course, create_if_necessary) # todo: change this to try/except, instead of passing `create_if_necessary` to the get function. getting gets. it shouldn't create. + self.canvas_obj.edit(assignment={'assignment_group_id':assignment_group_id}) + + + + def publish(self, course, overwrite=False, create_modules_if_necessary=False, create_assignment_group_if_necessary=False): + """ + if `overwrite` is False, then if an assignment is found with the same name already, the function will decline to make any edits. + + That is, if overwrite==False, then this function will only succeed if there's no existing assignment of the same name. + """ + + logger.info(f'starting translate and upload process for Assignment `{self.name}`') + + + # need a remote object to work with + assignment = None + try: + assignment = create_or_get_assignment(self.name, course, overwrite) + except AlreadyExists as e: + if not overwrite: + raise e + + self.canvas_obj = assignment + + self.translate_to_html(course) + + self.publish_linked_content_and_adjust_html(course, overwrite=overwrite) + + # now that we have the assignment, we'll update its content. + + new_props=self._construct_dict_of_props() + + # for example, + # ass[0].edit(assignment={'lock_at':datetime.datetime(2021, 8, 17, 4, 59, 59),'due_at':datetime.datetime(2021, 8, 17, 4, 59, 59)}) + # we construct the dict of values in the _construct_dict_of_props() function. + + assignment.edit(assignment=new_props) + + self.ensure_in_modules(course) + self.ensure_in_assignment_groups(course,create_if_necessary=create_assignment_group_if_necessary) + + logger.info(f'done uploading {self.name} to Canvas') + + return True + + + +class Image(CanvasObject): + """ + A wrapper class for images on Canvas + """ + + + def __init__(self, filename, alttext = ''): + super(Image, self).__init__() + + self.givenpath = filename + self.filename = filename + # self.name = path.basename(filename) + # self.folder = path.abspath(filename) + + self.name = path.split(filename)[1] + self.folder = path.split(filename)[0] + + self.alttext = alttext + + + #

+ # hauser_menagerie.jpg + #

+ + def publish(self, course, dest, overwrite=False, raise_if_already_uploaded = False): + """ + + + see also https://canvas.instructure.com/doc/api/file.file_uploads.html + """ + + if overwrite: + on_duplicate = 'overwrite' + else: + on_duplicate = 'rename' + + + # this still needs to be adjusted to capture the Canvas image, in case it exists + if overwrite: + logger.debug('uploading {} to {}'.format(self.givenpath, dest)) + success_code, json_response = course.upload(self.givenpath, parent_folder_path=dest,on_duplicate=on_duplicate) + logger.debug('success_code from uploading was {}'.format(success_code)) + logger.debug('json response from uploading was {}'.format(json_response)) + + if not success_code: + print(f'failed to upload... {self.givenpath}') + + self.canvas_obj = course.get_file(json_response['id']) + return self.canvas_obj + + else: + if is_file_already_uploaded(self.givenpath,course): + if raise_if_already_uploaded: + raise AlreadyExists(f'image {self.name} already exists in course {course.name}, but you don\'t want to overwrite.') + else: + img_on_canvas = find_file_in_course(self.givenpath,course) + else: + # get the remote image + print(f'file not already uploaded, uploading {self.name}') + + success_code, json_response = course.upload(self.givenpath, parent_folder_path=dest,on_duplicate=on_duplicate) + img_on_canvas = course.get_file(json_response['id']) + if not success_code: + print(f'failed to upload... {self.givenpath}') + + + self.canvas_obj = img_on_canvas + + return img_on_canvas + + def make_src_url(self,courseid): + """ + constructs a string which can be used to embed the image in a Canvas page. + + sadly, the JSON back from Canvas doesn't just produce this for us. lame. + + """ + import canvasapi + im = self.canvas_obj + assert(isinstance(self.canvas_obj, canvasapi.file.File)) + + n = im.url.find('/files') + + url = im.url[:n]+'/courses/'+str(courseid)+'/files/'+str(im.id)+'/preview' + + return url + + def make_api_endpoint_url(self,courseid): + import canvasapi + im = self.canvas_obj + assert(isinstance(self.canvas_obj, canvasapi.file.File)) + + n = im.url.find('/files') + + url = im.url[:n] + '/api/v1/courses/' + str(courseid) + '/files/' + str(im.id) + return url + # data-api-endpoint="https://uws-td.instructure.com/api/v1/courses/3099/files/219835" + + + def __str__(self): + result = "\n" + result = result + f'givenpath: {self.givenpath}\n' + result = result + f'name: {self.name}\n' + result = result + f'folder: {self.folder}\n' + result = result + f'alttext: {self.alttext}\n' + result = result + f'canvas_obj: {self.canvas_obj}\n' + url = self.make_src_url('fakecoursenumber') + result = result + f'constructed canvas url: {url}\n' + + return result+'\n' + + def __repr__(self): + return str(self) + + + + +class BareFile(CanvasObject): + """ + A wrapper class for bare, unwrapped files on Canvas, for link to inline. + """ + + + def __init__(self, filename): + super(BareFile, self).__init__() + + self.givenpath = filename + self.filename = filename + self.name = path.basename(filename) + self.folder = path.abspath(filename) + + # self.name = path.split(filename)[1] + # self.folder = path.split(filename)[0] + + + + def publish(self, course, dest, overwrite=False, raise_if_already_uploaded = False): + """ + + + see also https://canvas.instructure.com/doc/api/file.file_uploads.html + """ + + if overwrite: + on_duplicate = 'overwrite' + else: + on_duplicate = 'rename' + + + + # this still needs to be adjusted to capture the Canvas file, in case it exists + if overwrite: + success_code, json_response = course.upload(self.givenpath, parent_folder_path=dest,on_duplicate=on_duplicate) + if not success_code: + print(f'failed to upload... {self.givenpath}') + else: + print(f'overwrote {self.name}') + + self.canvas_obj = course.get_file(json_response['id']) + return self.canvas_obj + + else: + if is_file_already_uploaded(self.givenpath,course): + if raise_if_already_uploaded: + raise AlreadyExists(f'file {self.name} already exists in course {course.name}, but you don\'t want to overwrite.') + else: + file_on_canvas = find_file_in_course(self.givenpath,course) + else: + # get the remote file + print(f'file not already uploaded, uploading {self.name}') + + success_code, json_response = course.upload(self.givenpath, parent_folder_path=dest,on_duplicate=on_duplicate) + file_on_canvas = course.get_file(json_response['id']) + if not success_code: + print(f'failed to upload... {self.givenpath}') + + + self.canvas_obj = file_on_canvas + + return file_on_canvas + + + + + def make_href_url(self,courseid): + """ + constructs a string which can be used to reference the file in a Canvas page. + + sadly, the JSON back from Canvas doesn't just produce this for us. lame. + + """ + import canvasapi + file = self.canvas_obj + assert(isinstance(self.canvas_obj, canvasapi.file.File)) + + n = file.url.find('/files') + + url = file.url[:n]+'/courses/'+str(courseid)+'/files/'+str(file.id)+'/download?wrap=1' + + return url + + + def make_api_endpoint_url(self,courseid): + import canvasapi + file = self.canvas_obj + assert(isinstance(self.canvas_obj, canvasapi.file.File)) + + n = file.url.find('/files') + + url = file.url[:n] + '/api/v1/courses/' + str(courseid) + '/files/' + str(file.id) + return url + # data-api-endpoint="https://uws-td.instructure.com/api/v1/courses/3099/files/219835" + + + def __str__(self): + result = "\n" + result = result + f'givenpath: {self.givenpath}\n' + result = result + f'name: {self.name}\n' + result = result + f'folder: {self.folder}\n' + result = result + f'alttext: {self.alttext}\n' + result = result + f'canvas_obj: {self.canvas_obj}\n' + url = self.make_href_url('fakecoursenumber') + result = result + f'constructed canvas url: {url}\n' + + return result+'\n' + + def __repr__(self): + return str(self) + + + +class Link(CanvasObject): + """ + a containerization of url's, for uploading to Canvas modules + """ + def __init__(self, folder): + super(Link, self).__init__() + self.folder = folder + + import json, os + from os.path import join + + self.metaname = path.join(folder,'meta.json') + with open(self.metaname,'r',encoding='utf-8') as f: + self.metadata = json.load(f) + + if 'indent' in self.metadata: + self.indent = self.metadata['indent'] + else: + self.indent = 0 + + def __str__(self): + result = f"Link({self.metadata['external_url']})" + return result + + def __repr__(self): + return str(self) + + + def publish(self, course, overwrite=False): + + for m in self.metadata['modules']: + if link_on_canvas:= self.is_in_module(course, m): + if not overwrite: + n = self.metadata['external_url'] + raise AlreadyExists(f'trying to upload {self}, but is already on Canvas in module {m}') + else: + link_on_canvas.edit(module_item={'external_url':self.metadata['external_url'],'title':self.metadata['name'], 'new_tab':bool(self.metadata['new_tab'])}) + + else: + mod = create_or_get_module(m, course) + mod.create_module_item(module_item={'type':'ExternalUrl','external_url':self.metadata['external_url'],'title':self.metadata['name'], 'new_tab':bool(self.metadata['new_tab']), 'indent':self.indent}) + + + def is_already_uploaded(self, course): + for m in self.metadata['modules']: + if not self.is_in_module(course, m): + return False + + return True + + + + def is_in_module(self, course, module_name): + try: + module = get_module(module_name,course) + except DoesntExist as e: + return None + + + for item in module.get_module_items(): + + if item.type=='ExternalUrl' and item.external_url==self.metadata['external_url']: + return item + else: + continue + + return None + + + + +class File(CanvasObject): + """ + a containerization of arbitrary files, for uploading to Canvas + """ + def __init__(self, folder): + super(File, self).__init__(folder) + + import json, os + from os.path import join + + self.folder = folder + + self.metaname = path.join(folder,'meta.json') + with open(self.metaname,'r',encoding='utf-8') as f: + self.metadata = json.load(f) + + try: + self.title = self.metadata['title'] + except: + self.title = self.metadata['filename'] + + + if 'indent' in self.metadata: + self.indent = self.metadata['indent'] + else: + self.indent = 0 + + + def __str__(self): + result = f"File({self.metadata})" + return result + + def __repr__(self): + return str(self) + + + def _upload_(self, course): + pass + + + def publish(self, course, overwrite=False): + """ + publishes a file to Canvas in a particular folder + """ + + on_duplicate='overwrite' + if (file_on_canvas:= self.is_already_uploaded(course)) and not overwrite: + # on_duplicate='rename' + n = self.metadata['filename'] + # content_id = file_on_canvas.id + + raise AlreadyExists(f'The file {n} is already on Canvas and `not overwrite`.') + else: + root = get_root_folder(course) + + d = self.metadata['destination'] + d = d.split('/') + + curr_dir = root + for subd in d: + try: + curr_dir = get_subfolder_named(curr_dir, subd) + except DoesntExist as e: + curr_dir = curr_dir.create_folder(subd) + + filepath_to_upload = path.join(self.folder,self.metadata['filename']) + reply = curr_dir.upload(file=filepath_to_upload,on_duplicate=on_duplicate) + + if not reply[0]: + raise RuntimeError(f'something went wrong uploading {filepath_to_upload}') + + file_on_canvas = reply[1] + content_id = file_on_canvas['id'] + + + # now to make sure it's in the right modules + for module_name in self.metadata['modules']: + module = create_or_get_module(module_name, course) + + items = module.get_module_items() + is_in = False + for item in items: + if item.type=='File' and item.content_id==content_id: + is_in = True + break + + if not is_in: + module.create_module_item(module_item={'type':'File', 'content_id':content_id, 'title':self.title, 'indent':self.indent}) + # if the title doesn't match, update it + elif item.title != self.title: + item.edit(module_item={'type':'File', 'content_id':content_id, 'title':self.title},module=module) + + + def is_in_module(self, course, module_name): + file_on_canvas = self.is_already_uploaded(course) + + if not file_on_canvas: + return False + + module = get_module(module_name,course) + + for item in module.get_module_items(): + + if item.type=='File' and item.content_id==file_on_canvas.id: + return True + else: + continue + + return False + + + def is_already_uploaded(self,course, require_same_path=True): + files = course.get_files() + + for f in files: + if f.filename == self.metadata['filename']: + + if not require_same_path: + return f + else: + containing_folder = course.get_folder(f.folder_id) + if containing_folder.full_name.startswith('course files') and containing_folder.full_name.endswith(self.metadata['destination']): + return f + + + return None diff --git a/markdown2canvas/course_interaction_functions.py b/markdown2canvas/course_interaction_functions.py new file mode 100644 index 0000000..5b3e40a --- /dev/null +++ b/markdown2canvas/course_interaction_functions.py @@ -0,0 +1,280 @@ +""" +Functions for making or getting things in Canvas, mostly by names-as-strings. + +Note that `canvasapi` mostly uses numeric identifiers to get things. This annoyed me and so I wrote these functions. + +These functions do NOT require containerized content, so these functions are probably useful even without a containerized course using markdown2canvas. +""" + +__all__ = [ + 'is_file_already_uploaded', + 'find_file_in_course', + 'is_page_already_uploaded', + 'find_page_in_course', + 'is_assignment_already_uploaded', + 'find_assignment_in_course', + 'get_root_folder', + 'get_assignment_group_id', + 'create_or_get_assignment', + 'create_or_get_page', + 'create_or_get_module', + 'get_module', + 'get_subfolder_named', + 'delete_module' + ] + + +from markdown2canvas.exception import * +import canvasapi + +import os.path as path + + + + + + +def is_file_already_uploaded(filename,course): + """ + returns a boolean, true if there's a file of `filename` already in `course`. + + This function wants the full path to the file. + + See also `find_file_in_course` + """ + return ( not find_file_in_course(filename,course) is None ) + + + + +def find_file_in_course(filename,course): + """ + Checks to see of the file at `filename` is already in the "files" part of `course`. + + It tests filename and size as reported on disk. If it finds a match, then it's up. + + This function wants the full path to the file. + + Note that `canvasapi` does NOT differentiate + between files in different "folders" on Canvas, + so if you have multiple files of the same name, + this will find the first one that matches both name and size. + """ + import os + + base = path.split(filename)[1] + + files = course.get_files() + for f in files: + if f.filename==base and f.size == path.getsize(filename): + return f + + return None + + + + + +def is_page_already_uploaded(name,course): + """ + returns a boolean indicating whether a page of the given `name` is already in the `course`. + """ + return ( not find_page_in_course(name,course) is None ) + + +def find_page_in_course(name,course): + """ + Checks to see if there's already a page named `name` as part of `course`. + + tests merely based on the name. assumes assignments are uniquely named. + """ + + import os + pages = course.get_pages() + for p in pages: + if p.title == name: + return p + + return None + + + +def is_assignment_already_uploaded(name,course): + """ + returns a boolean indicating whether an assignment of the given `name` is already in the `course`. + """ + return ( not find_assignment_in_course(name,course) is None ) + + +def find_assignment_in_course(name,course): + """ + Checks to see if there's already an assignment named `name` as part of `course`. + + Tests merely based on the name. assumes assingments are uniquely named. + """ + import os + assignments = course.get_assignments() + for a in assignments: + + if a.name == name: + return a + + return None + + + + + +def get_root_folder(course): + """ + gets the Folder object at root level in your course. + """ + + for f in course.get_folders(): + if f.full_name == 'course files': + return f + + + + + + +def get_assignment_group_id(assignment_group_name, course, create_if_necessary=False): + """ + gets the ID number of an assignment group from its name-as-string. + + `create_if_necessary`: There are two distinct behaviours available: + + False: [default] If such a group doesn't exist, this will raise. + True: Will make such an assignment group if it doesn't exist. + + Gods, I hope the preceding description made you feel like "well duh" because my names were that spot-on. If not, let's grab a beer together and talk about it. If you read this, you're amazing, and I'm glad you're using my software. I'm trying so hard to leave positive legacy! + """ + + existing_groups = course.get_assignment_groups() + + if not isinstance(assignment_group_name,str): + raise RuntimeError(f'assignment_group_name must be a string, but I got {assignment_group_name} of type {type(assignment_group_name)}') + + + for g in existing_groups: + if g.name == assignment_group_name: + return g.id + + + + if create_if_necessary: + msg = f'making new assignment group `{assignment_group_name}`' + logger.info(msg) + + group = course.create_assignment_group(name=assignment_group_name) + group.edit(name=assignment_group_name) # this feels stupid. didn't i just request its name be this? + + return group.id + else: + raise DoesntExist(f'cannot get assignment group id because an assignment group of name {assignment_group_name} does not already exist, and `create_if_necessary` is set to False') + + + + + +def create_or_get_assignment(name, course, even_if_exists = False): + """ + gets the `canvasapi.Assignment`. Can tell it to make the assignment if it didn't exist. + """ + + if is_assignment_already_uploaded(name,course): + if even_if_exists: + return find_assignment_in_course(name,course) + else: + raise AlreadyExists(f"assignment {name} already exists") + else: + # make new assignment of name in course. + return course.create_assignment(assignment={'name':name}) + + + +def create_or_get_page(name, course, even_if_exists): + """ + gets the `canvasapi.Page`. Can tell it to make the page if it didn't exist. + """ + + if is_page_already_uploaded(name,course): + + if even_if_exists: + return find_page_in_course(name,course) + else: + raise AlreadyExists(f"page {name} already exists") + else: + # make new assignment of name in course. + result = course.create_page(wiki_page={'body':"empty page",'title':name}) + return result + + + + +def create_or_get_module(module_name, course): + """ + gets the `canvasapi.Module`. Can tell it to make the module if it didn't exist. + """ + + try: + return get_module(module_name, course) + except DoesntExist as e: + return course.create_module(module={'name':module_name}) + + + + +def get_module(module_name, course): + """ + returns + * canvasapi.Module if such a module exists, + * raises if not + """ + modules = course.get_modules() + + for m in modules: + if m.name == module_name: + return m + + raise DoesntExist(f"tried to get module {module_name}, but it doesn't exist in the course") + + +def get_subfolder_named(folder, subfolder_name): + """ + gets the `canvasapi.Folder` with matching name. + + this is likely broken if subfolder has a / in its name, / gets converted to something else by Canvas. don't use / in subfolder names, that's not allowed + + raises if doesn't exist. + """ + + assert '/' not in subfolder_name, "this is likely broken if subfolder has a / in its name, / gets converted to something else by Canvas. don't use / in subfolder names, that's not allowed" + + current_subfolders = folder.get_folders() + for f in current_subfolders: + if f.name == subfolder_name: + return f + + raise DoesntExist(f'a subfolder of {folder.name} named {subfolder_name} does not currently exist') + + +def delete_module(module_name, course, even_if_doesnt_exist): + ''' + Deletes a module by name-as-string. + ''' + + if even_if_doesnt_exist: + try: + m = get_module(module_name, course) + m.delete() + except DoesntExist as e: + return + + else: + # this path is expected to raise if the module doesn't exist + m = get_module(module_name, course) + m.delete() + + diff --git a/markdown2canvas/exception.py b/markdown2canvas/exception.py new file mode 100644 index 0000000..ef6a246 --- /dev/null +++ b/markdown2canvas/exception.py @@ -0,0 +1,45 @@ +''' +Exception types emitted by markdown2canvas +''' + +__all__ = [ + 'AlreadyExists', + 'SetupError', + 'DoesntExist' +] + +class AlreadyExists(Exception): + """ + Used to indicate that you're trying to do a thing cautiously, and the thing already existed on Canvas. + """ + + def __init__(self, message, errors=""): + super().__init__(message) + + self.errors = errors + +class SetupError(Exception): + """ + Used to indicate that markdown2canvas couldn't get off the ground, or there's something else wrong that's not content-related but meta or config. + """ + + def __init__(self, message, errors=""): + super().__init__(message) + + self.errors = errors + + + + +class DoesntExist(Exception): + """ + Used when getting a thing, but it doesn't exist. + """ + + def __init__(self, message, errors=""): + super().__init__(message) + + self.errors = errors + + + diff --git a/markdown2canvas/logging.py b/markdown2canvas/logging.py new file mode 100644 index 0000000..12117ea --- /dev/null +++ b/markdown2canvas/logging.py @@ -0,0 +1,59 @@ +''' +Logging utilities. Uses the built-in `logging` library. + +The default_* variables in here are just what the values start at. Setting the values of these variables does nothing. +You have to do something like `mc.logging.logger.setLevel(bla)` to actually get action. + +This part of the library could probably be improved to allow the user to + +* set their own levels +* turn on/off logging more easily +* control the output directory + +etc. Good luck. I believe in you. + +~silviana +''' + +__all__ = [ + 'today', 'default_log_dir', 'logger', 'file_handler' + ] + +import os.path as path +import os + +import logging + +# logging.basicConfig(encoding='utf-8') + +import datetime +today = datetime.datetime.today().strftime("%Y-%m-%d") + +default_log_level=logging.DEBUG + +default_log_dir = path.join(path.normpath(os.getcwd()), '_logs') + +if not path.exists(default_log_dir): + os.mkdir(default_log_dir) + +log_filename = path.join(default_log_dir, f'markdown2canvas_{today}.log') + + +default_log_encoding = 'utf-8' + +# make a logger object. we'll getLogger in the other files as needed. + +logger = logging.getLogger() # make a root-level logger using the defaulted options. see https://stackoverflow.com/questions/50714316/how-to-use-logging-getlogger-name-in-multiple-modules + +# adjust the logger for THIS module +logger.setLevel(default_log_level) + +# make a file handler and attach +file_handler = logging.FileHandler(log_filename, 'a', default_log_encoding) +logger.addHandler(file_handler) + +# a few messages to start +logging.debug(f'starting logging at {datetime.datetime.now()}') +logging.debug(f'setting logging level of `requests` and `canvasapi` to WARNING') +logging.getLogger('canvasapi.requester').setLevel(logging.WARNING) +logging.getLogger('requests').setLevel(logging.WARNING) \ No newline at end of file diff --git a/markdown2canvas/setup_functions.py b/markdown2canvas/setup_functions.py new file mode 100644 index 0000000..cf442e2 --- /dev/null +++ b/markdown2canvas/setup_functions.py @@ -0,0 +1,68 @@ +''' +Functions for making a `canvasapi.Canvas` object with which to work + +Uses environment variables to let you specify things. +''' + +__all__ = ['get_canvas_key_url', 'make_canvas_api_obj'] + + +import os.path as path +import os + +import logging +logger = logging.getLogger(__name__) + +import canvasapi + + + + + +def get_canvas_key_url(): + """ + reads a file using an environment variable, namely the file specified in `CANVAS_CREDENTIAL_FILE`. + + We need the + + * API_KEY + * API_URL + + variables from that file. + """ + from os import environ + + cred_loc = environ.get('CANVAS_CREDENTIAL_FILE') + if cred_loc is None: + raise SetupError('`get_canvas_key_url()` needs an environment variable `CANVAS_CREDENTIAL_FILE`, containing the full path of the file containing your Canvas API_KEY, *including the file name*') + + # yes, this is scary. it was also low-hanging fruit, and doing it another way was going to be too much work + with open(path.join(cred_loc),encoding='utf-8') as cred_file: + exec(cred_file.read(),locals()) + + if isinstance(locals()['API_KEY'], str): + logger.info(f'using canvas with API_KEY as defined in {cred_loc}') + else: + raise SetupError(f'failing to use canvas. Make sure that file {cred_loc} contains a line of code defining a string variable `API_KEY="keyhere"`') + + return locals()['API_KEY'],locals()['API_URL'] + + +def make_canvas_api_obj(url=None): + """ + - reads the key from a python file, path to which must be in environment variable CANVAS_CREDENTIAL_FILE. + - optionally, pass in a url to use, in case you don't want the default one you put in your CANVAS_CREDENTIAL_FILE. + """ + + key, default_url = get_canvas_key_url() + + if not url: + url = default_url + + return canvasapi.Canvas(url, key) + + + + + + diff --git a/markdown2canvas/tool.py b/markdown2canvas/tool.py new file mode 100644 index 0000000..2d3af9e --- /dev/null +++ b/markdown2canvas/tool.py @@ -0,0 +1,97 @@ +""" +Provides a base class from which to derive when writing tools that will interact with `markdown2canvas` or `canvasapi`. +""" + +__all__ = ['Tool'] + + +class Tool(object): + """ + A base class from which to derive. The purpose of this class is to carry these state variables: + + 1. `config`, a dictionary which is read from `config.json` (the default name). + 2. `canvas`, an instance of `Canvas` from the `canvasapi` library. + 3. `course`, an instance of `Course` from the `canvasapi` library. + + If you derive from this class, then you get access to these for free. + + I wrote this class to facilitate a few tools for automating creation of large numbers of assignments into Canvas, particularly ExternalTool assignments in the Webwork system. See the `tools` folder in the repo for markdown2canvas for examples of how I've used this `Tool` class. + """ + + + def __init__(self, config_name = 'config.json'): + """ + consruct a Tool. You must give it the name of a json file holding the following values: + + .. code-block:: json + + { + "course_id": 640131, + "canvas_url": "https://uweau.instructure.com" + } + + + Here's an example json from my webwork2canvas tool. + + .. code-block:: json + + { + "embed_webwork_in_canvas_page":false, + "name_map": "name_map_week.json", + "course_id": 640131, + "canvas_url": "https://uweau.instructure.com", + "dry_run": false, + "points_per_set": 100, + "graph_name_map":"webwork_name_to_node_name.json" + } + + + """ + + + + super(Tool, self).__init__() + + self.config = None + self.canvas = None + self.course = None + + self._read_config(config_name) + + + + + + def _read_config(self, config_name): + """ + reads `config_name` (config.json by default), and unpacks `course_id`. Stores the de-serialized json file in self.config. + """ + import json + with open(config_name,'r') as f: + config = json.load(f) + + config['course_id'] = int(config['course_id']) + + self.config = config + + + + def _require_have_config(self): + """ + a function that raises if `self.config` is `None`. + """ + + if self.config is None: + raise mc.SetupError("we don't have `self.config` yet, somehow. this should be impossible by construction, as it is constructed in the __init__ method for the Tool base class") + + + def _canvas_setup(self): + """ + gets the url and course_id from the internal `config` variable (which was deserialized from `config.json`), and populates the internal `course` and `canvas` properties of this Tool. + """ + + canvas_url = self.config['canvas_url'] # for actual teaching courses at uwec + course_id = self.config['course_id'] + + self.canvas = mc.make_canvas_api_obj(url=canvas_url) + self.course = self.canvas.get_course(course_id) \ No newline at end of file diff --git a/markdown2canvas/tool/__init__.py b/markdown2canvas/tool/__init__.py deleted file mode 100644 index c5cd283..0000000 --- a/markdown2canvas/tool/__init__.py +++ /dev/null @@ -1,41 +0,0 @@ -import markdown2canvas as mc - -# a base class -class Tool(object): - """docstring for Tool""" - def __init__(self, config_name = 'config.json'): - super(Tool, self).__init__() - - self.config = None - self.canvas = None - self.course = None - - self._read_config(config_name) - - - - - - def _read_config(self, config_name): - - import json - with open(config_name,'r') as f: - config = json.load(f) - - config['course_id'] = int(config['course_id']) - - self.config = config - - - - def _require_have_config(self): - if self.config is None: - raise mc.SetupError("we don't have self.config yet, somehow. this should be impossible by construction, as it is in the init method for the Tool base class") - - - def _canvas_setup(self): - canvas_url = self.config['canvas_url'] # for actual teaching courses at uwec - course_id = self.config['course_id'] - - self.canvas = mc.make_canvas_api_obj(url=canvas_url) - self.course = self.canvas.get_course(course_id) \ No newline at end of file diff --git a/markdown2canvas/translation_functions.py b/markdown2canvas/translation_functions.py new file mode 100644 index 0000000..91e5714 --- /dev/null +++ b/markdown2canvas/translation_functions.py @@ -0,0 +1,365 @@ +""" +Functions for translating markdown to html, +putting headers/footers around content, +and manipulating links to link to or embed images and files on Canvas. +""" + +__all__ = [ + 'generate_course_link', + 'find_in_containing_directory_path', + 'preprocess_replacements', + 'preprocess_markdown_images', + 'get_default_property', + 'get_default_style_name', + 'get_default_replacements_name', + 'default_markdown_extensions', + 'get_default_markdown_extensions', + 'apply_style_markdown', + 'apply_style_html', + 'markdown2html', + 'adjust_html_for_images', + 'adjust_html_for_files' +] + + +import os.path as path +import os + +import logging +logger = logging.getLogger(__name__) + +def generate_course_link(type,name,all_of_type,courseid=None): + ''' + Given a type (assignment or page) and the name of said object, generate a link + within course to that object. + ''' + if type in ['page','quiz']: + the_item = next( (p for p in all_of_type if p.title == name) , None) + elif type == 'assignment': + the_item = next( (a for a in all_of_type if a.name == name) , None) + elif type == 'file': + the_item = next( (a for a in all_of_type if a.display_name == name) , None) + if the_item is None: # Separate case to allow change of filenames on Canvas to names that did exist + the_item = next( (a for a in all_of_type if a.filename == name) , None) + # Canvas retains the name of the file uploaded and calls it `filename`. + # To access the name of the document seen in the Course Files, we use `display_name`. + else: + the_item = None + + + if the_item is None: + print(f"ℹī¸ No content of type `{type}` named `{name}` exists in this Canvas course. Either you have the name incorrect, the content is not yet uploaded, or you used incorrect type before the colon") + elif type == 'file' and not courseid is None: + # Construct the url with reference to the coruse its coming from + file_id = the_item.id + full_url = the_item.url + stopper = full_url.find("files") + + html_url = full_url[:stopper] + "courses/" + str(courseid) + "/files/" + str(file_id) + + return html_url + elif type == 'file': + # Construct the url - removing the "download" portion + full_url = the_item.url + stopper = full_url.find("download") + return full_url[:stopper] + else: + return the_item.html_url + + + +def find_in_containing_directory_path(target): + import pathlib + + target = pathlib.Path(target) + + here = pathlib.Path('.').absolute() + + testme = here / target + + found = testme.exists() + + while (not found) and here.parent!=here: + here = here.parent + testme = here / target + found = testme.exists() + + + if not found: + raise FileNotFoundError('unable to find {} in a containing folder of {}'.format(target, pathlib.Path('.').absolute())) + + return here / target + + + +def preprocess_replacements(contents, replacements_path): + """ + attempts to read in a file containing substitutions to make, and then makes those substitutions + """ + + if replacements_path is None: + return contents + with open(replacements_path,'r',encoding='utf-8') as f: + import json + replacements = json.loads(f.read()) + + for source, target in replacements.items(): + contents = contents.replace(source, target) + + return contents + + + + +def preprocess_markdown_images(contents,style_path): + + rel_style_path = find_in_containing_directory_path(style_path) + + contents = contents.replace('$PATHTOMD2CANVASSTYLEFILE',str(rel_style_path)) + + return contents + + +def get_default_property(key, helpstr, default_value = None): + + defaults_name = find_in_containing_directory_path(path.join("_course_metadata","defaults.json")) + + try: + logger.info(f'trying to use defaults from {defaults_name}') + with open(defaults_name,'r',encoding='utf-8') as f: + import json + defaults = json.loads(f.read()) + except Exception as e: + print(f'⚠ī¸ failed to load defaults from `{defaults_name}`. either you are not at the correct location to be doing this, or you need to create a json file at {defaults_name}. returning a default value {default_value}') + return default_value + + if key in defaults: + return defaults[key] + else: + print(f'no default `{key}` specified in {defaults_name}. add an entry with key `{key}`, being {helpstr}. returning a default value of {default_value}') + return default_value + + + + +def get_default_style_name(): + return get_default_property(key='style', + helpstr='a path to a file relative to the top course folder') + +def get_default_replacements_name(): + return get_default_property(key='replacements', + helpstr='a path to a json file containing key:value pairs of text-to-replace. this path should be expressed relative to the top course folder') + + +default_markdown_extensions = ['codehilite','fenced_code','md_in_html','tables','nl2br'] #: The default `markdown` extensions to use when translating from markdown to html during publishing + +def get_default_markdown_extensions(): + return get_default_property(key='markdown_extensions', + helpstr='a list of strings being extensions to the `markdown` library used to translate from markdown to html before uploading to canvas.', + default_value = default_markdown_extensions) + + +def apply_style_markdown(sourcename, style_path, outname): + from os.path import join + logger.debug(f'Applying markdown header and footer from `{style_path}`.') + # need to add header and footer. assume they're called `header.md` and `footer.md`. we're just going to concatenate them and dump to file. + + with open(sourcename,'r',encoding='utf-8') as f: + body = f.read() + + with open(join(style_path,'header.md'),'r',encoding='utf-8') as f: + header = f.read() + + with open(join(style_path,'footer.md'),'r',encoding='utf-8') as f: + footer = f.read() + + + contents = f'{header}\n{body}\n{footer}' + contents = preprocess_markdown_images(contents, style_path) + + with open(outname,'w',encoding='utf-8') as f: + f.write(contents) + + return contents + + + + +def apply_style_html(translated_html_without_hf, style_path, outname): + from os.path import join + logger.debug(f'Applying html header and footer from `{style_path}`.') + # need to add header and footer. assume they're called `header.html` and `footer.html`. we're just going to concatenate them and dump to file. + + with open(join(style_path,'header.html'),'r',encoding='utf-8') as f: + header = f.read() + + with open(join(style_path,'footer.html'),'r',encoding='utf-8') as f: + footer = f.read() + + contents = f'{header}\n{translated_html_without_hf}\n{footer}' + + with open(outname,'w',encoding='utf-8') as f: + f.write(contents) + + return contents + + + + + +def markdown2html(filename, course, replacements_path, markdown_extensions): + """ + This is the main routine in the library. + + This function returns a string of html code. + + It does replacements, emojizes, converts markdown-->html via `markdown.markdown`, and does page, assignment, and file reference link adjustments. + + If `course` is None, then you won't get some of the functionality. In particular, you won't get link replacements for references to other content on Canvas. + + If `replacements_path` is None, then no replacements, duh. Otherwise it should be a string or Path object to an existing json file containing key-value pairs of strings to replace with other strings. + + `markdown_extensions` is a list of strings (or functions) specifying extensions to the `markdown` library to use during + translation from markdown to HTML. + The list I've been using for DS710/DS150 has been + ['codehilite','fenced_code','md_in_html','tables','nl2br']. + You can find more on available extensions, and writing your own, at https://python-markdown.github.io/extensions/ + + You can provide your own list of extensions in the `_course_metadata/defaults.json` file + via the `markdown_extensions` key-value pair. + If no such key/value pair exists in `defaults.json`, then the following default will be provided for you: + `['codehilite','fenced_code','md_in_html','tables','nl2br']` + """ + logger.debug(f'Translating `{filename}` from markdown to html using replacements from `{replacements_path}`.') + + if course is None: + courseid = None + else: + courseid = course.id + + root = path.split(filename)[0] + + import emoji + import markdown + from bs4 import BeautifulSoup + + + with open(filename,'r',encoding='utf-8') as file: + markdown_source = file.read() + + markdown_source = preprocess_replacements(markdown_source, replacements_path) + + emojified = emoji.emojize(markdown_source) + + + html = markdown.markdown(emojified, extensions=markdown_extensions) # see https://python-markdown.github.io/extensions/ + soup = BeautifulSoup(html,features="lxml") + + all_imgs = soup.findAll("img") + for img in all_imgs: + src = img["src"] + if ('http://' not in src) and ('https://' not in src): + img["src"] = path.join(root,src) + + all_links = soup.findAll("a") + course_page_and_assignments = {} + if any(l['href'].startswith("page:") for l in all_links) and course: + course_page_and_assignments['page'] = course.get_pages() + if any(l['href'].startswith("assignment:") for l in all_links) and course: + course_page_and_assignments['assignment'] = course.get_assignments() + if any(l['href'].startswith("quiz:") for l in all_links) and course: + course_page_and_assignments['quiz'] = course.get_quizzes() + if any(l['href'].startswith("file:") for l in all_links) and course: + course_page_and_assignments['file'] = course.get_files() + for f in all_links: + href = f["href"] + root_href = path.join(root,href) + split_at_colon = href.split(":",1) + if path.exists(path.abspath(root_href)): + f["href"] = root_href + elif course and split_at_colon[0] in ['assignment','page','quiz','file']: + type = split_at_colon[0] + name = split_at_colon[1].strip() + get_link = generate_course_link(type,name,course_page_and_assignments[type],courseid) + if get_link: + f["href"] = get_link + + + return str(soup) + + + + + + + + + +def adjust_html_for_images(html, published_images, courseid): + """ + + published_images: a dict of Image objects, which should have been published (so we have their canvas objects stored into them) + + this function edits the html source, replacing local url's + with url's to images on Canvas. + """ + from bs4 import BeautifulSoup + + soup = BeautifulSoup(html,features="lxml") + + all_imgs = soup.findAll("img") + if all_imgs: + for img in all_imgs: + src = img["src"] + if src[:7] not in ['https:/','http://']: + # find the image in the list of published images, replace url, do more stuff. + local_img = published_images[src] + img['src'] = local_img.make_src_url(courseid) + img['class'] = "instructure_file_link inline_disabled" + img['data-api-endpoint'] = local_img.make_api_endpoint_url(courseid) + img['data-api-returntype'] = 'File' + + return str(soup) + + #

+ # hauser_menagerie.jpg + #

+ + + + + + +def adjust_html_for_files(html, published_files, courseid): + + + # need to write a url like this : + # Download + + + from bs4 import BeautifulSoup + + soup = BeautifulSoup(html,features="lxml") + + all_files = soup.findAll("a") + + if all_files: + for file in all_files: + href = file["href"] + if path.exists(path.abspath(href)): + # find the image in the list of published images, replace url, do more stuff. + local_file = published_files[href] + file['href'] = local_file.make_href_url(courseid) + file['class'] = "instructure_file_link instructure_scribd_file" + file['title'] = local_file.name # what it's called when you download it??? + file['data-api-endpoint'] = local_file.make_api_endpoint_url(courseid) + file['data-api-returntype'] = 'File' + + return str(soup) + + + + + + + diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..869bb99 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,22 @@ +[build-system] +requires = ["setuptools>=61.0"] +build-backend = "setuptools.build_meta" + +[project] +name = "markdown2canvas" +version = "0.2" +authors = [ + { name="Silviana Amethyst", email="amethyst@uwec.edu" }, +] +description = "code for publishing markdown documents to Canvas pages" +readme = "README.md" +requires-python = ">=3.10" +classifiers = [ + "Programming Language :: Python :: 3", + "License :: ", + "Operating System :: OS Independent", +] + +[project.urls] +Homepage = "https://github.com/ofloveandhate/markdown2canvas" +Issues = "https://github.com/ofloveandhate/markdown2canvas/issues" diff --git a/test/_styles/custom/Ruapehu_and_Ngauruhoe.jpg b/test/_styles/custom/Ruapehu_and_Ngauruhoe.jpg new file mode 100644 index 0000000..f24a886 Binary files /dev/null and b/test/_styles/custom/Ruapehu_and_Ngauruhoe.jpg differ diff --git a/test/_styles/custom/footer.html b/test/_styles/custom/footer.html new file mode 100644 index 0000000..12fd014 --- /dev/null +++ b/test/_styles/custom/footer.html @@ -0,0 +1,4 @@ + + +
+ diff --git a/test/_styles/custom/footer.md b/test/_styles/custom/footer.md new file mode 100644 index 0000000..6e916b5 --- /dev/null +++ b/test/_styles/custom/footer.md @@ -0,0 +1,4 @@ +--- + +Header image credit: Jeremy Visser, CC BY-SA 4.0 , via Wikimedia Commons. + diff --git a/test/_styles/custom/header.html b/test/_styles/custom/header.html new file mode 100644 index 0000000..1ea1d39 --- /dev/null +++ b/test/_styles/custom/header.html @@ -0,0 +1,7 @@ + + + + +
+ + diff --git a/test/_styles/custom/header.md b/test/_styles/custom/header.md new file mode 100644 index 0000000..72d4842 --- /dev/null +++ b/test/_styles/custom/header.md @@ -0,0 +1,4 @@ + +![This is a photo of Mount Ruapehu and Mount Ngauruhoe looking west from the Desert Road in Tongariro National Park (New Zealand) in January 2015.]($PATHTOMD2CANVASSTYLEFILE/Ruapehu_and_Ngauruhoe.jpg) + +--- diff --git a/test/_styles/generic/header.html b/test/_styles/generic/header.html index df618fc..1ea1d39 100644 --- a/test/_styles/generic/header.html +++ b/test/_styles/generic/header.html @@ -2,6 +2,6 @@ -
+
diff --git a/test/course_id.py b/test/course_id.py index 9586ff5..7c6fdd2 100644 --- a/test/course_id.py +++ b/test/course_id.py @@ -1 +1 @@ -test_course_id = 537006 +test_course_id = 705022 diff --git a/test/a_file.file/ds150_course_logo.pdf b/test/file_uploads/a_file.file/ds150_course_logo.pdf similarity index 100% rename from test/a_file.file/ds150_course_logo.pdf rename to test/file_uploads/a_file.file/ds150_course_logo.pdf diff --git a/test/a_file.file/meta.json b/test/file_uploads/a_file.file/meta.json similarity index 80% rename from test/a_file.file/meta.json rename to test/file_uploads/a_file.file/meta.json index 1b6bd2d..fac4a8f 100644 --- a/test/a_file.file/meta.json +++ b/test/file_uploads/a_file.file/meta.json @@ -2,6 +2,6 @@ "type":"file", "title":"automatically uploaded file: ds150_course_logo.pdf", "filename":"ds150_course_logo.pdf", - "modules":["Automatically Added Test Module", "Another automatically added test module"], + "modules":["Automatically Added Test Module", "Another automatically added test module", "module created by file upload"], "destination": "automatically_uploaded_files/a_subfolder" } \ No newline at end of file diff --git a/test/file_uploads/ab_file.file/meta.json b/test/file_uploads/ab_file.file/meta.json new file mode 100644 index 0000000..820083d --- /dev/null +++ b/test/file_uploads/ab_file.file/meta.json @@ -0,0 +1,7 @@ +{ + "type":"file", + "title":"automatically uploaded file: ds150_course_logo_2.pdf", + "filename":"../ds150_course_logo_2.pdf", + "modules":["Automatically Added Test Module", "Another automatically added test module", "module created by file upload"], + "destination": "automatically_uploaded_files/a_subfolder" +} \ No newline at end of file diff --git a/test/file_uploads/ds150_course_logo_2.pdf b/test/file_uploads/ds150_course_logo_2.pdf new file mode 100644 index 0000000..93748dc Binary files /dev/null and b/test/file_uploads/ds150_course_logo_2.pdf differ diff --git a/test/test_assignment.py b/test/test_assignment.py index 41c8e83..85dde44 100644 --- a/test/test_assignment.py +++ b/test/test_assignment.py @@ -46,12 +46,12 @@ def test_can_publish(self, course, assignment): def test_can_find_published(self, course, assignment): assignment.publish(course,overwrite=True) - assert mc.is_assignment_already_uploaded(assignment.name,course) + assert mc.course_interaction_functions.is_assignment_already_uploaded(assignment.name,course) def test_published_has_properties(self, course, assignment): assignment.publish(course,overwrite=True) - on_canvas = mc.find_assignment_in_course(assignment.name,course) + on_canvas = mc.course_interaction_functions.find_assignment_in_course(assignment.name,course) assert on_canvas.points_possible == assignment.points_possible assert 'jpg' in on_canvas.allowed_extensions @@ -64,15 +64,15 @@ def test_already_online_raises(self, course, assignment): assignment.publish(course,overwrite=True) # the second publish, with overwrite=False, should raise - with pytest.raises(mc.AlreadyExists): + with pytest.raises(mc.exception.AlreadyExists): assignment.publish(course,overwrite=False) # default is False def test_doesnt_find_deleted(self, course, assignment): name = assignment.name assignment.publish(course,overwrite=True) - assert mc.is_assignment_already_uploaded(name,course) - f = mc.find_assignment_in_course(name,course) + assert mc.course_interaction_functions.is_assignment_already_uploaded(name,course) + f = mc.course_interaction_functions.find_assignment_in_course(name,course) f.delete() # print([i.name for i in course.get_assignments()]) - assert not mc.is_assignment_already_uploaded(name,course) + assert not mc.course_interaction_functions.is_assignment_already_uploaded(name,course) diff --git a/test/test_download_pages_to_markdown.py b/test/test_download_pages_to_markdown.py index cd68c8c..1d3ccd5 100644 --- a/test/test_download_pages_to_markdown.py +++ b/test/test_download_pages_to_markdown.py @@ -32,7 +32,7 @@ def test_aaa_can_download_all_pages(self): if os.path.exists(destination): shutil.rmtree(destination) - mc.download_pages(destination, self.course, even_if_exists=False) + mc.canvas2markdown.download_pages(destination, self.course, even_if_exists=False) def test_aaa_can_download_some_pages(self): import os, shutil @@ -41,7 +41,7 @@ def test_aaa_can_download_some_pages(self): shutil.rmtree(destination) my_filter = lambda title: 'test' in title.lower() - mc.download_pages(destination, self.course, even_if_exists=False, name_filter=my_filter) + mc.canvas2markdown.download_pages(destination, self.course, even_if_exists=False, name_filter=my_filter) if __name__ == '__main__': diff --git a/test/test_droplets.py b/test/test_droplets.py index 818cef3..7170bd8 100644 --- a/test/test_droplets.py +++ b/test/test_droplets.py @@ -48,7 +48,7 @@ def test_already_online_raises(self, course, page): page.publish(course,overwrite=True) # the second publish, with overwrite=False, should raise - with pytest.raises(mc.AlreadyExists): + with pytest.raises(mc.exception.AlreadyExists): page.publish(course,overwrite=False) # default is False @@ -57,17 +57,17 @@ def test_doesnt_find_deleted(self, course, page): name = page.name page.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(name,course) - f = mc.find_page_in_course(name,course) + assert mc.course_interaction_functions.is_page_already_uploaded(name,course) + f = mc.course_interaction_functions.find_page_in_course(name,course) f.delete() # print([i.name for i in course.get_pages()]) - assert not mc.is_page_already_uploaded(name,course) + assert not mc.course_interaction_functions.is_page_already_uploaded(name,course) def test_can_find_published(self, course, page): page.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(page.name,course) + assert mc.course_interaction_functions.is_page_already_uploaded(page.name,course) diff --git a/test/test_file.py b/test/test_file.py index e32a7da..06cee04 100644 --- a/test/test_file.py +++ b/test/test_file.py @@ -4,6 +4,8 @@ import markdown2canvas as mc import pytest +import json +import datetime @pytest.fixture(scope='class') def course(): @@ -16,12 +18,28 @@ def course(): yield canvas.get_course(test_course_id) @pytest.fixture(scope='class') -def content(course): +def content(): import os folder = 'a_file.file' yield mc.File(folder) +#the following gives all instances of that file, if it lives in multiple locations +@pytest.fixture(scope='class') +def file_list(course): + yield course.get_files(search_term = 'ds150_course_logo.pdf') + +#gives the current instance, based on the destination in meta.json +@pytest.fixture(scope='class') +def current_file(course, file_list): + with open('a_file.file/meta.json', "r", encoding="utf-8") as f: + folder_name = json.loads(f.read())['destination'] + for instance in file_list: + if instance.folder_id == course.get_folders(search_term=folder_name)[0].id: + yield instance + + + class TestFile(): @@ -33,18 +51,41 @@ def test_meta(self, content): def test_can_publish(self, course, content): content.publish(course,overwrite=True) assert content.is_already_uploaded(course) + + def test_in_modules(self, course, content): + content.publish(course,overwrite=True) for m in content.metadata['modules']: assert content.is_in_module(course, m) + module_test = course.get_modules(search_term = m)[0] + assert module_test.get_module_items(search_term = 'ds150')[0].title == 'automatically uploaded file: ds150_course_logo.pdf' + #tests that it ends up in the folder you specified this time (it can simultaneously be in another folder if you put it there previously) + def test_in_folder(self, course, content, file_list, current_file): + content.publish(course,overwrite=True) + with open('a_file.file/meta.json', "r", encoding="utf-8") as f: + folder_name = json.loads(f.read())['destination'] + folder_list=[] + for instance in file_list: + folder_list.append(instance.folder_id) + assert course.get_folders(search_term=folder_name)[0].id in folder_list + assert current_file.folder_id == course.get_folders(search_term=folder_name)[0].id def test_already_online_raises(self, course, content): # publish once, forcefully. content.publish(course,overwrite=True) # the second publish, with overwrite=False, should raise - with pytest.raises(mc.AlreadyExists): + with pytest.raises(mc.exception.AlreadyExists): content.publish(course,overwrite=False) # default is False + def test_attributes(self, course, content, current_file): + content.publish(course,overwrite=True) + assert current_file.filename == 'ds150_course_logo.pdf' + + + + + diff --git a/test/test_image.py b/test/test_image.py index 9d4f0b3..6d2f8bd 100644 --- a/test/test_image.py +++ b/test/test_image.py @@ -40,20 +40,22 @@ def test_can_publish_image(self, course, image): def test_can_find_published_image(self, course, image): image.publish(course,'images',overwrite=True) - assert mc.is_file_already_uploaded(file_to_publish,course) + assert mc.course_interaction_functions.is_file_already_uploaded(file_to_publish,course) def test_doesnt_find_deleted_image(self, course, image): image.publish(course,'images',overwrite=True) - assert mc.is_file_already_uploaded(file_to_publish,course) - f = mc.find_file_in_course(file_to_publish,course) + assert mc.course_interaction_functions.is_file_already_uploaded(file_to_publish,course) + + f = mc.course_interaction_functions.find_file_in_course(file_to_publish,course) f.delete() - assert not mc.is_file_already_uploaded(file_to_publish,course) + + assert not mc.course_interaction_functions.is_file_already_uploaded(file_to_publish,course) def test_can_get_already_published_image(self, course, image): # first, definitely publish image.publish(course,'images',overwrite=True) - img_on_canvas = mc.find_file_in_course(file_to_publish,course) + img_on_canvas = mc.course_interaction_functions.find_file_in_course(file_to_publish,course) assert img_on_canvas.filename == filename diff --git a/test/test_link.py b/test/test_link.py index 05058b0..49b4073 100644 --- a/test/test_link.py +++ b/test/test_link.py @@ -51,7 +51,7 @@ def test_already_online_raises(self,course,link): link.publish(course,overwrite=True) # the second publish, with overwrite=False, should raise - with pytest.raises(mc.AlreadyExists): + with pytest.raises(mc.exception.AlreadyExists): link.publish(course,overwrite=False) # default is False diff --git a/test/test_link_to_local_file.py b/test/test_link_to_local_file.py index d910e99..a63d956 100644 --- a/test/test_link_to_local_file.py +++ b/test/test_link_to_local_file.py @@ -49,7 +49,7 @@ def test_already_online_raises(self, course, page): page.publish(course,overwrite=True) # the second publish, with overwrite=False, should raise - with pytest.raises(mc.AlreadyExists): + with pytest.raises(mc.exception.AlreadyExists): page.publish(course,overwrite=False) # default is False @@ -60,17 +60,17 @@ def test_doesnt_find_deleted(self, course, page): name = page.name page.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(name,course) - f = mc.find_page_in_course(name,course) + assert mc.course_interaction_functions.is_page_already_uploaded(name,course) + f = mc.course_interaction_functions.find_page_in_course(name,course) f.delete() # print([i.name for i in course.get_pages()]) - assert not mc.is_page_already_uploaded(name,course) + assert not mc.course_interaction_functions.is_page_already_uploaded(name,course) def test_can_find_published(self, course, page): page.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(page.name,course) + assert mc.course_interaction_functions.is_page_already_uploaded(page.name,course) diff --git a/test/test_page.py b/test/test_page.py index 19fa8b7..c31b41a 100644 --- a/test/test_page.py +++ b/test/test_page.py @@ -38,19 +38,23 @@ def test_already_online_raises(self, course, page_has_local_images): page_has_local_images.publish(course,overwrite=True) # the second publish, with overwrite=False, should raise - with pytest.raises(mc.AlreadyExists): + with pytest.raises(mc.exception.AlreadyExists): page_has_local_images.publish(course,overwrite=False) # default is False def test_doesnt_find_deleted(self, course, page_has_local_images): name = page_has_local_images.name - page_has_local_images.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(name,course) - f = mc.find_page_in_course(name,course) + f = mc.course_interaction_functions.find_page_in_course(name,course) f.delete() - assert not mc.is_page_already_uploaded(name,course) + assert not mc.course_interaction_functions.is_page_already_uploaded(name,course) def test_can_find_published(self, course, page_has_local_images): page_has_local_images.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(page_has_local_images.name,course) + assert mc.course_interaction_functions.is_page_already_uploaded(page_has_local_images.name,course) + + def test_content(self, course): + content = course.get_pages(search_term='Test Has Local Images')[0].show_latest_revision().body + assert 'testing source including images' in content + assert 'alt="A menagerie of Herwig Hauser surfaces"' in content + assert ('## an image using html' not in content) and "The markdown header was not translated to html." diff --git a/test/test_page_in_module.py b/test/test_page_in_module.py index d48152b..cebe368 100644 --- a/test/test_page_in_module.py +++ b/test/test_page_in_module.py @@ -31,11 +31,9 @@ def destination_modules(page_plain_text_in_a_module): yield page.metadata['modules'] -#self._delete_test_modules() - -def _delete_test_modules(self): - for m in self.destination_modules: - mc.delete_module(m, self.course, even_if_exists=True) +def _delete_test_modules(course, destination_modules): + for m in destination_modules: + mc.course_interaction_functions.delete_module(m, course, even_if_doesnt_exist=True) @@ -54,29 +52,29 @@ def test_already_online_raises(self, course, page_plain_text_in_a_module): page_plain_text_in_a_module.publish(course,overwrite=True) # the second publish, with overwrite=False, should raise - with pytest.raises(mc.AlreadyExists): + with pytest.raises(mc.exception.AlreadyExists): page_plain_text_in_a_module.publish(course,overwrite=False) # default is False def test_can_make_modules(self, course, destination_modules): for m in destination_modules: - mc.create_or_get_module(m,course) + mc.course_interaction_functions.create_or_get_module(m,course) def test_can_delete_modules(self, course, destination_modules): - + _delete_test_modules(course, destination_modules) for m in destination_modules: - mc.create_or_get_module(m,course) + mc.course_interaction_functions.create_or_get_module(m,course) for m in destination_modules: - mc.delete_module(m, course, even_if_exists=False) + mc.course_interaction_functions.delete_module(m, course, even_if_doesnt_exist=False) def test_page_in_module_after_publishing(self, course, page_plain_text_in_a_module, destination_modules): page_plain_text_in_a_module.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(page_plain_text_in_a_module.name,course) + assert mc.course_interaction_functions.is_page_already_uploaded(page_plain_text_in_a_module.name,course) page_plain_text_in_a_module.ensure_in_modules(course) @@ -91,15 +89,15 @@ def test_page_in_module_after_publishing(self, course, page_plain_text_in_a_modu # name = self.page.name # self.page.publish(self.course,overwrite=True) - # self.assertTrue(mc.is_page_already_uploaded(name,self.course)) + # self.assertTrue(mc.course_interaction_functions.is_page_already_uploaded(name,self.course)) # f = mc.find_page_in_course(name,self.course) # f.delete() # # print([i.name for i in self.course.get_pages()]) - # self.assertTrue(not mc.is_page_already_uploaded(name,self.course)) + # self.assertTrue(not mc.course_interaction_functions.is_page_already_uploaded(name,self.course)) # def test_zzz_can_find_published(self): # self.page.publish(self.course,overwrite=True) - # self.assertTrue(mc.is_page_already_uploaded(self.page.name,self.course)) + # self.assertTrue(mc.course_interaction_functions.is_page_already_uploaded(self.page.name,self.course)) diff --git a/test/test_replacements.py b/test/test_replacements.py index d8c7381..91140ef 100644 --- a/test/test_replacements.py +++ b/test/test_replacements.py @@ -2,8 +2,11 @@ sys.path.insert(0,'../') import markdown2canvas as mc +import json + import pytest + @pytest.fixture(scope='class') def course(): import os @@ -17,21 +20,66 @@ def course(): @pytest.fixture(scope='class') -def page_using_defaults(course): +def page_using_defaults(): import os folder = 'uses_replacements_default' yield mc.Page(folder) + @pytest.fixture(scope='class') -def page_using_custom(course): +def page_using_custom(): import os folder = 'uses_replacements_custom' yield mc.Page(folder) +@pytest.fixture(scope='class') +def default_filename(): + with open('_course_metadata/defaults.json', "r", encoding="utf-8") as f: + defaults = f.read() + yield json.loads(defaults)['replacements'] + +@pytest.fixture(scope='class') +def replacements_default(default_filename): + with open(default_filename, "r", encoding="utf-8") as f: + yield f.read() + +@pytest.fixture(scope='class') +def uses_defaults_source(): + with open('uses_replacements_default/source.md', "r", encoding="utf-8") as f: + yield f.read() + + +@pytest.fixture(scope='class') +def html_using_defaults(course): + a = course.get_pages(search_term = 'Test replacements using default replacements file')[0] + rev = a.show_latest_revision() + yield rev.body + + +@pytest.fixture(scope='class') +def replacements_custom(): + with open('_course_metadata/replacements2.json', "r", encoding="utf-8") as f: + yield f.read() + +@pytest.fixture(scope='class') +def uses_custom_source(): + with open('uses_replacements_custom/source.md', "r", encoding="utf-8") as f: + yield f.read() + +@pytest.fixture(scope='class') +def html_using_custom(course): + a = course.get_pages(search_term = 'Test replacements with custom replacements file')[0] + rev = a.show_latest_revision() + yield rev.body + + + + + class TestPage(): def test_can_publish(self, course, page_using_defaults, page_using_custom): @@ -39,7 +87,57 @@ def test_can_publish(self, course, page_using_defaults, page_using_custom): page_using_custom.publish(course,overwrite=True) - ##Removed a " as e_info" after the def in the following... doesn't seem to have hurt it? + def test_get_default_replacements_name(self): + path = mc.translation_functions.get_default_replacements_name() + assert path == '_course_metadata/replacements.json' + + + def test_removed_default(self, html_using_defaults, replacements_default, uses_defaults_source): + replacements_dict_default = json.loads(replacements_default) + for key in replacements_dict_default: + if key in uses_defaults_source: + assert key not in html_using_defaults + #Want to add something about the new thing being in the html + #assert replacements_dict_default[key] in html_using_defaults + + def test_replaced_default(self, html_using_defaults): + #default replacements that should translate seamlessly + assert 'with this text' in html_using_defaults + assert 'destination_without_spaces' in html_using_defaults + #check specific video options + assert '560' in html_using_defaults + assert '315' in html_using_defaults + assert 'https://www.youtube.com/embed/dQw4w9WgXcQ?si=BqTm4nbZOLTHaxnz' in html_using_defaults + assert 'YouTube video player' in html_using_defaults + assert 'accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share' in html_using_defaults + assert 'allowfullscreen' in html_using_defaults + + def test_removed_custom(self, html_using_custom, uses_custom_source, replacements_custom): + replacements_dict_custom = json.loads(replacements_custom) + for key in replacements_dict_custom: + if key in uses_custom_source: + assert key not in html_using_custom + assert replacements_dict_custom[key] in html_using_custom + + def test_replaced_custom(self, html_using_custom): + #custom replacements that should translate seamlessly + assert 'target custom replacement without space' in html_using_custom + assert 'target custom replacement from nospace' in html_using_custom + + def test_incorrect_replacement_custom(self, html_using_custom): + #First check that none of the default replacements show up in the custom replacements file + assert 'with this text' not in html_using_custom + assert 'destination_without_spaces' not in html_using_custom + assert 'https://www.youtube.com/embed/dQw4w9WgXcQ?si=BqTm4nbZOLTHaxnz' not in html_using_custom + + + def test_incorrect_replacement_default(self, html_using_defaults): + #First check that none of the default replacements show up in the custom replacements file + assert 'target custom replacement without space' not in html_using_defaults + assert 'target custom replacement from nospace' not in html_using_defaults + + + def test_missing_replacements(self): # constructing a page with a replacements file that doesn't exist should raise with pytest.raises(FileNotFoundError): diff --git a/test/test_style.py b/test/test_style.py index 8ad8bd0..27971b9 100644 --- a/test/test_style.py +++ b/test/test_style.py @@ -17,45 +17,80 @@ def course(): yield canvas.get_course(test_course_id) @pytest.fixture(scope='class') -def page_uses_droplets_via_style(course): +def page_uses_droplets_via_style_generic(): import os folder = 'uses_droplets_via_style' yield mc.Page(folder) +@pytest.fixture(scope='class') +def page_uses_droplets_via_style_custom(): + import os + folder = 'uses_droplets_via_style_custom' + + yield mc.Page(folder) + + +@pytest.fixture(scope='class') +def page_contents_generic(course, page_uses_droplets_via_style_generic): + page_uses_droplets_via_style_generic.publish(course,overwrite=True) + a = course.get_pages(search_term = 'Test Uses Droplets via Style')[1] + rev = a.show_latest_revision() + yield rev.body + +@pytest.fixture(scope='class') +def page_contents_custom(course,page_uses_droplets_via_style_custom): + page_uses_droplets_via_style_custom.publish(course,overwrite=True) + a = course.get_pages(search_term = 'Test Uses Droplets via Style Custom')[0] + rev = a.show_latest_revision() + yield rev.body class TestStyle(): - def test_meta(self, page_uses_droplets_via_style): - assert page_uses_droplets_via_style.name == 'Test Uses Droplets via Style' + def test_meta(self, page_uses_droplets_via_style_generic,page_uses_droplets_via_style_custom): + assert page_uses_droplets_via_style_generic.name == 'Test Uses Droplets via Style' + assert page_uses_droplets_via_style_custom.name == 'Test Uses Droplets via Style Custom' - def test_can_publish(self, course, page_uses_droplets_via_style): - page_uses_droplets_via_style.publish(course,overwrite=True) + def test_can_publish(self, course, page_uses_droplets_via_style_generic,page_uses_droplets_via_style_custom): + page_uses_droplets_via_style_generic.publish(course,overwrite=True) + page_uses_droplets_via_style_custom.publish(course,overwrite=True) - def test_already_online_raises(self, course, page_uses_droplets_via_style): + def test_already_online_raises(self, course,page_uses_droplets_via_style_custom): # publish once, forcefully. - page_uses_droplets_via_style.publish(course,overwrite=True) - + page_uses_droplets_via_style_custom.publish(course,overwrite=True) # the second publish, with overwrite=False, should raise - with pytest.raises(mc.AlreadyExists): - page_uses_droplets_via_style.publish(course,overwrite=False) # default is False - + with pytest.raises(mc.exception.AlreadyExists): + page_uses_droplets_via_style_custom.publish(course,overwrite=False) # default is False - def test_doesnt_find_deleted(self, course, page_uses_droplets_via_style): - name = page_uses_droplets_via_style.name + def test_doesnt_find_deleted(self, course, page_uses_droplets_via_style_generic): + name = page_uses_droplets_via_style_generic.name - page_uses_droplets_via_style.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(name,course) - f = mc.find_page_in_course(name,course) + page_uses_droplets_via_style_generic.publish(course,overwrite=True) + assert mc.course_interaction_functions.is_page_already_uploaded(name,course) + f = mc.course_interaction_functions.find_page_in_course(name,course) f.delete() # print([i.name for i in course.get_pages()]) - assert not mc.is_page_already_uploaded(name,course) + assert not mc.course_interaction_functions.is_page_already_uploaded(name,course) - def test_can_find_published(self, course, page_uses_droplets_via_style): - page_uses_droplets_via_style.publish(course,overwrite=True) - assert mc.is_page_already_uploaded(page_uses_droplets_via_style.name,course) + def test_can_find_published(self, course, page_uses_droplets_via_style_generic): + page_uses_droplets_via_style_generic.publish(course,overwrite=True) + assert mc.course_interaction_functions.is_page_already_uploaded(page_uses_droplets_via_style_generic.name,course) + def test_default_style_implemented(course, page_contents_generic): + assert 'Markdown header content here' in page_contents_generic + assert 'Header image credit: Medoffer, CC BY-SA 4.0' in page_contents_generic + assert 'This is a photo of a natural heritage site in Ukraine, id: 59-247-5004.' in page_contents_generic + + def test_custom_style_implemented(course, page_contents_custom): + assert 'Header image credit: Jeremy Visser, CC BY-SA 4.0' in page_contents_custom + assert 'This is a photo of Mount Ruapehu and Mount Ngauruhoe looking west from the Desert Road in Tongariro National Park (New Zealand) in January 2015.' in page_contents_custom + assert ('![This is a photo' not in page_contents_custom) and "The header image was not translated to html." + + def test_incorrect_style_used(course, page_contents_generic, page_contents_custom): + assert 'Header image credit: Medoffer, CC BY-SA 4.0' not in page_contents_custom + assert 'Markdown header content here' not in page_contents_custom + assert 'Header image credit: Jeremy Visser, CC BY-SA 4.0' not in page_contents_generic diff --git a/test/uses_droplets_via_style/meta.json b/test/uses_droplets_via_style/meta.json index a7446e6..f6a9ba5 100644 --- a/test/uses_droplets_via_style/meta.json +++ b/test/uses_droplets_via_style/meta.json @@ -1,6 +1,5 @@ { "type": "page", "name": "Test Uses Droplets via Style", - "style":"_styles/generic", "modules":["Automatically Added Module"] } diff --git a/test/uses_droplets_via_style_custom/hauser_menagerie.jpg b/test/uses_droplets_via_style_custom/hauser_menagerie.jpg new file mode 100644 index 0000000..fc1dc27 Binary files /dev/null and b/test/uses_droplets_via_style_custom/hauser_menagerie.jpg differ diff --git a/test/uses_droplets_via_style_custom/meta.json b/test/uses_droplets_via_style_custom/meta.json new file mode 100644 index 0000000..f2319d2 --- /dev/null +++ b/test/uses_droplets_via_style_custom/meta.json @@ -0,0 +1,6 @@ +{ + "type": "page", + "name": "Test Uses Droplets via Style Custom", + "style": "_styles/custom", + "modules":["Automatically Added Module"] +} diff --git a/test/uses_droplets_via_style_custom/source.md b/test/uses_droplets_via_style_custom/source.md new file mode 100644 index 0000000..4e139b1 --- /dev/null +++ b/test/uses_droplets_via_style_custom/source.md @@ -0,0 +1,39 @@ + +# Testing upload of pages using Droplets + + +## an image using droplets + +
+ A menagerie of Herwig Hauser surfaces +
+ + +
+ +## ^^^ a horizontal rule + +## Code + +
+import numpy as np
+x,y= np.linspace(-1,1,20),np.linspace(-1,1,21)
+X,Y = np.meshgrid(x,y)
+
+ +## testing callouts + +
+

Heading

+

Fred and George, who had been spying on the Slytherin team, had seen for themselves the speed of those new Nimbus Two Thousand and Ones.

+
+ + + +## a table should be below + +| header 1 | header 2 | +| --- | --- | +| val 1 | val 2 | +| arst | axzcv | +