Alpha 1 #2

haydenjackson01 · 2017-09-18T02:14:12Z

The render service prepped and ready for local development. This means others should be able to run locally and test their own additions. This is a huge PR and is best viewed on your own machine, take your time and leave lots of comments but most attention should be on the /renderservice folder.

Current known issues:

Images are not scaled correctly in output pdfs
More documentation for top level elements e.g. dev folder
It isn't the nicest due to all the constraints and needs work

How do I test this?
Aside for normal methods such as unit tests I have added a script in the dev folder that acts as a client on the queue service, allowing developers to create tasks and retrieve output.

Some examples of usage would be:

python queue_client.py add which adds a task.
python queue_client.py list which lists up to 100 tasks.
python queue_client.py document which saves out any documents found in the task queue.

Since all queue data is stored in the redis server the command redis-cli will log in and allow the user to interact with the data using commands such as keys * to list all keys, get <key> to view contents of a string key, zrange <key> 0 -1 to list all the items in a sortedset key. More commands are found at https://redis.io/commands.

Since the render service runs many programs at once using just the docker logs many not be very informative i.e. docker-compose logs render or ./render. Other logs more informative logs are available by using the shell ./render dev shell or docker-compose exec render bash and using cat and tail to view the contents of the logs. The render daemon logs are found at /renderservice/logs/, while the web-server access log is found at /renderservice/access.log.

- Added configuration files for travis, flake8, etc. - Added script for handling docker interactions.

- Queue service now correctly increments retry_count after leasing. - Render service now correctly logs errors during generation. - Fix typos with variables names.

codecov · 2017-09-18T02:14:14Z

Codecov Report

❗ No coverage uploaded for pull request base (master@9ad16c1). Click here to learn what that means.
The diff coverage is 91.03%.

@@            Coverage Diff            @@
##             master       #2   +/-   ##
=========================================
  Coverage          ?   91.04%           
=========================================
  Files             ?       52           
  Lines             ?     2500           
  Branches          ?      225           
=========================================
  Hits              ?     2276           
  Misses            ?      197           
  Partials          ?       27

Impacted Files	Coverage Δ
renderservice/render/webserver/__init__.py	`100% <ø> (ø)`
renderservice/render/daemon/RenderDaemon.py	`0% <0%> (ø)`
renderservice/render/daemon/__main__.py	`0% <0%> (ø)`
renderservice/render/webserver/wsgi.py	`0% <0%> (ø)`
...erservice/render/tests/daemon/test_file_manager.py	`100% <100%> (ø)`
...ervice/render/tests/resources/test_modulo_clock.py	`100% <100%> (ø)`
...er/tests/resources/test_barcode_checksum_poster.py	`100% <100%> (ø)`
renderservice/render/resources/job_badges.py	`100% <100%> (ø)`
...rservice/render/tests/resources/test_job_badges.py	`100% <100%> (ø)`
renderservice/render/resources/arrows.py	`100% <100%> (ø)`
... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9ad16c1...07371c6. Read the comment docs.

hayleycodes · 2017-09-19T22:43:51Z

docs/source/changelog.rst

+The first major step in releasing a open source version of CS Unplugged.
+While some existing content from the classic version of CS Unplugged have yet
+to be adapted into the new format and system, we are releasing this version as
+a sneak peek for teachers.


copy/paste leftovers :P

Mmmm copy pasta.

hayleycodes · 2017-09-21T22:46:20Z

renderservice/render/daemon/FileManager.py

+    the directory is a mounted bucket.
+
+    Since files with the same filepath could belong to different
+    directories only the first found is used, based on the order


files with the same filepath could belong to different directories?

The render system has two locations for static files. A local copy for do don't distribute resources and things for speed, and the cloud bucket system for all other static files. Since the bucket is mounted as another folder in the file-system this class enables the use of both these locations as a single entity.

hayleycodes · 2017-09-21T22:49:17Z

renderservice/render/daemon/FileManager.py

+        self.directories = tuple(args)
+        self.save_directory = kwargs.get("save_directory", None)
+
+    @cachedmethod(lambda cls: type(cls).__cache)


what does this magic do?

This caches the result of the method based on the input arguments. Useful when retrieving things on a external server that are used frequently.

hayleycodes · 2017-09-21T22:58:32Z

renderservice/render/daemon/FileManager.py

+        """Get the full filepath for a given resource.
+
+        Finds the first file that matches the filepath (this means
+        if a folder matches first it is not returned.)


this seems terrible?

I hope the previous comments clarified why this is necessary. If not since there are multiple folders being treated as a single filesystem this is necessary, since there is a chance that there maybe similarly named files in the same apparent location. The class allows ordering of the filesystems to determine which to favour in these situations.

hayleycodes · 2017-09-22T22:16:25Z

renderservice/render/daemon/RenderDaemon.py

+        Tasks will be run to recieve a result, saved if necessary
+        then deleted. Tasks that return a none result or are
+        interupted by an exception will not be deleted and have
+        their lease reduced for another daemon. Tasks that have


what does this mean, "have their lease reduced for another daemon" ?

A task is leased for a specified time from the queue (and remains in the queue) by a daemon. This here means that a daemon is telling the queue that it is finished with it's lease early (but the task should not be removed) so that the queue may lease it out to another daemon. This happens when something goes wrong, like going over the allowed time. When the task is next leased it maybe given more time based on how many times it has been leased before.

hayleycodes · 2017-09-22T22:16:51Z

renderservice/render/daemon/RenderDaemon.py

+        interupted by an exception will not be deleted and have
+        their lease reduced for another daemon. Tasks that have
+        surpassed their retry limit will have a failure result
+        saved if necessary otherwise they will be deleted.


what is deemed as "necessary" and what happens if it is? i.e. do they stay on the queue?

This is changed, they will all have a failure result saved and be deleted.

hayleycodes · 2017-09-22T22:18:01Z

renderservice/render/daemon/RenderDaemon.py

+            else:
+                result = self.handle_retry_limit(task_descriptor)
+
+            # Save out documents


"out documents" ?

Fixed, was probably trying to capture the idea that they are not saved on the current machine.

hayleycodes · 2017-09-22T22:28:27Z

renderservice/render/daemon/ResourceGenerator.py

+    def generate_resource(self, task, resource_generator):
+        """Retrieve page(s) for one copy of resource from resource generator.
+
+        Images are resized to size.


This is what is was on the previous system. Changed now to "resized to fit page".

hayleycodes · 2017-09-22T22:28:49Z

renderservice/render/daemon/ResourceGenerator.py

+
+        Args:
+            task: The specification of file to generate as a dictionary.
+            resource_image_generator: The file generation module.


incorrect arg

hayleycodes · 2017-09-22T22:30:14Z

renderservice/render/daemon/__main__.py

+    """Load and request a daemon to perform an action.
+
+    Do not trust code or the process to be running after this method
+    as certain actions such as 'start' cause the main thread to be


lol what, start causes the main thread to stop?!

Yup, to start a daemon in UNIX you start a process from the current process, then kill it. You are left with a orphan process, which then creates the another process which will be the daemon and then kills itself. Your daemon on it's double orphan process is then adopted by the main system thread. Look it up when you need it.

hayleycodes · 2017-09-22T22:32:54Z

renderservice/render/daemon/utils.py

+    return details
+
+
+def get_recommended_number_of_daemons():


so, this function figures out how many CPUs you can have running on a particular machine?

Yes that is correct.

hayleycodes · 2017-09-22T22:33:08Z

renderservice/render/daemon/utils.py

+        with open('/proc/self/status') as f:
+            m = re.search(r'(?m)^Cpus_allowed:\s*(.*)$', f.read())
+        if m:
+            res = bin(int(m.group(1).replace(',', ''), 16)).count('1')


having trouble processing how this works

I've edited it to make it cleaner, but basically the number of CPUs is stored as a number but that number isn't actually a count but a bitmap of the cores you are allowed to run on. So the number of ones in the bitmap is how many cores are available.

haydenjackson01 added 27 commits September 6, 2017 16:36

Update README with badges.

5a09f0a

Add sphinx documentation.

378d6a1

Add queue service.

ed008bf

Add queue service.

4019f4f

Add docker build scripts and requirements file.

aa46d48

Configuration and render script.

f0439d1

- Added configuration files for travis, flake8, etc. - Added script for handling docker interactions.

Fix travis commands to start render service.

87f7acb

Configure and setup flake8 and coverage for travis.

c36aafd

Add resources that will be mounted externally.

88b0b3d

Fix missing image and adjust pointer.

e56d301

Merge in new resources.

330fdc2

Update and merge in unittests from csu.

3dbe6ec

Merge in updates to resources from csu.

f4971b1

Remove 2 copy testing requirement.

a3a9cae

Fix daemon tests by extending timeout timer.

0ea8e33

Update documentation with some summary information.

95d7abd

Fix errors in consumption pipeline.

5b3d41b

- Queue service now correctly increments retry_count after leasing. - Render service now correctly logs errors during generation. - Fix typos with variables names.

Remove testing entrypoint.

44250e1

Add missing resources, and update treasure hunt.

3ec0138

Structure and bug fixes.

a150b89

Add some documentation around render tasks.

4e99d57

Use tags to separate tasks and store results.

995a09d

Fix if statement.

60e5414

Setup dev test file and host networking.

6bbf5f0

Fix incorrect queue name and lengthen generation time.

c326322

Rename dev script to a sensible name.

f1b116e

Remove incorrect instruction from queue_client header.

35ec627

haydenjackson01 added 2 commits September 18, 2017 14:21

Style fixes.

28bc02e

Style fix.

cdc5954

Add inline documentation to queue_client script.

0d8ee94

haydenjackson01 added the enhancement label Sep 18, 2017

haydenjackson01 self-assigned this Sep 18, 2017

haydenjackson01 requested a review from hayleycodes September 18, 2017 03:44

hayleycodes reviewed Sep 21, 2017

View reviewed changes

hayleycodes reviewed Sep 22, 2017

View reviewed changes

haydenjackson01 added 12 commits September 25, 2017 12:40

Documentation fixes from talk with Hayley.

fe7cd05

Basic tidy based on PR.

47292bb

Fix incorrect scaling of the images in resources.

9bcf529

Add argument and return types to docstrings.

79d6c01

Enable deployment of images to gcloud service.

1e6903e

Clarify error message in queue service.

f5bb667

Python documentation style fix.

6330a94

Fix bug from find and replace error.

13424f5

Remove ports from render as it is running in host mode.

0237999

Configure docker to not use host mode for network.

855a2f1

Fixes issues with mounting bucket in cloud.

936dc00

Final fixes for mounting buckets in cloud.

07371c6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alpha 1 #2

Alpha 1 #2

haydenjackson01 commented Sep 18, 2017

codecov bot commented Sep 18, 2017 •

edited

Loading

hayleycodes Sep 19, 2017

haydenjackson01 Sep 24, 2017

hayleycodes Sep 21, 2017

haydenjackson01 Sep 24, 2017

hayleycodes Sep 21, 2017

haydenjackson01 Sep 24, 2017

hayleycodes Sep 21, 2017

haydenjackson01 Sep 24, 2017

hayleycodes Sep 22, 2017

haydenjackson01 Sep 24, 2017

hayleycodes Sep 22, 2017

haydenjackson01 Sep 24, 2017

hayleycodes Sep 22, 2017

haydenjackson01 Sep 24, 2017

hayleycodes Sep 22, 2017

haydenjackson01 Sep 24, 2017

hayleycodes Sep 22, 2017

hayleycodes Sep 22, 2017

haydenjackson01 Sep 25, 2017

hayleycodes Sep 22, 2017

haydenjackson01 Sep 25, 2017

hayleycodes Sep 22, 2017

haydenjackson01 Sep 25, 2017

Alpha 1 #2

Are you sure you want to change the base?

Alpha 1 #2

Conversation

haydenjackson01 commented Sep 18, 2017

codecov bot commented Sep 18, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 18, 2017 •

edited

Loading