Using a Moodle Yesterday Instance for Self Service Course Repairs and Improved Site Performance

Why might this be a better solution than automated course backups?

The Moodle course backup and restore feature, while a convenient mechanism for moving courses between Moodle sites, is not very efficient. Many also confuse this system with a true disaster recovery solution for Moodle. It is not. The automated course backup facility is best suited for teachers to self-service individual courses when they are accidentally damaged. The Moodle course backup system uses zip files to store individual course backups. Anytime a single file or database entry changes in a course the entire course backup has to be recreated and zipped into a new archive file. It takes a lot of CPU and IO resources to create these files. The zipping process also makes even the most minimal changes in a course invisible to most backup systems. Because of Moodle’s file structure it can also be impractical to exclude these files from the overall the server's backup system. This can have a very negative impact on the backup systems performance and cost.
The yesterday instance script relies on rsync to intelligently backup only the Moodle data files that have changed since the last run, but still allows teacher’s to self service restores for accidentally damaged courses without IT involvement.  It is however worth noting that the generic solution isn’t the most efficient method for database duplication, but for most site’s the database is 1/10th the size of Moodle data so this is generally a good tradeoff.

Important warning

Be sure you have a good backup before running! And be careful about not mixing around the source and yesterdayinstance variables as if you do you can potentially delete your production database. If you have a dev environment you should probably try it out there first. For example I did my testing with a copy of the Moodle installer on my local workstation.

General notes

I have tried to make this very generic so that it will run on most servers with standard unix / unix like tools installed. I did the testing on my Mac work station but should also work on Linux based server. Windows based server would need to have something like Cygwin installed. I have added comments to explain each section. This is intended to be setup to run once per day via cron.

Is it going to nearly double the size of the storage / virtual machine that runs Moodle? 

It depends a bit on the file system sitting behind the VM. Generally speaking we assume it will at least double the storage used. So one assumption about this concept is that the storage used is cheaper than either the labor to manually do restores for teachers, or alternately the extra server resources to run Moodle course backups (which also likely double or more storage use).   Also keep in mind that the yesterday instance is going to have a lot less usage so it could be run on cheaper storage (could also be a separate VM with lesser resources if you want to isolate things more). Additionally a lot of high and mid tier storage solutions offer built-in de-duplication. With dedup in use from the VM perspective double the storage is used, but on the backend side much less than double is being used. And this is a key advantage vs automated disk backups which compress the data into zip files which are a lot less likely to match up with the dedup functionality on the backend.
Your storage solution might also support snapshots which can be used in a similar fashion while not taking as much of a storage hit. However most snapshots are read only, so you would still end up copying files over when it came time to actually use the snapshot to restore a course and database backups can be less reliable with snapshots if not supported by the storage solution.
A more sophisticated version of this solution can use hard-links to keep more than a single copy of the yesterday instance. You could integrate this with something like rsnapshot which is a backup solution that uses rsync and hard links to provide multiple backup copies while saving disk space. Potentially the most minimal footprint for the yesterday instance is if it is actually integrated into the backup solution. At the same time SATA disks are extremely inexpensive these days, so the trade off between engineering and time to optimize the space for a single site, may not be worth the costs versus less then $100 drive.
 

Does this script duplicate the database and the Moodle data files? 

Yes it duplicates both of these as well as the Moodle source code
Does this also create the new yesterday web directory or will that need to be done first manually?
The script will duplicate the folders to the paths listed. As long as the path exists all the way up to the last directory it will probably make the last directory for you. It's up to you to configure the web server if needed to use the new folder. I think in my test case I just made it a subdirectory of the webroot so I didn't have to do an config changes. I would recommend configuring a new folder that is not a subdirectory of the Moodle folder, as this can cause you problems when you try to do upgrades and can also has backup implications. I would then make the new folder a virtual host so it's something like http://yesterday.my_moodle.domain so its something that users can remember easily.
Let me know if you end up using this or a variation. I would value feedback on how the generic version works out for you.

Example Yesterday Script
#!/bin/sh
# Yesterday instance script
# Copyright Jonathan Moore 
# eLearning Consultancy
# Released under GPLv3 license
#
# Script to copy a Moodle site to yesterday instance on the same server
# Install script into a non-web folder, run daily from cron during off hours
# A more sophisticated version could use data from a backup at least 12-24 hours old so that 
# 'yesterday' instance always lags the site by a set interval. This basic example is meant to be run at night during
# off hours to provide a safety net for accidental course edits / errors made during the next day.
# This is intended as an alternative to using Moodle's automated course backups which are very I/O and CPU intensive
# Requires a working Moodle environment running mysql, sed, and rsync (all common for unix / unix like environments



# Fill in locations and variables here for your site
# Set because my dev environment doesn't set the mysql path
MYSQL_PATH=/Applications/MAMP/Library/bin
DB_USER=root
# Note that using .my.cnf file with 0600 permissions is a more secure option for DB password, using as variable here for simplicity of example
DB_PASS='root'
DB_HOST=localhost
SOURCE_DB=moodle27
YESTERDAY_DB=moodle27_yesterday
SOURCE_MOODLE=/Applications/MAMP/htdocs/moodle27
SOURCE_MOODLEDATA=/Applications/MAMP/moodledata_27
YESTERDAY_MOODLE=/Applications/MAMP/htdocs/moodle27_yesterday
YESTERDAY_MOODLEDATA=/Applications/MAMP/moodledata_27_yesterday
URL=http://localhost:8888/moodle27
YESTERDAY_URL=http://localhost:8888/moodle27_yesterday

# Make new DB copy (maybe use one liner, may need to delete old copy of DB, or can use mysqldump into Moodle data)
# and Update site URL
echo "Dropping yesterday database"
echo "DROP DATABASE IF EXISTS ${YESTERDAY_DB}; CREATE DATABASE ${YESTERDAY_DB};" | $MYSQL_PATH/mysql  -u $DB_USER -p${DB_PASS}
echo "Copying database"
# Note mysqldump isn't the most efficient method to duplicate DB, but is a very general approach and works well as long as DB isn't too large
# Sed is used here to re-write site URL inline while database is copied
$MYSQL_PATH/mysqldump -u ${DB_USER} -p${DB_PASS} $SOURCE_DB | sed -e "s|$URL|$YESTERDAY_URL|g" | $MYSQL_PATH/mysql -u $DB_USER -p${DB_PASS} $YESTERDAY_DB


# Rsync (with delete to prevent orphaned files) Moodle source yesterday source
echo "Copying source code"
rsync -arp $SOURCE_MOODLE/* $YESTERDAY_MOODLE --delete

# Set correct config.php values for yesterday instance
echo "Updating config.php to path of yesterday instance"
sed -i -e "s|$SOURCE_MOODLE|$YESTERDAY_MOODLE|g" $YESTERDAY_MOODLE/config.php 
sed -i -e "s|$SOURCE_MOODLEDATA|$YESTERDAY_MOODLEDATA|g" $YESTERDAY_MOODLE/config.php
sed -i -e "s/$SOURCE_DB/$YESTERDAY_DB/g" $YESTERDAY_MOODLE/config.php

# Rsync (with delete to prevent orphaned files) Moodledata to yesterday Moodledata
echo "Copying Moodle data"
rsync -arp $SOURCE_MOODLEDATA/* $YESTERDAY_MOODLEDATA --delete

# Delete yesterday moodledata cache to weird errors in Moodle UI (similar to running purge caches from UI)
echo "Clearing cache"
rm -rf $SOURCE_MOODLEDATA/cache/*

Moodle Configurable Reports Plugin All Variables and Filters

Configurable Reports is a popular plugin for Moodle that allows administrators to create custom reports for Moodle and make them available to their end users. In working with this plugin I have noticed that it can be difficult to track down all the variables and filters you can use in creating reports. Below are a list of all of the options.

Variables

%%CATEGORYID%%

%%COURSEID%%

%%CURRENTUSER%%

%%DEBUG%%

%%FILTER_VAR%%

%%STARTTIME%%','%%ENDTIME%%

%%USERID%%

%%WWWROOT%%

Filters

%%FILTER_CATEGORIES:

%%FILTER_COURSEMODULEID:

%%FILTER_COURSEMODULEFIELDS:

%%FILTER_COURSEMODULE:

%%FILTER_COURSES:

%%FILTER_COURSEENROLLEDSTUDENTS:

%%FILTER_USERS:

%%FILTER_ROLE:

%%FILTER_SEARCHTEXT:

%%FILTER_SEMESTER:

%%FILTER_STARTTIME:

%%FILTER_ENDTIME:

%%FILTER_SUBCATEGORIES:

%%FILTER_COURSEUSER:

%%FILTER_SYSTEMUSER:

%%FILTER_YEARHEBREW:

%%FILTER_YEARNUMERIC:

Using Variables

Variables can be used anywhere inside of your SQL query. 

Using Filters

When using a filter you need at least one static WHERE condition then you list one or more filters. For example

WHERE 1

%%FILTER_COURSES:c.id%%

You can technically add multiple filters, but once a user applies one filter to a report all filters for report are applied. So it can be a little tricky to have multiple filters on the same report. For example if you add a course and date filter to a report, then if you just want to filter by course you have to make sure your date filters are set to date range that covers all courses.

Many of the filters are hard code to be drop down menus which are sorted by database id number instead of alphabetically. This can be problematic if you have a large number of users, courses, or categories. However, you can use one search text filter per report, which can be used to work around the issue in some cases.

Many filters require a logical operator to specify how the filter works. For example

%%FILTER_STARTTIME:l.TIME:>%% %%FILTER_ENDTIME:l.TIME:<%%

Moodle 2.9 Top 5 New Features for Admins

Moodle 2.9 was release earlier this week. Here are my top five new improvements for administrators

Allow backup and restore on the front page  http://tracker.moodle.org/browse/MDL-31500 Backup report again links to the individual course backup summaries  http://tracker.moodle.org/browse/MDL-30937 Assign global roles using CSV upload http://tracker.moodle.org/browse/MDL-15187 CLI version of admin/replace.php http://tracker.moodle.org/browse/MDL-25763 Configure temporary file deletion, to avoid moodledata/temp/backup filling up the filesystem https://tracker.moodle.org/browse/MDL-32547

Improving Moodle Web Services Usability

The Good

Moodle, as of the version 2 release, includes a built-in Web Services (WS) API that supports a variety of WS protocols including SOAP, REST, and XML-RPC. Since the API is built-in it's generally available whether the site is self-hosted or commercially hosted. The API is easily extensible to add greater functionality. The API provides a single standard method for WS for Moodle simplifying what was a more complex patchwork of third-party solutions.

The Bad

In my experience with LMS development, WS libraries are typically used to provide realtime integrations between specialized third-party software such as a student management system or a content repository. The core Moodle WS library is poorly suited to this use case because it doesn't provide functions to find specific objects in the Moodle database using an arbitrary matching field. This is an unfortunate oversight given Moodle's otherwise excellent support for this methodology with its consistent use of the ID Number field throughout its system. Without being able to do these targeted matches against a value known by the third-party software, one has to rely on function calls to get the entire list of objects (example full user list). This is simply impractical from a performance perspective for any realtime system of scale.

We ran across this problem during a recent project creating an integration between a membership portal and a Moodle 2.7 site. 

We did some research and found that:

1. There is a related improvement request in the Moodle development backlog see tracker https://tracker.moodle.org/browse/MDL-26886.
2. Community member, Craig Baker,  submitted a conceptual solution and prototype code in the comments to address the issue. Craigs tracker profile https://tracker.moodle.org/secure/ViewProfile.jspa?name=cbaker118.

The Fix

Using the extensible nature of the WS API, we created a local plugin to allow clean and easy installation of our additions. We based the new calls on the conceptual solution posted by Craig. Many thanks to Craig for his contribution. 

You can download a copy of the eLC WS library from git at

https://github.com/moorejon/moodle-local_elcws

Be aware this was an emergency solution that to date has only been used on one Moodle site. As always with contributed code you should test for your particular use case. Please share your experiences with the code if you decide to try it and whether there are any improvements you would like to see.

This solution only really deals with allowing enrollment of a user into a course without knowing the users internal database ID number. Additional functions are needed to fully address the issue across all aspects of the API. As we do more WS related projects we plan to add additional functions.

Use apache mod_deflate to make your Moodle website faster

Mod_deflate is an Apache web server module that uses compression to reduce the bandwidth needed to send web pages between the server and the end user's computer.

See http://httpd.apache.org/docs/2.4/mod/mod_deflate.html

Modern web applications such as Moodle send a lot of text from HTML code, to CSS styles, to javascript, all of which can achieve high compression rates.

So how does reducing bandwidth usage make your website faster? It's actually a bit of a trick. The website itself probably isn't really faster in terms of the number of users or pages it has the capacity to generate. Each page probably will take about the same build time as before on the server. In fact if your server's undersized in terms of CPU resources it might even reduce the server's overall capacity. But for the end user the perceived responsiveness of the website is significantly increased. This is because it takes a lot less time for the page to be transported from the server to the user's computer. This can have a significant impact on user satisfaction and is relatively easy to setup for knowledgable administrators. On a recent engagement, I actually saw a 2-3 second improvement in page load times for the end user, even while accessing from the local campus network. Needless to say the Moodle administrator for this site was very happy with the improvement and you will be too!

Enabling Moodle course and activity completions en masse for Moodle 2.7

This week I am going to talk about a method to enable course completions en-masse across a library of existing Moodle courses.

Stupid Moodle Tricks!

This is my pet name for these little tips, #stupidmoodletrick anyone? Think Letterman's stupid human tricks (video below).  These are generally one-off tips, tricks, hacks to manipulate Moodle, generally as a time saver or to accomplish something Moodle isn't generally known to do.

So last week, a client asked me if I could help save them some time in converting some old Moodle 1.9 courses. Actually the more honest version is my lovely and dear wife, Michelle, volunteered that I could totally save the client hours of work by "changing a few database settings." And it turned out she was right!

The following assumes you a have already enabled course completion tracking in your site administration settings. Be aware all of these changes are accomplished by running the below queries directly on your Moodle database. As always when making these sort of changes, be sure you have a good backup and that you know how to recover it to make a working site. Things can and do go wrong and in unexpected ways. I also recommend you use a test site first to become familiar with the technique.

This method was tested on Moodle version 2.7.1+, and 2.7.2+. As with any method that directly interacts with the Moodle database, things almost certainly will change with new versions. It probably will also work with older Moodle 2 versions, but again test first.

Enable course completions on all courses

First we need to enable completion tracking for all courses. 

update mdl_course set enablecompletion = 1 where 1;

Turn on activity completion for all course modules (ie activities and resources)

This query enables the setting for all course modules across the site. It sets 'Completion Tracking' to 'Show activity as completed when conditions are met.' It also sets 'Require view' as the criteria, as illustrated in the screenshot below.

update mdl_course_modules set completion = 2, completionview = 1 where 1;

In contrast, if you wanted to set the activity completion to be marked manually by the student, you would run the following query instead.

update mdl_course_modules set completion = 1 where 1;

Which results in the following settings

Limiting which type of modules are set

Let's say we only want to enable completion tracking for all quiz modules. We would first need to have a look at the mdl_modules table and find the id assigned to the quiz module. We can use the following query.

SELECT * from mdl_modules;

Results from my test site are shown below.

We can see that the quiz module id is set to 25. Be aware this will be different for different sites. For example, I tried this on another test site and the quiz id was set to 16.

So now we can create a query to only modify quiz modules.

update mdl_course_modules set completion = 2, completionview = 1 where module=25;

Turn on grade required completion element for all quiz modules

The completion grade requirement is stored by setting a value of 0 in the completiongradeitemnumber field. Knowing this we can set the value and combined with the where clause above to limit to only quizzes. This can be a very useful technique since many compliance courses will use a quiz grade as their completion criteria.

update mdl_course_modules set completiongradeitemnumber=0 where module=25;

Note that this method only changes the setting to require a grade. It doesn't require a specific grade which need additional manual setup in the Moodle grade book. I think setting a specific grade is considerably less trivial. Let me know in the comments if you have a query / method to do this.

The overall result of applying this technique for the requesting client was to save 800 course setting edits, and about 2,500 activity setting edits which probably would have taken several days to a week to do manually.

A note about method

When developing a #stupidmoodletrick, I use a throw away test site. I usually just use the Moodle all-in-one installer to create the site. I then setup the test site with whatever test users / courses, and settings which are needed for the scenario. I then have a look at the database. The all-in-one installer includes phpMyAdmin which is an easy way to have a look at the database structure in a graphical environment. Moodle is good about using descriptive names for its tables and fields. I try to identify which table I think will be used to save the information and which are likely fields. Then, I go into Moodle and manually change the setting manually. After this, I go back to the database and look at the changes made and attempt to determine what is getting set.

Next, I attempt to manually edit the database using the information I have learned. I make the edit and then I purge all caches in Moodle. Finally, I access Moodle again to confirm that I have achieved the desired result. I also look to see if anything appears to be broken. I repeat the process until I have a working method that doesn't break anything.

Note a more thorough method is to trace the actual Moodle queries in the source code, but I find I normally don't have to do this. In particular think of settings that may impact more than one table (related tables and fields), we want to avoid introducing later hard to find problems.

Multi-language certificate tips for Moodle

I recently completed a project to provide a customized certificate including course completion elements for a course to be delivered in 14 languages. It was a great learning experience. This post shares a few tips and tricks learned along the way.

Some Context

This post is targeted toward any developers (or soon to be developers) that need to produce multi-language certificates of completion for Moodle. The Moodle certificate module delivers certificates by generating PDF files. I had assumed that PDFs benefited from the same gains that web applications have with the advent of unicode character sets. The reality is more nuanced when it comes to multi-language documents.

Moodle's PDF font support is broken, but soon to be fixed

During the Moodle 2 development process, Moodle HQ made a sensible decision to remove some of the less commonly used PDF fonts to reduce the size of the Moodle source code installation. This especially made sense given that there was little or no core functionality that used the TCPDF library that contained the fonts and the fonts used a lot of disk space. More recently the TCPDF library version was updated for the Moodle 2.6 release. Changes in the underlying library broke Moodle's support for installing additional fonts. See https://tracker.moodle.org/browse/MDL-47333 for details (fix slated for release around November 14th M2.6.6 and M2.7.3 releases). 

The idea is that you can install the full set of PDF fonts from the TCPDF project by

1. Downloading fonts file from project http://sourceforge.net/projects/tcpdf/files/
2. Unzip and place the fonts folder into your Moodle dataroot folder
3. Reference additional fonts by creating a custom certificate type. See https://docs.moodle.org/19/en/Certificate_module#Customize_format for creating custom certificate type. Although Moodle 1.9 specific documentation, the Moodle 2.x version of the document doesn't have the details for making a custom type and the basic instructions are still the same.

This combination of changes makes it difficult to support some languages with certificates. Because of the bug, you will need to either replace the moodle/lib/tcpdf/fonts folder with the complete font folder download, or apply the patch / work around from the above Moodle tracker. This is especially important if you need to produce a certificate using any of the Asian languages such as Chinese, Japanese, or Korean (also referred to as CJK, or CID-0 fonts).

No universal free unicode font for all languages

For our project, we were hoping to make one custom certificate type that would work with any of the 14 language versions of the course. This would allow us to use Moodle's built-in language pack and editing capabilities to provide the correct unicode text for each language. What we found in practice is there was no single free font that had all 40k+ characters needed for universal coverage.

We also found that some fonts would have the characters for a language in one style but not another. For example we used the freeserif and freesans fonts (included in Moodle) for the prototype. We found that with a bold font style we couldn't output Hindi characters.

With some research, we did find a commercial font ($165 license fee) that has all the characters needed to make universal certificate types, but the client was not sure if the licensing terms made it legal to use in a web application such as this and opted for creating multiple certificate types.

Related links

• http://www.tcpdf.org/fonts.php
• http://stackoverflow.com/questions/8737772/tcpdf-encode-chinese-character
• http://www.fonts.com/font/ascender/arial-unicode

Asian fonts are special

CID-0 fonts are "non-embeddable" fonts. The idea is that they are supposed to be provided by the PDF reader. They are a bit like core fonts in this regard, but unlike core fonts which are generally included with your PDF reader download, these fonts need to be downloaded separately. The TCPF configuration for each of these fonts seems to reference the same basic font with an additional code to specify the specific language and character set. In practice, this meant we had to generate 5 custom certificate types to get complete coverage for the 14 language course.

• Latin, cyrillic, arabic cert type (fonts freesans, freeserif without use of bold style)
• Traditional Chinese cert type (font cid0ct)
• Simplified Chinese cert type (font cid0cs)
• Korean cert type (font cid0kr)
• Japanese cert type (font cid0jp)

If you are an Adobe Acrobat user, you will automatically be prompted to download the CID-0 fonts if you open a PDF that uses them. This brings us to our final tip.

Mac preview gotchas

My development workstation is a Mac and by default it uses the Preview application to open PDF files. We spent a lot of time trying to figure out why our CID-0 based certificates were blank. After trying the same PDF files on a Windows workstation and finding that they worked correctly, we finally determined that Preview will not prompt for the missing fonts. We also found that installing Adobe Acrobat, opening the files, and downloading the CID-0 fonts package did not fix the issue for Preview which apparently uses a different font path. I have yet to determine a procedure to fix the issue for Preview, but it's pretty easy to work around once the root cause is discovered (right click on PDF and choose open with Acrobat, or change the default PDF viewing application).