Wednesday, October 29, 2014

Multi-language certificate tips for Moodle

I recently completed a project to provide a customized certificate including course completion elements for a course to be delivered in 14 languages. It was a great learning experience. This post shares a few tips and tricks learned along the way.

Some Context

This post is targeted toward any developers (or soon to be developers) that need to produce multi-language certificates of completion for Moodle. The Moodle certificate module delivers certificates by generating PDF files. I had assumed that PDFs benefited from the same gains that web applications have with the advent of unicode character sets. The reality is more nuanced when it comes to multi-language documents.

Moodle's PDF font support is broken, but soon to be fixed

During the Moodle 2 development process, Moodle HQ made a sensible decision to remove some of the less commonly used PDF fonts to reduce the size of the Moodle source code installation. This especially made sense given that there was little or no core functionality that used the TCPDF library that contained the fonts and the fonts used a lot of disk space. More recently the TCPDF library version was updated for the Moodle 2.6 release. Changes in the underlying library broke Moodle's support for installing additional fonts. See https://tracker.moodle.org/browse/MDL-47333 for details (fix slated for release around November 14th M2.6.6 and M2.7.3 releases). 

The idea is that you can install the full set of PDF fonts from the TCPDF project by
  1. Downloading fonts file from project http://sourceforge.net/projects/tcpdf/files/
  2. Unzip and place the fonts folder into your Moodle dataroot folder
  3. Reference additional fonts by creating a custom certificate type. See https://docs.moodle.org/19/en/Certificate_module#Customize_format for creating custom certificate type. Although Moodle 1.9 specific documentation, the Moodle 2.x version of the document doesn't have the details for making a custom type and the basic instructions are still the same.
This combination of changes makes it difficult to support some languages with certificates. Because of the bug, you will need to either replace the moodle/lib/tcpdf/fonts folder with the complete font folder download, or apply the patch / work around from the above Moodle tracker. This is especially important if you need to produce a certificate using any of the Asian languages such as Chinese, Japanese, or Korean (also referred to as CJK, or CID-0 fonts).

No universal free unicode font for all languages

For our project, we were hoping to make one custom certificate type that would work with any of the 14 language versions of the course. This would allow us to use Moodle's built-in language pack and editing capabilities to provide the correct unicode text for each language. What we found in practice is there was no single free font that had all 40k+ characters needed for universal coverage.

We also found that some fonts would have the characters for a language in one style but not another. For example we used the freeserif and freesans fonts (included in Moodle) for the prototype. We found that with a bold font style we couldn't output Hindi characters.

With some research, we did find a commercial font ($165 license fee) that has all the characters needed to make universal certificate types, but the client was not sure if the licensing terms made it legal to use in a web application such as this and opted for creating multiple certificate types.

Related links
  • http://www.tcpdf.org/fonts.php
  • http://stackoverflow.com/questions/8737772/tcpdf-encode-chinese-character
  • http://www.fonts.com/font/ascender/arial-unicode

Asian fonts are special

CID-0 fonts are "non-embeddable" fonts. The idea is that they are supposed to be provided by the PDF reader. They are a bit like core fonts in this regard, but unlike core fonts which are generally included with your PDF reader download, these fonts need to be downloaded separately. The TCPF configuration for each of these fonts seems to reference the same basic font with an additional code to specify the specific language and character set. In practice, this meant we had to generate 5 custom certificate types to get complete coverage for the 14 language course.
  • Latin, cyrillic, arabic cert type (fonts freesans, freeserif without use of bold style)
  • Traditional Chinese cert type (font cid0ct)
  • Simplified Chinese cert type (font cid0cs)
  • Korean cert type (font cid0kr)
  • Japanese cert type (font cid0jp)
If you are an Adobe Acrobat user, you will automatically be prompted to download the CID-0 fonts if you open a PDF that uses them. This brings us to our final tip.

Mac preview gotchas

My development workstation is a Mac and by default it uses the Preview application to open PDF files. We spent a lot of time trying to figure out why our CID-0 based certificates were blank. After trying the same PDF files on a Windows workstation and finding that they worked correctly, we finally determined that Preview will not prompt for the missing fonts. We also found that installing Adobe Acrobat, opening the files, and downloading the CID-0 fonts package did not fix the issue for Preview which apparently uses a different font path. I have yet to determine a procedure to fix the issue for Preview, but it's pretty easy to work around once the root cause is discovered (right click on PDF and choose open with Acrobat, or change the default PDF viewing application).

Friday, July 11, 2014

PHP 5.5 and OpCache for Moodle / Moodle 2.7 on RHEL / CentOS 5

The Problem

On a recent consulting engagement, I was tasked with helping a school eliminate a severe performance issue on its Moode site. I determined the primary solution was to add a PHP accelerator such as APC or OPCache.

Red Hat still supports RHEL 5 for security updates, but the default PHP installation is rapidly becoming unusable for current versions of Moodle. The server's administrators had manually upgraded from the PHP 5.1 version included by Red Hat to a third party PHP 5.3 build. While this allowed them to install a newer version of Moodle, it also ruined the sites performance. The third party build didn't include a PHP accelerator. Because the site was in active use there wasn't enough time to upgrade the entire operating system.

After discusion, with the client, we decided the best option forward was to upgrade to PHP 5.5 and use OPCache for PHP acceleration. This had the additional benefit of making the server ready for Moodle 2.7 and beyond. Here is how we did it.

The solution

We determined we could get a high quality PHP 5.5 build from the Remi repository (http://rpms.famillecollet.com/) that would continue to receive security updates. OPCache is included with PHP 5.5 and beyond.

Setup Remi repository (from cli)

wget http://download.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
rpm -ivh epel-release-5-4.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-5.rpm
rpm -Uvh remi-release-5*.rpm

See http://linuxdrops.com/how-to-add-epel-and-remi-repository-to-centos/

Remove old PHP (from cli)

yum remove php php-soap php-intl php-cli php-mbstring php-common php-pdo php-gd php-xml php-ldap php-mysql php-xmlrpc

Install new PHP (from cli)

yum --enablerepo=remi,remi-php55 install php httpd php-soap php-intl php-cli php-mbstring php-common php-pdo php-gd php-xml php-ldap php-mysql php-xmlrpc php-opcache

See http://www.if-not-true-then-false.com/2010/install-apache-php-on-fedora-centos-red-hat-rhel/

OpCache Configuration

See the following Moodle docs page for Moodle specific OPCache settings http://docs.moodle.org/26/en/OPcache

Risks

Some risks to consider before upgrading

Upgrading PHP can break older PHP applications

Make sure that your version of Moodle will work with a new PHP including all third party plugins. Also be sure any other PHP web applications are your server support the new version.

The new install may fail

There is also a chance that if you do something wrong in the repository setup or have an unusual configuration that you won't be able to successfully install the new version after removing your old PHP, so plan accordingly!

Missing PHP libraries

Each PHP build is different, not every build includes files for every library. Make sure the library you need are in the new version.

Friday, April 11, 2014

How Heartbleed is Breaking Your Moodle Security and How to Fix It!

What is Heartbleed?

Discussion of the Heartbleed vulnerability has been burning up the Internet this past week, so I won’t go into great detail about its details. Good articles here and here.

The High Points of Heartbleed

  • A vulnerability in OpenSSL
  • Impacts Apache and Ngnx web servers
  • Present for over 2 years
  • Exploit leaves no trace in server logs
  • Allows theft of security certificates, user credentials, and other site data
  • Thought to impact between 15%-33% of web-servers worldwide


How to Know if You Are Impacted

Chromebleed Plugin for Chrome

Written by security researcher Jamie Hoyle, not only does this provide an easy way to check for Heartbleed it also provides you with protection against unknowingly logging into sites with the vulnerability.


Heartbleed Checking Website

If you are not a Chrome user, Filippo Valsorda has posted a Heartbleed checking website. Just click on the URL and enter the URL of your Moodle site to check if you have the vulnerability. “




Source:


What It Means for Moodle Admins

If you have the Heartbleed vulnerability what you can’t know is if its been exploited by an outside party. Here are the overall steps you need to take to deal with the situation:


  1. Patch your webserver to use the new OpenSSL 1.0.1g release or later
  2. Generate a new SSL key, install, and revoke old key via your Certificate Authority
  3. Confirm you are no longer vulnerable
  4. Reset your users passwords (only after 1-3)


Steps one through three are absolutely required to prevent future leakage of data from your site. This is especially important now that the vulnerability is widely known.


Step four is the subject of a lot of discussion because of the challenges of resetting so many user accounts and without knowing if user data has actually been compromised. The longer you wait to patch your site the more likely that bad actors are harvesting your site’s data. Ask yourself why did I install SSL in the first place? This will help answer how important it is to force a reset of passwords. Most major cloud providers are recommending to users that they reset their passwords.


Here are some reasons to consider in favor of resetting
  • You have proprietary or sensitive business information stored in your Moodle
  • Your courses involve discussions of private or controversial topics that users wouldn’t want known outside of their classmates
  • Your user profiles contain private information
  • Your Moodle site is tied to your enterprise authentication system (LDAP, MS Active Directory, etc)
  • You’re concerned about government(s) spying on your site, or the users of your site might be in danger if the local government becomes aware of their participation in your site
  • You want to help protect your users who probably are reusing the same passwords across multiple services


How To Force Your Moodle Users to Reset Passwords

Luckily Moodle provides a built-in function which will allow you to force all your users to reset their passwords. It’s called Bulk user actions and it allows an administrator to select a group of users and apply a change across all of them. Remember that you need to do steps 1-3 before resetting passwords will help you!


Login as an administrator to your Moodle site. Expand the Site Administration menu and select Users. Under Users you will find the Bulk User actions option. Users -> Bulk User actions. Click the “Select all users” button. From pull down labelled “With selected users…”   select the “Force password change” option and you're done! Users will be asked to reset their passwords on their next login.

bulk password reset.png

Friday, March 8, 2013

How to list the largest files in a Moodle 2.x site

Earlier this week, I was working with a client whose Moodle server was full. They asked me to help figure out why it was full. This can be a challenging question when using Moodle 2.x due to the fact that files are saved in a single hashed space. In this client's case, 6 SCORM files were using 3/5 of all their storage. Here is the simple query I developed to create the list showing the original filename and path, the storage used in megabytes, the content hash (ie filename used in moodle data folder), and which component of Moodle owns the file. The output is sorted by largest files first with a limit of the 50 largest files.


SELECT DISTINCT filename, filepath, FORMAT( (filesize /1024 /1024), 1 ) AS filesize_MB, contenthash, component
FROM  `mdl_files` 
ORDER BY filesize DESC 
LIMIT 0 , 50


If you are a Remote-Learner hosting client you can run this query via your phpMyAdmin link contained in your customer kit, or request via our support portal.

During this research I also found that its is possible for a file to become orphaned. In this situation the file is still stored in the moodle data folder, but is no longer referenced in the mdl_files table of the database. Look for a future post on how to track down orphaned files.

Monday, October 8, 2012

Why Your Moodle Site is Slow: Five Simple Settings


Here are some simple settings that any Moodle administrator can change from the Site Administration menu that will help to improve his/her Moodle site's performance. Some of these changes come at the cost of functionality, so sometimes you have to pick between a particular feature and performance (or a faster server), but often these settings are just enabled out of curiosity with no discernible reward other than a slower site.

Automated backup setup

Disable Automated backup setup for courses by setting Active to disabled in the pull down menu. The Moodle course backup and restore feature, while a convenient mechanism for moving courses between Moodle sites, is not very efficient. As the number of courses and the size of the courses on your site increase, the scheduled course backup process takes longer and longer to complete. This can cause extreme site slowdowns while it's running. Additionally, it makes it harder for your server's backup system to complete its own nightly backups.
Site administration / ▶ Courses / ▶ Backups / ▶ Automated backup setup

Many users mistakenly view this as a way to restore their Moodle site in the event of a disaster. While it's possible to use these course backups to restore most of a site, it's not ideal or guaranteed to work when you need it. Disaster recovery backups should be performed outside of the scheduled course backup feature.

Statistics

Make sure Enable statistics is unchecked. Many administrators discover this setting and turn it on simply to see what it does. A common scenario is that a site will be performing well for a year or more and then suddenly experience problems when this setting is enabled. This setting includes options for how long to allow the statistics gathering to run in one session and how far back to process. If you must run statistics make sure to limit how long you allow the process to run. I recommend letting it run no more than 2 hours per session.
Site administration / ▶ Advanced features
Upcoming versions of Moodle may have an improved version of statistics that should reduce the load on the server's database by several orders of magnitude. So if you really need this feature, keep your eyes on MDL-30643.

Theme designer mode

Disable Theme designer mode on production Moodle sites. Theme designer mode is a great Moodle version 2 feature, if you are updating your theme and want to quickly see the impact of changes. It does this by having the Moodle server tell web browsers to not cache any of the images, css, and javascript that make up your theme. When this feature is enabled, every page view for every user must download every media object directly from the Moodle server. This makes viewing pages very very slow and greatly increases the load on the server. Administrators often enabled this to update their theme and then forget to turn it off. The best practice is to develop your theme on a testing site and then move it over to production only after it has been completed.
Site administration / ▶ Appearance / ▶ Themes / ▶ Theme settings

Cleanup

Set Keep logs for and Grade history lifetime to 365 days or less. By default Moodle will keep both your logs and grade history forever. For long established sites, these logs start to become the majority of the database. I have seen cases where log tables consumed 90% or more of the database. As a general rule the more of your database that fits into your server RAM the faster your database will perform. By limiting the size of these logs, you can make your site much faster.

Site administration / ▶ Server / ▶ Cleanup
Note: If you have a really large log, it may make your site even slower while the cleanup runs. Consider changing this setting during a low-use time or during a maintenance window. In extreme cases, it can take several hours for the cleanup query to complete the first time.

Session Handling

Disable Use database for session information by unchecking the box. Starting with Moodle version 2 and above the default setting is to have database sessions enabled. In our infrastructure, we have seen much better performance using file based sessions rather than having the database handle them. This is dependent on the server setup, but it's worth disabling to see if your performance improves.

Site administration / ▶ Server / ▶ Session handling

Pro-tips 

Below are some additional tips that take a bit more effort, but provide additional benefits related to ensuring great performance for your Moodle site.

Running multiple simultaneous manual course backups can sometimes cause problems. Avoid creating situations where many course backup / restores need to run at the same time. This is a common problem for the start of a new school term. Often instructors will be asked to duplicate their prior courses for the new term. Moodle doesn't provide any internal mechanism to queue backup requests so that they don't overwhelm your server. If enough instructors are doing this at the same time, it can make a Moodle site nearly unusable.

While all of these settings can be accessed from the Moodle interface, if you want to really make sure none of your site administrators accidentally sets one of these incorrectly, enter the desired value in your config.php file so that it cannot be enabled accidentally.

Consider starting with a clean install of Moodle once a year to have the smallest database footprint and ensure the best performance for your site.



Thursday, June 10, 2010

The Little Moodle Moot that Could

I just returned from attending the Oklahoma Moodle Moot. This was my third time attending and it reinforced with me how consistently and well operated this conference is. I helped organize the first three US Moodle Moots and I have been lucky enough to travel internationally to attend both the UK and Canadian Moots. My feeling is the OK Moot is under appreciated and an incredible value at only $99 for the main two-day conference and $75 for the one-day pre-conference. Matt Campbell and his team (Sarah Carper and Tim Loughmiller from Metro Technology Centers, and Ed Hatch from Moore Norman Technology Center) did an excellent job organizing the Moot.

What they do well:
  • Pre-conference
  • Network access
  • Facilities
  • Food
  • Presenter amenities
  • Breadth of presentations
Every year this conference goes off like clockwork. They provide a great one day pre-conference with both in depth instructional and admin training. The network access is always rock solid, due to the venue being located at a vocational technical center geared out to take much more usage than the Moot needs. The rooms are of ample size and have the latest technology. The ball room was outfitted with four giant projection screens projecting on opposing walls; there was no bad seat in the house. The food is always well above average and this year was no exception. I was especially a fan of the yogurt and fresh fruit  snack on the afternoon of the first day. This year the conference added a presenters' lounge with snacks and beverages. I thought this was a really nice perk for the presenters that few other conferences offer. Matt even paid out of his own pocket to take the presenters to dinner (thanks Matt!). The conference offered 5 presentation strands: Business, Curriculum, Hands-on, Technical, and Moodle 2.0. The Business strand was a nice addition this year and I enjoyed getting to see some of the new features of Moodle 2.0 in action. This was further bolstered by the presence of Helen Foster, the Moodle Community Manager, who was able to take direct feedback back to the Moodle development team. If you are shopping for a conference for next summer, give this Moot a close look.

Upcoming posts will include interviews with some of the attendees including Helen Foster.

Links:
Oklohama Moot Official Site
Oklohoma Moot Moodle 2.0 Test Site
Oklahoma Moot Experts Panel video

Thursday, May 27, 2010

A Year of Writing Moodle 1.9 Extension Development - Part 4

Part 4: Post Writing Production
This is the final of a four part series on my experiences writing my first book, Moodle 1.9 Extension Development. The book was co-authored by Mike Churchward. The four parts are detailed below:

  • Part 1: Getting a book deal
  • Part 2: Writing the first half
  • Part 3: Writing the last half
  • Part 4: Post writing production

I lied to myself and I knew I was doing it while I was doing it. I told myself, "if I can just finish the writing, the time commitment will go down and my life will become somewhat normal again." I had a suspicion that this wasn't the case, but it did help me get through that last push during the holidays. I think in fact that I ended up spending more time per week in editing and reviewing than during the writing. Part of this is probably due to the fact that I ended up writing some new sections from the feedback during editing. I could have said no to this, but I really wanted to have a quality book when we were done.

Technical and editorial review
We sometimes received notes just from Packt editors and sometimes we received a combined set of notes from both the editors and our technical reviewers. We got many requests to clarify small sections of text, to add new sections, and for more screen shots. An interesting note on the publishing process, while the author knows a bit of the economics of the book in terms of target page counts, etc., technical reviewers are just given a book description and the chapters. This gives a very different perspective and made it really clear that 250+ pages covers the core concepts, but that we could easily write 500 pages to make a comprehensive work on Moodle development.

Kudos to Anthony Burrow for doing such a great job with the technical review. I don't think I know a nicer, more generous person than Anthony. Anthony you are a kind soul!

Most chapters ended up with 2-3 drafts before being submitted for copy editing. Some of the early chapters had as many as 5 or 6 drafts. Packt has a very specific naming convention for drafts that helped keep track.

This process took about 2 months.

Too many pages!
Midway through the process our editor noticed that every chapter we submitted was growing in size from the original draft. We had quickly burnt through an extra 25 approved pages and needed to cut 6 pages. On top of this, I still had requests from an unedited chapter to add at least one major new section and more screen shots. It was decided the only solution was to cut the last chapter from the print version of the book and make it available for free download. This was a confusing time, but eventually we got through it and ended up with a better book as a result. Packt encouraged us to add anything we needed for the book to be really great. In fact, at the last minute before publication, our editor was able to get approval for the extra pages to include the final chapter in the printed book.

Code Review / Copy Edit
The final stage of production was copy editing and code review. We worked with a couple of great guys named Chris and Hitesh.  They were equally conversant with chasing dangling participles as they were at parsing PHP code! What a cool and unusual combination of skills. This team performed fixes to word usage, punctuation, code formatting, and installed every bit of code from the book and tested it. I actually received back some notes like "we had to edit line X of the sample to get our test environment to match your screen shot." I was simply blown away.

This part of the process went pretty quickly, typically 45 minutes to an hour per chapter. The exception was one code sample I decided needed to be re-written to fix an issue with the user interface. Major thanks to this editorial team. They really made us look good.





It's Done!
It's been a long and challenging year, but a growing year. I would recommend to anyone who, like myself, always wanted to write a book, to take the opportunity if presented. Or even better, decide today to make your own opportunity.