Automated Checks

Automated checks are a way for a course to run automated verifications on student submissions. For instance it can:

  • build the student code
  • run a linter (e.g. hlint) to make sure that the code is minimally readable
  • run automated tests on the student code
  • etc.

The results of automated checks are shown in the submission page to both students and graders.

Note

fire will not automatically accept/reject student submission based on automated checks, this is left at the discretion of the graders.

Overview

../_images/autochecks-overview.svg

Here is how automated checks works

  • ① Students create a submission on fire
  • ② The fire server notify the test server by requesting a particular URL and pass it two addresses: one where the test server can download the students’ submission and one where it can submit feedback
  • ③ Once the test server has finished running the tests, it notifies the fire server using the passed URL and sends the check status (pass or fail) and optionally an address where more information are displayed (test log, generated report, etc...)
  • ④ The fires server displays the check status on the submission page to both the grader and the students.

Use case: using Jenkins as a test server

In this example we will see how to use Jenkins as a test server for student submissions. We assume that you already have a fire instance and a sever with jenkins installed.

Create a Jenkins job

  • Click on New Item, give your job a name and select “Freestyle project”.

  • Tick the check-box “This build is parameterized”

  • Add two Password Parameter: submission_url and feedback_url

  • Bellow, under Build Triggers, check Trigger builds remotely (e.g. from scripts)

  • Create a good authentication token (you may want to use pwgen). In this example we’ll use s3cr3t for brevity.

  • Under Build Environment, check Delete workspace befor build starts

  • Under Build, add an Execute shell build step with the following script:

    #!/bin/sh -eux

curl --silent $submission_url | tar zxf -

if [ $[RANDOM % 2] = 0 ]; then
  status=pass
else
  status=fail
fi

curl --silent -X POST \
  --data status=$status \
  --data details_url=$BUILD_URL/console \
  $feedback_url

    (of course a real check will not choose a status at random)

  • Click Save

Congratulations, your check job is now ready!

Set-up fire to use the Jenkins job

  • In fire, go to the lab for which you want to add an automated check and click the Checks tab.

  • Give a name to your check and the following url:

    <url to your jenkins server/job/<job name>/buildWithParameters?token=<your token>&submission_url=$sumbission_url&feedback_url=$feedback_url

  • Click Add

Now your check is ready. You might want to create a phony student to make a test submission to make sure that everything works as expected.