Ralph is a command-line tool to fetch, extract, convert and push your tracking logs from various storage backends to your LRS or any other compatible storage or database backend.

Overview

Ralph

Ralph is a command-line tool to fetch, extract, convert and push your tracking logs (aka learning events) from various storage backends to your LRS or any other compatible storage or database backend.

Quick start guide

Ralph is distributed as a standard python package; it can be installed via pip or any other python package manager (e.g Poetry, Pipenv, etc.):

$ pip install ralph-malph

Once installed, the ralph command should be available in your PATH. Try to invoke the program usage thanks to the --help flag:

$ ralph --help

You should see a list of available commands and global flags for ralph. Note that each command has its own usage that can be invoked via:

$ ralph COMMAND --help

You should substitute COMMAND by the target command, e.g. list, to see its usage.

Documentation

We try our best to maintain an up-to-date reference documentation for this project. If you intend to install, test or contribute to ralph, we invite you to read this documentation and give us feedback if some parts are unclear or your use case is not (or poorly) covered.

Contributing

This project is intended to be community-driven, so please, do not hesitate to get in touch if you have any question related to our implementation or design decisions.

We try to raise our code quality standards and expect contributors to follow the recommandations from our handbook.

License

This work is released under the MIT License (see LICENSE).

Comments
  • ✨(models) add edx navigational event pydantic models

    ✨(models) add edx navigational event pydantic models

    Purpose

    With the validate and select command newly implemented, it is now possible to implement edx tracking logs events pydantic models to validate incoming events in ralph.

    Proposal

    The first events implemented are the easiest (navigation).

    • [x] event models
    • [x] event fixtures
    • [x] event tests
    needs review feature 
    opened by quitterie-lcs 9
  • Add WebSocket to backends documentation

    Add WebSocket to backends documentation

    Purpose

    Add WebSocket to Backends documentation

    Proposal

    WebSocket

    The webSocket backend is read-only and can be used to get real-time events.

    If you use OVH's Logs Data Platform (LDP), you can retrieve a WebSocket URI to test your data stream by following instructions from the official documentation.

    Backend parameters

    • ws_uri: the WebSocket uri (e.g. wss://example.com/websocket)
    improvement WIP 
    opened by p-bizouard 6
  • ✨(converters) add edx server event xapi converter

    ✨(converters) add edx server event xapi converter

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to convert them into xAPI format.

    Proposal

    ServerEventSchema validates the edx server event.

    ServerEventToXapi uses the ServerEventSchema to validate the edx server event and then converts it to xAPI format.

    XapiConverterSelector deserializes the edx events and tries to match a known converter for the event conversion.

    feature 
    opened by SergioSim 6
  • Add helm chart

    Add helm chart

    Purpose

    Add Helm chart for Ralph

    Description

    Created the chart under src/helm I tried to migrate the logic from the .j2 tray files. The version label was replaced as the helm convention for app.kubernetes.io/version The checksum from the secret name was deleted. The annotation checksum/config: was used instead on the secret, the checksum is automatically generated from vault.yaml it help to update the secret object every time a value is updated. Note that the .Chart.AppVersion is better used to track the version deployed and it should reflect the image.tag There is only one vars files values.yaml the vault is under specific file vault.yaml used only for generating the secrets.

    Questions

    • Why do you have two service with the same config? the only difference is the label static
    • What is deployment_stamp ?
    • What is the purpose of prefix ?
    • How would you like to manage env.?

    ToDo

    • [ ] Add README.md

    You can use helm template . under ./src/helm/ralph to check the output of the chart, it will show you the final manifest that will be deployed.

    feature 
    opened by waammar 5
  • Add MongoDB backend

    Add MongoDB backend

    Purpose

    MongoDB is a classical database for LRSs such as LearningLocker or Apereo Learning Analytics Initiative. Supporting this backend seems a must-have for Ralph.

    Proposal

    • [x] add LearningLocker parser
    • [x] add MongoDB backend
    • [x] ~add learning locker parser~ edit: postponed to a new PR
    needs review feature 
    opened by jmaupetit 5
  • ✨(converter) add edx to xapi video converter

    ✨(converter) add edx to xapi video converter

    Purpose

    edX video events can be converted to xAPI for most of them. The interaction events can still not be converted.

    Proposal

    • [x] load_video to initialized
    • [x] play_video to played
    • [x] pause_video to paused
    • [x] stop_video to terminated
    • [x] seek_video to seeked
    improvement needs review FUN 
    opened by quitterie-lcs 4
  • ✨(models) add xAPI page_viewed and page_terminated models

    ✨(models) add xAPI page_viewed and page_terminated models

    Purpose

    These pydantic models can be used for xAPI format validation.

    Proposal

    The page_viewed pydantic model correspond to the edX simple server event. The page_terminated pydantic model correspond to the edX page_close browser event.

    • [X] page_viewed pydantic model
    • [X] page_terminated pydantic model
    • [x] tests
    feature WIP 
    opened by SergioSim 4
  • ✨(models) add edx server event model

    ✨(models) add edx server event model

    Purpose

    The edx server events are the most simple and common events in the edx tracking logs. They are logged at almost all requests made to the edx lms. We want to validate them.

    Proposal

    • [x] ServerEventModel
    • [x] ServerEventFactory
    • [x] Tests
    feature WIP 
    opened by SergioSim 4
  • Add test fixtures

    Add test fixtures

    Purpose

    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    feature WIP 
    opened by SergioSim 4
  • Add

    Add "event" test fixture factory

    Purpose

    • Document the different events that are logged as "tracking logs" in the LMS to understand how, were and when things get logged
    • A event test fixture factory could help to test the function of the GELFParser

    Proposal

    • [ ] Implementation of the event test fixture factory
    • [ ] Events documentation
    feature WIP 
    opened by SergioSim 4
  • ✨(cli) add the new

    ✨(cli) add the new "auth" command

    Purpose

    Generating credentials to authenticate to our LRS should be smooth and painless for our users.

    Proposal

    • [x] add a new auth command to ease new user credentials generation
    • [x] add pydantic models for user credentials checking and credentials file validation
    • [x] track authentication issues to the LRS server
    needs review feature 
    opened by jmaupetit 3
  • `make bootstrap` is failing due to missing build dependences

    `make bootstrap` is failing due to missing build dependences

    Bug Report

    Hi all, I'm excited to get started developing with Ralph. Getting set up I ran into an issue, however. On the master branch make bootstrap errors out pip installing psutil due to gcc not being available.

    Expected behavior/code make bootstrap builds the docker images cleanly.

    Actual Behavior An error occurs:

    #0 22.22       creating build/temp.linux-aarch64-cpython-39/psutil
    #0 22.22       gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=594 -DPy_LIMITED_API=0x03060000 -DPSUTIL_LINUX=1 -DPSUTIL_ETHTOOL_MISSING_TYPES=1 -I/usr/local/include/python3.9 -c psutil/_psutil_common.c -o build/temp.linux-aarch64-cpython-39/psutil/_psutil_common.o
    #0 22.22       C compiler or Python headers are not installed on this system. Try to run:
    #0 22.22       sudo apt-get install gcc python3-dev
    #0 22.22       error: command 'gcc' failed: No such file or directory
    

    Steps to Reproduce

    1. Sync the Ralph master branch
    2. run make bootstrap
    3. The above error occurs

    Environment

    • Ralph version: master branch
    • Platform: Docker Desktop on Mac OS 13.0.1

    Possible Solution I was able to get a successful build by adding the required development tools, see the PR here: https://github.com/bmtcril/ralph/pull/1

    bug 
    opened by bmtcril 1
  • 📝(project) add helm chart tutorial

    📝(project) add helm chart tutorial

    Purpose

    Ralph can be deployed to a Kubernetes cluster using its official Helm Chart. We need a step-by-steph guide to use it.

    Proposal

    This tutorial guides you to the deployment of a complete learning analytics stack to Kubernetes using Helm.

    FUN WIP 
    opened by jmaupetit 1
  • Implement xAPI forum models in `ralph`

    Implement xAPI forum models in `ralph`

    Purpose

    xAPI forum used in ashley models have to be described in ralph for validation purpose and usage of ralph as a library

    Proposal

    • [ ] Define pydantic models associated to each forum learning statement templates
    • [ ] Define selector for each model
    • [ ] Write model and selector tests
    feature 
    opened by quitterie-lcs 0
  • Integrate mypy for Python type checking

    Integrate mypy for Python type checking

    Purpose

    It has been decided that we use typing in all ralph project. We have to integrate a static type checker in our linting toolbox.

    Proposal

    • Integrate mypy in the project (local tooling for development + CI)
    improvement 
    opened by quitterie-lcs 0
  • 🏗️(backends) unify data backends fs

    🏗️(backends) unify data backends fs

    Purpose

    Storage and Database backends have similar interfaces and usage.

    Proposal

    Unify the interfaces of database and storage backends into a single DataBackend interface.

    • [x] add Data and LRS Backend interfaces
    • [x] add FileSystemDataBackend
    needs review feature 
    opened by SergioSim 1
  • Ora2 events

    Ora2 events

    Purpose

    ORA2 events describe the activity for assessments pair reviewing. Edx defined models and their selectors are described with Pydantic.

    Proposal

    Define Pydantic model for:

    • openassessmentblock.create_submission,
    • openassessmentblock.get_peer_submission,
    • openassessmentblock.get_submission_for_staff_grading,
    • openassessmentblock.peer_assess,
    • openassessmentblock.save_submission,
    • openassessmentblock.self_assess,
    • openassessmentblock.staff_assess,
    • openassessmentblock.student_training_assess_example,
    • openassessment.submit_feedback_on_assessments,
    • openassessment.upload_file,
    needs review feature 
    opened by quitterie-lcs 0
Releases(v3.1.0)
  • v3.1.0(Nov 17, 2022)

    Added

    • EdX to xAPI converters for video events

    Changed

    • Improve Ralph's library integration by unpinning dependencies (and prefer ranges)
    • Upgrade fastapi to 0.87.0
    Source code(tar.gz)
    Source code(zip)
  • v3.0.0(Oct 19, 2022)

    Added

    • Implement edx video browser events pydantic models
    • Create a post endpoint for statements implementing the LRS spec
    • Implement support for the MongoDB database backend
    • Implement support for custom queries when using database backends get method (used in the fetch command)
    • Add dotenv configuration file support and python-dotenv dependency
    • Add host and port options for the runserver cli command
    • Add support for database selection when running the Ralph LRS server
    • Implement support for xAPI statement forwarding
    • Add database backends status checking
    • Add health LRS router
    • Tray: add LRS server support

    Changed

    • Migrate to python-legacy handler for mkdocstrings package
    • Upgrade click to 8.1.3
    • Upgrade elasticsearch to 8.3.3
    • Upgrade fastapi to 0.79.1
    • Upgrade ovh to 1.0.0
    • Upgrade pydantic to 1.9.2
    • Upgrade pymongo to 4.2.0
    • Upgrade python-keystoneclient to 5.0.0
    • Upgrade python-swiftclient to 4.0.1
    • Upgrade requests to 2.28.1
    • Upgrade sentry_sdk to 1.9.5
    • Upgrade uvicorn to 0.18.2
    • Upgrade websockets to 10.3
    • Make backends yield results instead of writing to standard streams (BC)
    • Use pydantic settings management instead of global variables in defaults.py
    • Rename backend and parser parameter environment variables (BC)
    • Make project dependencies management more modular for library usage

    Removed

    • Remove YAML configuration file support and pyyaml dependency (BC)

    Fixed

    • Tray: do not create a cronjobs list when no cronjob has been defined
    • Restore history mixin logger
    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Apr 25, 2022)

    Added

    • Implement edx problem interaction events pydantic models
    • Implement edx textbook interaction events pydantic models
    • ws websocket stream backend (compatible with the fetch command)
    • bundle jq, curl and wget in the fundocker/ralph Docker image
    • Tray: enable ralph app deployment command configuration
    • Add a runserver command with basic auth and a whoami route
    • Create a get endpoint for statements implementing the LRS spec

    Changed

    • Upgrade uvicorn to 0.17.4
    • Upgrade elasticsearch to 7.17.0
    • Upgrade sentry_sdk to 1.5.5
    • Upgrade fastapi to 0.73.0
    • Upgrade pyparsing to 3.0.7
    • Upgrade pydantic to 1.9.0
    • Upgrade python-keystoneclient to 4.4.0
    • Upgrade python-swiftclient to 3.13.0
    • Upgrade pyyaml to 6.0
    • Upgrade requests to 2.27.1
    • Upgrade websockets to 10.1
    Source code(tar.gz)
    Source code(zip)
  • v2.0.1(Jul 15, 2021)

  • v2.0.0(Jul 12, 2021)

    Added

    • xAPI video interacted pydantic models
    • xAPI video terminated pydantic models
    • xAPI video completed pydantic models
    • xAPI video seeked pydantic models
    • xAPI video initialized pydantic models
    • xAPI video paused pydantic models
    • convert command to transform edx events to xAPI format
    • EdX to xAPI converters for page viewed andpage_close events
    • Implement core event format converter
    • xAPI video played pydantic models
    • xAPI page viewed and page terminated pydantic models
    • Implement edx navigational events pydantic models
    • Implement edx enrollment events pydantic models
    • Install security updates in project Docker images
    • Model selector to retrieve associated pydantic model of a given event
    • validate command to lint edx events using pydantic models
    • Support all available bulk operation types for the elasticsearch backend (create, index, update, delete) using the --es-op-type option

    Changed

    • Upgrade elasticsearch to 7.13.2
    • Upgrade python-swiftclient to 3.12.0
    • Upgrade click to 8.0.1
    • Upgrade click-option-group to 0.5.3
    • Upgrade pydantic to 1.8.2
    • Upgrade sentry_sdk to 1.1.0
    • Rename edx models
    • Migrate model tests from factories to hypothesis strategies
    • Tray: switch from openshift to k8s (BC)
    • Tray: remove useless deployment probes

    Fixed

    • Tray: remove version immutable field in DC selector
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Feb 26, 2021)

    Added

    • EdX server event pydantic model and factory
    • EdX page_close browser event pydantic model and factory
    • Tray: allow to specify a self-generated elasticsearch cluster CA certificate

    Fixed

    • Tray: add missing Swift variables in the secret
    • Tray: fix pods anti-affinity selector

    Removed

    • pandas is no longer required
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 4, 2021)

    Added

    • Support for Swift storage backend
    • Use the push command --ignore-errors option to ignore ES bulk import errors
    • The elasticsearch backend now accepts passing all supported client options

    Changed

    • Remove click_log package dependency
    • Upgrade pyyaml to 5.4.1
    • Upgrade pandas to 1.2.1
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jan 13, 2021)

    Added

    • Implement base CLI commands (list, extract, fetch & push) for supported backends
    • Support for ElasticSearch database backend
    • Support for LDP storage backend
    • Support for FS storage backend
    • Parse (gzipped) tracking logs in GELF format
    • Support for application's configuration file
    • Add optional sentry integration
    • Distribute Arnold's tray to deploy Ralph in a k8s cluster as cronjobs
    Source code(tar.gz)
    Source code(zip)
Owner
France Université Numérique
Collaboratively Building Innovative Digital Learning Services
France Université Numérique
liquidctl – liquid cooler control Cross-platform tool and drivers for liquid coolers and other devices

Cross-platform CLI and Python drivers for AIO liquid coolers and other devices

1.7k Jan 08, 2023
Code for the Open Data Day 2022 publicbodies.org Nepal data scraping activities.

Open Data Day Publicbodies.org Nepal We've gathered on Saturday, 5th March 2022 with Open Knowledge Nepal in order to try and automate the collection

Augusto Herrmann 2 Mar 12, 2022
keep your machine's shell history synchronize

SyncShell Yet another tool for laziness Keep your machine's shell history synchronize Get SyncShell Currently, SyncShell is just available on PyPi and

Masoud Ghorbani 53 Dec 12, 2022
🌈 Generate color palettes based on Neovim colorschemes.

Iris Iris is a Neovim plugin that generates a normalized color palette based on your colorscheme. It is named for the goddess Iris of Greek mythology,

N. G. Scheurich 45 Jul 28, 2022
Pynavt is a cli tool to create clean architecture app for you including Fastapi, bcrypt and jwt.

Pynavt _____ _ | __ \ | | | |__) | _ _ __ __ ___ _| |_ | ___/ | | | '_ \ / _` \ \ / /

Alejandro Castillo 1 Dec 13, 2021
A CLI/Shell supporting OpenRobot API and more!

A CLI/Shell supporting JeyyAPI, OpenRobot API and RePI API.

OpenRobot Packages 1 Jan 06, 2022
Simple Python Library to display text with color in Python Terminal

pyTextColor v1.0 Introduction pyTextColor is a simple Python Library to display colorful outputs in Terminal, etc. Note: Your Terminal or any software

Siddhesh Chavan 1 Jan 23, 2022
An anime command-line system information tool written in python.

Animefetch - v0.0.3 An anime command-line system information tool written in python. Description Animefetch is an anime command-line system informatio

Thadeuks 6 Jun 17, 2022
This is a CLI program which can help you generate your own QR Code.

Python-QR-code-generator This is a CLI program which can help you generate your own QR Code. Single.py This will allow you only to input a single mess

1 Dec 24, 2021
Projeto Reverse Shell For Python

Use com sabedoria!!! Modo de uso: Linux (inclui Android e Mac): - apt-get update - apt install python3 (ou "python" apenas) - git clone https://github

1 Jan 03, 2022
CLI tool to computes CO2 emissions of HPC computations following green-algorithms.org methodology

gqueue gqueue is a CLI (command line interface) tool that computes carbon footprint of HPC computations on clusters running slurm. It follows the meth

4 Dec 10, 2021
🖍️This is a feature-complete clone of the awesome Chalk (JavaScript) library.

Terminal string styling done right This is a feature-complete clone of the awesome Chalk (JavaScript) library. All credits go to Sindre Sorhus. Highli

Fabian Keller 132 Dec 27, 2022
Unpacks things.

$ unp_ unp is a command line tool that can unpack archives easily. It mainly acts as a wrapper around other shell tools that you can find on v

Armin Ronacher 405 Jan 03, 2023
This tool is a free and unlimited python CLI for google translate. based on google_trans_new.

GoTransPy A free and unlimited python CLI for google translate based on google_trans_new. It's very easy to use and solve the problem that the old api

Youssef Mohamed 2 Jan 10, 2022
Tmux Based Dropdown Dashboard For Python

sextans It's a private configuration and an ongoing experiment while I use Archlinux. A simple drop down dashboard based on tmux. It includes followin

秋葉 4 Dec 22, 2021
Python script to tabulate data formats like json, csv, html, etc

pyT PyT is a a command line tool and as well a library for visualising various data formats like: JSON HTML Table CSV XML, etc. Features Print table o

Mobolaji Abdulsalam 1 Dec 30, 2021
A simple cli utility for importing or exporting dashboard json definitions using the Grafana HTTP API.

P4CMD 🌴 A Python Perforce package that doesn't bring in any other packages to work. Relies on p4cli installed on the system. p4cmd The p4cmd module h

Beam Connectivity 31 Jan 06, 2023
A simple discord slash command handler for for discord.py.

A simple discord slash command handler for discord.py About ⦿ Installation ⦿ Disclaimer ⦿ Examples ⦿ Documentation ⦿ Discussions Note that master bran

641 Jan 03, 2023
Spongebob-cli - Watch classic spongebob from the terminal

spongebob-cli Watch classic spongebob from the terminal! Thanks to everyone that

523 Jan 02, 2023
Pymongo based CLI client, to run operation on existing databases and collections

Mongodb-Operations-Console Pymongo based CLI client, to run operation on existing databases and collections Program developed by Gustavo Wydler Azuaga

Gus 1 Dec 01, 2021