...
- User Analytic Logging Engine (User ALE) refers to a simple Application Program Interface (API) and backend infrastructure. User ALE provides “instrumentation” for software tools, such that each user interaction within the application can be logged, and sent as a JSON message to an Elasticsearch/Logstash/Kibana (Elastic Stack) backend.
- The API provides a robust schema that makes user activities human readable, and provides an interpretive context for understanding that activity’s functional relevance within the application. The schema provides highly granular information best suited for advanced analytics. This hierarchical schema is as follows:
- Element Group: App features that share function (e.g., map group)
- Element Sub: Specific App feature (e.g., map tiles)
- Element Type: Category of feature (e.g., map)
Wiki Markup Element ID: \[attribute\] id- Activity: Human imposed label (e.g., “search”)
- Action: Event class (e.g., zoom, hover, click)
- The API can either be manually embedded in the app source code, or implemented automatically by inserting a script tag in the source code.
- Users can either setup up their own Elastic stack instance, or use Vagrant, a virtualization environment, to deploy a fully configured Elastic stack instance to ship and ingest user activity logs and visualize their log data with Kibana.
- RESTful APIs allow other services to access logs directly from Elasticsearch.
- User ALE allows adopters to own the data they collect from users outright, and utilize it as they see fit.
- The API provides a robust schema that makes user activities human readable, and provides an interpretive context for understanding that activity’s functional relevance within the application. The schema provides highly granular information best suited for advanced analytics. This hierarchical schema is as follows:
- Distill is an analytics stack for processing user activity logs collected through User ALE. Distill is fully implemented in Python, dependent on graph-tool to support graph analytics and other external python libraries to query Elasticsearch. The two principle functions of Distill are segmentation and graph analytics:
- Segmentation allows for partitioning of the available data along multiple axes. Subsets of log data can be selected via their attributes in User ALE (e.g. Element Group or Activity), and by users/sessions. Distill also has the capability to ingest and segment data by additional attributes collected through other channels (e.g. survey data, demographics).This allows adopters to focus their analysis of log data on precisely the attributes of their app (or users) they care most about.
- Distill’s usage metrics are derived from a probabilistic representation of the time series of users’ interactions with the elements of the application. A directed network is constructed from the representation, and metrics from graph theory (e.g. betweenness centrality, in/out-degree of nodes) are derived from the structure. These metrics provide adopters ways of understanding how different facets of the app are used together, and they capture canonical usage patterns of their application. This broad analytic framework provides adopters a way to develop and utilize their own metrics
- The Test Application Portal (TAP) provides a single, user-friendly interface to Software as a Sensor™ Project components, including visualization functionality for Distill Outputs leveraging Django, React, and D3.js. It has two key functions:
- It allows adopters to register apps, providing metadata regarding location, app name, version, etc., as well as permissions regarding who can access user data. This information is propagated to all other components of the larger system.
- The portal also stages visualization libraries that make calls to Distill. This allows adopters to analyze their data as they wish to; it’s “dashboard” feel provides a way to customize their views with adopter-generated widgets (e.g., D3 libraries) beyond what is included in the initial open source offering.
- The Subject Tracking and Online User Testing (STOUT) application is an optional component that turns Software as a Sensor™ Technology into a research/experimentation enterprise. Designed for psychologists and HCI/UX researchers, STOUT allows comprehensive human subjects data protection, tracking, and tasking for formal research on software tools. STOUT is primarily python, with Django back-end for authentication, permissions, and tracking, MongoDB for databasing, and D3 for visualization. STOUT includes a number of key features:
- Participants can register in studies of software tools using their own preferred credentials. As part of registration, participants can be directed through human subjects review board compliant consent forms before study enrollment.
- STOUT stores URLs to web/network accessible software tools as well as URLs to third party survey services (e.g., surveymonkey), this allows adopters to pair software tools with tasks, and collect survey data and comments from participants prior to, during, or following testing with software tools.
- STOUT tracks participants’ progress internally, and by appending a unique identifier, and task identifier to URLs. This information can be passed to other processes (e.g., User ALE) allowing for disambiguation between participants and tasks in experiments on the open web.
- STOUT supports between and within-subjects experimental designs, with random assignment to experimental conditions. This allows for testing across different versions of applications.
- STOUT can also use Django output (e.g., task complete) to automate other processes, such as automated polling applications serving 3rd party form data APIs (e.g.,SurveyMonkey), and python or R scripts to provide automated post-processing on task or survey data.
- STOUT provides adopters a comprehensive dashboard view of data collected and post-processed through its extensions; in addition to user enrollment, task completion, and experiment progress metrics, STOUT allows adopters to visualize distributions of scores collected from task and survey data.
...