Goals

• Allow for scale-ability in analytics framework for Apache SensSoft
  • Distill 0.2.0 will allow us to grow the incumbent analytical/modeling capability of Distill, including:
    • Pre-packaged preprocessing methods for filtering, sequencing and packaging time-series event data, with meta, as portable python Dictionaries
    • Pre-packaged graph and time-series modeling methods
    • Limited packaging of statistical and data processing python packages (e.g., NumPy, SciPy, Pandas, etc.)
• Allow for customizable user generated python content within Distill
  • Distill 0.2.0 will allow users to generate their own libraries for Distill, that implement different pre-packaged functions, as well as user-generated ones.
  • Distill 0.2.0 will enforce a predictable library structure, making it a less error-prone process, and reducing support burden for Distill
• Allow for processed user log data portability to different environments (e.g., visualization, other analytic environments, i.e., anaconda)
  • Distill 0.2.0 will generate a predictable output structure, making it easier for developing readers into different environments
  • Distill 0.2.0 will generate predictable outputs that can be consumed by other services, applications
  • Distill 0.2.0 output will be parsible within other python batch scripts, scripts, or IDEs
• Allow for reduced-to practice methods allowing User Activity data from Software Environments to be useful for a variety of use-cases:
  • Automation: other processes, applications are able to consume processed Distill data to drive logic or analytic-bases processes
    • Example: A web application changes the rate at which messages are pushed to users based on what and how users are making use of application features.
    • Example: Distill is regularly called to generate updated model-derived features for secondary processing, such as Machine Learning.
  • Off-Line Analytics:  DataScientists, Analysts, and Statisticians of various levels of technical depth are able to quickly process user activity data and draw Distill output in to their analytics pipeline.
  • Visual Analytics: Visualization and Interactive data visualizations (e.g., TAP) can utilize Distill in configuring interactive features (menus, toggles, drop downs) to interact with user activity data and extract model parameters that are used for rendering returns through visualization.

Background and strategic fit

• UserALE data is structurally rich, and while manageable, requires technical depth to process beyond simple aggregation
• Distill 0.2.0 will make UserALE data usable by a wider range of professionals and academics with canonical offerings that can reduce pre-processing workload substantially for novices.
• Distill 0.2.0 will make UserALE data useful for a wider range of use-cases that depend on deep expertise in working with time-series nested multi-dimensional, categorical and semantic data.
• Distill 0.2.0 will provide the Apache SensSoft community with more discrete, bounded programming projects to add to the code based with more immediate value to contributors. 
• Distill 0.2.0 will provide Apache SensSoft with substantial horizontal growth opportunities in order to grow the community base.

Assumptions

• Distill will act as an abstraction layer around Elastic Search, and will not provide direct access to the Elastic Search api
• When consumed as a python library, Distill will return python data objects (e.g. dictionaries)
• When consumed as a web service, Distill will return json objects
• Distill will never modify the source data (read operations only)
• Distill will not provide any storage or caching for transformed data
• Distill will only support Python 3.5+
• Distill will only support x64 architecture

Requirements

#	Title	User Story	Importance	Notes
1	Accommodate arbitrary data schemas (not just UserALE): allows us to expand to cover arbitrarily many data sources without needing to write new methods for Distill
2	Distill is primarily designed to support UserALE.js Distill may be modified to support other time-dependent, sequential event data with meta-data
2	Distill will support json datastores for datasource/query 0.2.0 will only support Elastic 5.6.3+
3	Installable as a Python library (via pip install)
4	Consumable as a RESTful web service
5	Provide a set of built-in data queries for common use cases
6	Allow users to make custom queries not covered by the built-in queries
7	Provide a set of built-in data transformations for common use cases
8	Thoroughly document the API using Sphinx so that users can extend Distill functionality to custom data transformations, analytics, schemas, etc.
9	Support Windows, Linux, and Mac users with OS-specific eggs/wheels
10	End-to-end encryption
11	Provide convenience libraries with pre-built data schemas for UserALE and other data streams

Include any mockups, diagrams or visual designs relating to these requirements.

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Analytics & Processing Examples

These examples are here for drawing out higher-level goals for Distill's functionality. This section can be removed once the goals have been solidified.

Here is a model data pipeline for Apache SensSoft: RAW DATA>QUERY>FILTER/Q&A>TRANSFORMATION>PRIMITIVE FEATURE EXTRACTION>TRAINED MODELING>DERIVED FEATURE EXTRACTION

There are a few different classes of libraries that Distill might include in support of this pipeline; they have different consequences for workflows with in larger analytic pipelines.

1. QUERY: We may want to be able to recreate previous queries used for other analyses, not necessarily "save" queries.
   
   
2. FILTERING/Q&A: Elimination of data from query return, when that data can't be eliminated by query alone because some pattern to be filtered is fully nested within some query index.
   1. EX: Filter out specific save events from osquery object access data that do not coincide with click/keyboard activity with KM Logger.
   2. EX: Random resampling of km-logger events time-series–random sample every 1/min interval
      
      
3. TRANSFORMATION: Native format of Lucene-like DataStores is a list of records, called as JSON through querie
   1. EX: Query data from one or more data sources (bunch of JSON), impose structure on JSON so that we represent as list object of logs ordered by timestamp (TS)
      
      
4. PRIMITIVE FEATURE EXTRACTION
   1. EX: Query for UserALE.js data and type=="click", then by userId, then aggregate across some time interval (e.g., count) by unique userId (within-user), return 
   2. EX: Query for UserALE.js data and type=="click", then by userId, then aggregate over logs by unique user Id (e.g., count, mean, media, mode, variance, range) (between-user)
   3. EX: Using count data (EX a), bin by "path", create probabiliy of clicking on X path.
      
      
5. TRAINED MODELING
   
   1. EX: Call or recreate PRIMITIVE FEATURES, then feed features to Graph Methods, NN or HMM, etc. (see this paper), return model params to Python Env.
   2. EX: Build a simple directed graph (like bowie http://senssoft.incubator.apache.org/) that shows stochastic relationships between elements, or pages, return model params to Python Env. as well as in/out degree, centrality metrics.
      
      
6. DERIVED FEATURE EXTRACTION
   1. EX: Call or recreate 5, extract model features as in 4, return to Python Env.

• Build intervals from matching sequences of raw events
• Filter out unwanted events
  • Noisy/irrelevant events
    • May be conditional on neighboring events
  • "dangling" events (e.g. a stop event with no corresponding start)
• Collapse duplicate events into a single event (when is this preferable to creating an interval?)
• Create "sandwiches" (a set of events bookended by, e.g., a related start and stop event)
• Replace some logs/data with other logs/data

Not Doing

• We are NOT competing with Anaconda.
• We are NOT supporting multiple versions of Python.