Page History

...

Analytics & Processing Examples

These examples are here for drawing out higher-level goals for Distill's functionality. This section can be removed once the goals have been solidified.

Here is a model data pipeline for SENSSOFT: RAW DATA>QUERY>FILTER/Q&A>TRANSFORMATION>PRIMITIVE FEATURE EXTRACTION>TRAINED MODELING>DERIVED FEATURE EXTRACTION

There are a few different classes of libraries that Distill might include in support of this pipeline; they have different consequences for workflows with in larger analytic pipelines.

1. QUERY: We may want to be able to recreate previous queries used for other analyses, not necessarily "save" queries.
   
   
2. FILTERING/Q&A: Elimination of data from query return, when that data can't be eliminated by query alone because some pattern to be filtered is fully nested within some query index.
   1. EX: Filter out specific save events from osquery object access data that do not coincide with click/keyboard activity with KM Logger.
   2. EX: Random resampling of km-logger events time-series–random sample every 1/min interval
      
      
3. TRANSFORMATION: Native format of Lucene-like DataStores is a list of records, called as JSON through querie
   1. EX: Query data from one or more data sources (bunch of JSON), impose structure on JSON so that we represent as list object of logs ordered by timestamp (TS)
      
      
4. PRIMITIVE FEATURE EXTRACTION
   1. EX: Query for UserALE.js data and type=="click", then by userId, then aggregate across some time interval (e.g., count) by unique userId (within-user), return 
   2. EX: Query for UserALE.js data and type=="click", then by userId, then aggregate over logs by unique user Id (e.g., count, mean, media, mode, variance, range) (between-user)
   3. EX: Using count data (EX a), bin by "path", create probabiliy of clicking on X path.
      
      
5. TRAINED MODELING
   
   1. EX: Call or recreate PRIMITIVE FEATURES, then feed features to Graph Methods, NN or HMM, etc. (see this paper), return model params to Python Env.
   2. EX: Build a simple directed graph (like bowie http://senssoft.incubator.apache.org/) that shows stochastic relationships between elements, or pages, return model params to Python Env. as well as in/out degree, centrality metrics.
      
      
6. DERIVED FEATURE EXTRACTION
   1. EX: Call or recreate 5, extract model features as in 4, return to Python Env.

• Build intervals from matching sequences of raw events
• Filter out unwanted events
  • Noisy/irrelevant events
    • May be conditional on neighboring events
  • "dangling" events (e.g. a stop event with no corresponding start)
• Collapse duplicate events into a single event (when is this preferable to creating an interval?)
• Create "sandwiches" (a set of events bookended by, e.g., a related start and stop event)
• Replace some logs/data with other logs/data

...