Summary
Initial draft of the SystemDS 2.1 release changelogImportant highlights from this release
- Federated
- Builtins
- Native Runtime
All updates for this release
Epic
- SYSTEMDS-2716 REOPENED A new Performance test suite
- SYSTEMDS-1973 CLOSED Optimization of parameters, Hyperparameters, and testing.
New Feature
- SYSTEMDS-2969 RESOLVED Federated rowProds, colProds
- SYSTEMDS-2957 RESOLVED Python API Importer
- SYSTEMDS-2952 RESOLVED logcosh loss function
- SYSTEMDS-2900 RESOLVED Sherlock builtin function
- SYSTEMDS-2877 RESOLVED Tomeklink builtin function
- SYSTEMDS-2862 RESOLVED SDC colgroup Sparse Dictionary Encoding
- SYSTEMDS-2851 RESOLVED Compressed n+ operation
- SYSTEMDS-2844 RESOLVED Random Forest to Builtin
- SYSTEMDS-2843 RESOLVED Kaplan Meir Estimates to Builtin
- SYSTEMDS-2842 RESOLVED Cspline to Builtin
- SYSTEMDS-2841 RESOLVED Cox algoritm to Builtin
- SYSTEMDS-2812 RESOLVED Conv1D
- SYSTEMDS-2811 RESOLVED Compressed Slice
- SYSTEMDS-2757 RESOLVED PCA Predict and Inverse
- SYSTEMDS-2741 RESOLVED Compressed overlapping unary aggregates
- SYSTEMDS-2736 RESOLVED Federated Ternary Aggregate
- SYSTEMDS-2719 RESOLVED Lineage exploitation in the buffer pool
- SYSTEMDS-2718 RESOLVED Matrix Mult Accelerator Comparison
- SYSTEMDS-2717 RESOLVED Micro benchmarks for the new performance test suite
- SYSTEMDS-2705 RESOLVED Federated Writer Level 0
- SYSTEMDS-2704 RESOLVED Federated Read
- SYSTEMDS-2702 RESOLVED Row & Column Bind Python
- SYSTEMDS-2701 RESOLVED Kmeans Predict
- SYSTEMDS-2700 RESOLVED Left hand side ops
- SYSTEMDS-2698 RESOLVED Initial cuda codegen functionality
- SYSTEMDS-2696 RESOLVED Overlapping relational operators and above support
- SYSTEMDS-2695 RESOLVED Left Mult Row based parallelization
- SYSTEMDS-2694 CLOSED Compressed Rowwise Cache-conscious operations
- SYSTEMDS-2693 RESOLVED Compressed Overlapping Semi-additive support
- SYSTEMDS-2689 RESOLVED Decompress Lop Operation
- SYSTEMDS-2686 RESOLVED Compressed Overlapping Column Groups
- SYSTEMDS-2681 CLOSED Federated Bivariate Statistics
- SYSTEMDS-2677 RESOLVED read unknowd dims for algorithms input
- SYSTEMDS-2676 RESOLVED Python toString Function
- SYSTEMDS-2675 RESOLVED Python Print Function
- SYSTEMDS-2550 RESOLVED Federated Parameter Server
- SYSTEMDS-2548 CLOSED Federated left and right indexing
- SYSTEMDS-2545 RESOLVED Federated Append
- SYSTEMDS-2544 RESOLVED Federated Aggregations Rowwise
- SYSTEMDS-2543 RESOLVED Federated Aggregations Columwise
- SYSTEMDS-979 CLOSED Add support for bayesian optimization
Improvement
- SYSTEMDS-2967 RESOLVED Docker Image from Ubuntu not R-base
- SYSTEMDS-2945 CLOSED CLA Isolated Dictionary SubPackage
- SYSTEMDS-2940 RESOLVED Add .asf.yml to control GitHub settings
- SYSTEMDS-2939 RESOLVED CLA BruteForce Cocode
- SYSTEMDS-2938 CLOSED CLA Reintroduce Bin-Packing
- SYSTEMDS-2930 OPEN Remove function pointer based matrix accessor
- SYSTEMDS-2912 CLOSED AggregateFinalResults avoid Generic correction
- SYSTEMDS-2905 RESOLVED Add MetaAll
- SYSTEMDS-2904 CLOSED Federated ternary
- SYSTEMDS-2896 CLOSED Spark Log4j propagation
- SYSTEMDS-2887 RESOLVED CLA Row aggregate optimization
- SYSTEMDS-2886 RESOLVED CLA InsertionSort for SDC groups
- SYSTEMDS-2885 RESOLVED CLA MMChain Optimization
- SYSTEMDS-2884 RESOLVED CLA Shared Mapping abstract
- SYSTEMDS-2882 RESOLVED Python Frame support
- SYSTEMDS-2871 RESOLVED Python API Autogenerator
- SYSTEMDS-2870 RESOLVED Add Builtin Decision Tree
- SYSTEMDS-2863 RESOLVED Federated ctable instruction
- SYSTEMDS-2853 RESOLVED Refactor spoof cuda runtime operations
- SYSTEMDS-2852 CLOSED Improve SPOOF CUDA compilation
- SYSTEMDS-2827 RESOLVED Rowwise template for CUDA codegen
- SYSTEMDS-2826 CLOSED Sparse input support for CUDA codegen
- SYSTEMDS-2825 CLOSED Avoid recompiling generated cuda operators
- SYSTEMDS-2821 RESOLVED Python Stability
- SYSTEMDS-2810 CLOSED Encoder and Decoder serialization
- SYSTEMDS-2803 RESOLVED Compressed Squeeze overlap to compressed
- SYSTEMDS-2800 RESOLVED Agressive Compression Rewrite
- SYSTEMDS-2798 RESOLVED Builtin (de)compress function
- SYSTEMDS-2794 RESOLVED custom dense reader compression
- SYSTEMDS-2792 RESOLVED Sparse Overlapping Matrix
- SYSTEMDS-2787 CLOSED Compression Steps Reorganization
- SYSTEMDS-2786 CLOSED Outputbuffering Correctly effect RScript Tests
- SYSTEMDS-2782 CLOSED Built-in mdedup
- SYSTEMDS-2766 CLOSED Federated covariance
- SYSTEMDS-2762 CLOSED Federated rshape instruction
- SYSTEMDS-2760 RESOLVED Transpose microbenchmark
- SYSTEMDS-2756 RESOLVED Scale builtin docs
- SYSTEMDS-2752 RESOLVED Compression seeding management
- SYSTEMDS-2751 RESOLVED Restructure Abstract Compressed block
- SYSTEMDS-2748 RESOLVED Compressed TSMM Optimization
- SYSTEMDS-2743 RESOLVED Compressed Divide Opt
- SYSTEMDS-2740 RESOLVED Add saved and missed computation time to Lineage cache statistics
- SYSTEMDS-2739 RESOLVED Improve CostNsize with ref count
- SYSTEMDS-2738 RESOLVED Federated rdiag, rev and uppertri instruction
- SYSTEMDS-2732 RESOLVED Federated Remove Empty
- SYSTEMDS-2731 RESOLVED Federated Timeout
- SYSTEMDS-2730 RESOLVED Federated removeEmpty
- SYSTEMDS-2729 RESOLVED Federated QPick
- SYSTEMDS-2728 RESOLVED Federated qsort
- SYSTEMDS-2727 RESOLVED Federated CM
- SYSTEMDS-2726 RESOLVED Federated Right Indexing
- SYSTEMDS-2725 RESOLVED NNZ counting for native blas
- SYSTEMDS-2724 RESOLVED Cast to matrix Federated
- SYSTEMDS-2723 RESOLVED Cast to frame Federated
- SYSTEMDS-2721 RESOLVED Add Federated SSL
- SYSTEMDS-2714 RESOLVED Seeded NN layers
- SYSTEMDS-2692 RESOLVED move cuda codebase
- SYSTEMDS-2669 RESOLVED Python Mnist LogReg tutorial
- SYSTEMDS-2642 CLOSED Isolated Statistics for workers in same JVM
- SYSTEMDS-2621 CLOSED DBScan & distance built in
- SYSTEMDS-2614 RESOLVED Right Compressed Matrix Mult
- SYSTEMDS-2613 RESOLVED Compressed Matrix Mult Sparse
- SYSTEMDS-152 RESOLVED Rename static readDML... and readR... methods in AutomatedTestBase
- SYSTEMDS-151 RESOLVED AutomatedTestBase calls deprecated TestUtils methods
Bug
- SYSTEMDS-2975 CLOSED Matrix-Frame conversions failing for large dense blocks
- SYSTEMDS-2960 CLOSED Codegen: ColVector Operation not considering RowIndex
- SYSTEMDS-2955 CLOSED Federated binary matrix-vector fails for 1x1 vector
- SYSTEMDS-2954 CLOSED Cleaning Enumerator fails in Spark context
- SYSTEMDS-2950 RESOLVED Threads wait forever for a removed entry in the lineage cache
- SYSTEMDS-2949 CLOSED Function return on same calls
- SYSTEMDS-2935 CLOSED Eval function loading failing in hyperband-parfor
- SYSTEMDS-2932 CLOSED Namespace issue in Parfor
- SYSTEMDS-2929 CLOSED Namespace conflicts - Builtins not loading after importing UDFs with builtins call
- SYSTEMDS-2928 CLOSED CSV parsing with non-default delimiter
- SYSTEMDS-2927 OPEN SPOOF memory estimate for exec = singlenode
- SYSTEMDS-2924 CLOSED Eval() does not support functions from namespaces other than default
- SYSTEMDS-2920 CLOSED Spark rexpand fails for non-literal max parameters
- SYSTEMDS-2915 RESOLVED Lost GPUObjects memory leak
- SYSTEMDS-2907 CLOSED Corrupted memory estimates
- SYSTEMDS-2888 CLOSED Codegen Cellwise with cbind
- SYSTEMDS-2868 CLOSED Spark transformencode fails for recode columns w/ spaces
- SYSTEMDS-2860 CLOSED Native BLAS tsmm fails for vector outer product
- SYSTEMDS-2859 CLOSED Federated KMeans, LM, L2SVM fail w/ 1 worker
- SYSTEMDS-2858 RESOLVED Federated pramameterserver scheme isolated argument
- SYSTEMDS-2855 CLOSED Federated LM Pipeline transfers X to coordinator
- SYSTEMDS-2823 CLOSED source() command failing while importing UDFs
- SYSTEMDS-2818 CLOSED Read of multi-block binary matrix fails
- SYSTEMDS-2816 CLOSED Unnecessary overhead due to incorrect spark broadcast cleanup
- SYSTEMDS-2814 CLOSED GC Overhead due to invalid Frame Reblock Caching
- SYSTEMDS-2802 CLOSED parfor-loop is not executed in certain cases when counting backwards
- SYSTEMDS-2796 CLOSED Federated right indexing indices literals
- SYSTEMDS-2788 CLOSED Appending matrix to a list not working
- SYSTEMDS-2759 CLOSED Federated lmCG lacks important rewrites
- SYSTEMDS-2745 CLOSED Addition assignment (+=) is interpreted as a normal assignment
- SYSTEMDS-2733 CLOSED Federated Lm pipeline fails w/ index out-of-bounds
- SYSTEMDS-2715 CLOSED Source functions arguments with default value
- SYSTEMDS-2712 RESOLVED Read CSV bug
- SYSTEMDS-2711 RESOLVED One Hot encode Python Bug
- SYSTEMDS-2710 CLOSED GMM fails during IPA due to missing kmeans args
- SYSTEMDS-2703 CLOSED Reading Matrix CSV
- SYSTEMDS-2690 RESOLVED Old website is still responding
- SYSTEMDS-2687 RESOLVED Transpose self matrix multiplication bug
- SYSTEMDS-2679 CLOSED Creation of rmvar instructions for non existent variables on recompile
- SYSTEMDS-2509 CLOSED Missing transformencode suport for binning over frames
Task
- SYSTEMDS-2943 RESOLVED Use staging and publish stages for website build
- SYSTEMDS-2942 RESOLVED Actions for website build
- SYSTEMDS-2937 RESOLVED NaiveBayesPredict builtin function
- SYSTEMDS-2925 RESOLVED Federated left indexing
- SYSTEMDS-2921 CLOSED Federated cumulative aggregates
- SYSTEMDS-2908 OPEN Refactor Datasets in SystemDS
- SYSTEMDS-2895 RESOLVED Builtin for Denial Constraint validation
- SYSTEMDS-2891 RESOLVED Gaussian Classifier
- SYSTEMDS-2881 RESOLVED String Tokenizer API
- SYSTEMDS-2876 OPEN Complete supported operators in CUDA codegen
- SYSTEMDS-2875 OPEN Implement OuterProduct template
- SYSTEMDS-2874 OPEN Implement MultiAggregate template
- SYSTEMDS-2872 RESOLVED Bayesian Optimization Algorithm
- SYSTEMDS-2867 RESOLVED Cleanup federated binary operations
- SYSTEMDS-2857 RESOLVED Federated unary matrix/frame operations
- SYSTEMDS-2856 CLOSED Multi-threaded binary elementwise operations
- SYSTEMDS-2820 CLOSED Ctable parameter for empty block outputs
- SYSTEMDS-2801 CLOSED Rewrite rule to remove (par)for-loops over empty sequences
- SYSTEMDS-2774 CLOSED Federated privacy exception handling
- SYSTEMDS-2772 CLOSED Autoencoder built-in
- SYSTEMDS-2764 CLOSED Frame Constructor
- SYSTEMDS-2747 CLOSED Federated Quaternary Operations
- SYSTEMDS-2709 CLOSED Federated mean
- SYSTEMDS-2688 CLOSED Diag V2M operations w/ shallow data copy
- SYSTEMDS-2685 RESOLVED systemml.a.o should redirect to systemds.a.o
- SYSTEMDS-2629 RESOLVED Federated reorg instruction
- SYSTEMDS-2623 CLOSED Federated parameterized builtins
Test
- SYSTEMDS-2878 CLOSED test encoder serialization
- SYSTEMDS-2707 RESOLVED Split Tests
Sub-task
- SYSTEMDS-2961 CLOSED Refactor Pipelines (2)
- SYSTEMDS-2918 CLOSED Improved IPA for parameter server functions
- SYSTEMDS-2911 CLOSED Asynchronous accrue gradients for epoch synchronization
- SYSTEMDS-2903 CLOSED Refactor Pipelines
- SYSTEMDS-2901 CLOSED Restructure Encoders
- SYSTEMDS-2806 CLOSED Design and APIs
- SYSTEMDS-2804 CLOSED Builtin functions ALS-predict and ALS-topk-predict
- SYSTEMDS-2799 CLOSED Lineage tracing and reuse of federated UDFs
- SYSTEMDS-2797 CLOSED Builtin for computing NA statistics (missing values in data)
- SYSTEMDS-2795 RESOLVED Reuse of FED instruction results in Coordinator
- SYSTEMDS-2790 CLOSED Builtin function outlierByArima
- SYSTEMDS-2789 CLOSED Builtin function for Disguised Missing Values Detection
- SYSTEMDS-2784 RESOLVED Lineage-based reuse in federated workers
- SYSTEMDS-2773 CLOSED Builtin function arima (time series forecasting)
- SYSTEMDS-2722 CLOSED Builtin function split for train/test splits
- SYSTEMDS-2683 CLOSED PPCA builtin function
- SYSTEMDS-2682 CLOSED Lasso builtin function
- SYSTEMDS-2680 CLOSED ALS builtin functions
- SYSTEMDS-2662 CLOSED Various Minor Built-ins
- SYSTEMDS-2661 CLOSED Pipelines Optimizer (Hyperband)
- SYSTEMDS-2586 CLOSED Compare compressed lineage DAGs
- SYSTEMDS-2583 CLOSED Reuse of compressed DAGs
- SYSTEMDS-2582 RESOLVED Hash and equal lineage item over compressed items
- SYSTEMDS-1993 RESOLVED Implementation of Gaussian Process Classification
Proposal
- SYSTEMDS-2599 CLOSED Change name of "master branch" to main