The Universal Recommender Version Log
- Pagination support in query using
"from": 0, "num": 2 will return 2 recs from the first available,
"from": 2, "num": 2 will return 2 starting at the 3rd since
"from" is 0 based.
This tag take precedence over 0.7.0, which should not be used. Changes:
- Removes the need to build Mahout from the ActionML's fork and so is much easier to install.
- Fixes a bug in the integration test which made it fail for macOS High Sierra in East Asian time zones.
This README Has Special Build Instructions! These are obsolete now but you can see them in the GitHub README.md
This tag is for the UR integrated with PredictionIO 0.12.0 using Scala 2.11, Spark 2.1.x, and most importantly Elasticsearch 5.x. Primary differences from 0.6.0:
- Faster indexing, and queries due to the use of Elasticsearch 5.x
- Faster model building due to speedups in the ActionML fork of Mahout, which requires the user to build Mahout locally. This step will be removed in a later version of the UR. UPDATE: See v0.7.1
- Several upgrades such as Scala 2.10 --> Scala 2.11, Python 2.7 --> Python 3
- Spark 2.1.x support, PIO has a minor incompatibility with Spark 2.2.x
- Prediction 0.12.1 support
- Requires Elasticsearch 5.x. using the ES REST APIs exclusively now, enabling ES authentication use optionally. ES 5.x also improves indexing and query performance over previous versions.
- Fixed a bug in exclusion rules based on item properties
WARNING: Upgrading Elasticsearch or HBase will wipe existing data if any, so follow the special instructions below before installing any service upgrades and making backups.
Special Instructions (not reflected on ActionML.com yet)
You must build PredictionIO with the default parameters so just run
./make-distribution this will require you to install Scala 2.11 and Python 3 (as the default Scala and Python). You can also run up to Spark 2.1.x (but not 2.2.x), ES 5.5.2 or greater (but 6.x has not been tested), Hadoop 2.6 or greater, you can get away with using older versions of services except ES must be 5.x. If you have issues getting pio to build and run send questions to the PIO mailing list.
Backup your data, moving from ES 1 to ES 5 will delete all data!!!! Actually even worse it is still in HBase but you can’t get at it so to upgrade do the following:
pio export with pio < 0.12.0 =====Before upgrade!=====
pio data-delete all your old apps =====Before upgrade!=====
- build and install pio 0.12.0 including all the services =====The point of no return!=====
pio app new … and
pio import … any needed datasets
Once PIO is running test with
pio status and
pio app list. To test your setup and UR integration, run
./examples/integration-test from the URs home.
Config for PIO 0.12.1 and the UR 0.7.0
a sample of pio-env.sh that works with one type of setup is below, but you'll have to change paths to match yours. This example shows the new way to configure for Elasticsearch 5.x, which uses a new port number:
# SPARK_HOME: Apache Spark is a hard dependency and must be configured.
# using Spark 2.1.2 here
# ES_CONF_DIR: You must configure this if you have advanced configuration for
# using ES 5.6.3
# HADOOP_CONF_DIR: You must configure this if you intend to run PredictionIO
# using hadoop 2.8 here
# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
# using HBase 1.2.x here or whatever the highest numbered stable release is
# Filesystem paths where PredictionIO uses as block storage.
# Storage Repositories
# ES config
PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200 # <===== notice 9200 now
PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=elasticsearch_xyz # <===== should match what you have in you ES config file
This is the last version that will support PredictionIO 0.11.0, Elasticsearch 1.7.6, and Scala 2.10. Bug fixes may be added to this branch where needed but it will be left in the "UR v0.6.0"
This is a major upgrade release with several new features. Backward compatibility with 0.5.0 is maintained. Note: We no longer have a default
engine.json file so you will need to copy
engine.json and edit it to fit your data. See the Universal Recommender Configuration docs.
- Performance: Nearly a 40% speedup for most model calculation, and a new tuning parameter that can yield further speed improvements by filtering out unused or less useful data from model building. See
minEventsPerUser in the UR configuration docs.
- Complimentary Purchase aka Item-set Recommendations: "Shopping-cart" type recommendations. Can be used for wishlists, favorites, watchlists, any list based recommendations. Used with list or user data.
- Exclusion Rules: now we have business rules for inclusion, exclusion, and boosts based on item properties.
- PredictionIO 0.11.0: Full compatibility, but no support for Elasticsearch 5, an option with PIO-0.11.0.
- New Advanced Tuning: Allows several new per indicator / event type tuning parameters for tuning model quality in a more targeted way.
- Norms Support: For large dense datasets norms are now the default for model indexing and queries. This should result in slight precision gains, so better results.
- Mahout 0.13.0 Support: the UR no longer requires a local build of Mahout.
- GPU Support: via Mahout 0.13.0 the core math of the UR now supports the use of GPUs for acceleration.
- Timeout Protection: Queries for users with very large histories could cause a timeout. We now correctly limit the amount of user history that is used as per documentation, which will all but eliminate timeouts.
- Bug Fixes: The use of
blackListEvents as defined in
engine.json was not working for an empty list, which should and now does disable any blacklisting except explicit item blacklists contained in the query.
- Apache PIO Compatible: The first UR version compatible with Apache PredictionIO-0.10.0-incubating. All past versions do not work and should be upgraded to this. The ActionML build of PIO is permanently deprecated since it is merged with Apache PIO.
v0.4.2 Replaces 0.4.1
- Fixes bug when a
pio build failure triggered by the release of Apache PIO. If you have problems building v0.4.0 use this version. It is meant to be used with PredictionIO-0.9.7-aml.
- Requires a custom build of Apache Mahout: instructions on the doc site This is temporary until the next Mahout release, when we will update to 0.4.3 (uses predicitonio-0.9.7-aml) and 0.5.0 (which uses predictionio-0.10.0 from Apache)
- This version requires PredictionIO-0.9.7-aml found here.
- New tuning params are now available for each "indicator" type, making indicators with a small number of possible values much more useful—things like gender or category-preference. See docs for configuring the UR and look for the
- New forms of recommendations backfill allow all items to be recommended even if they have no user events yet. Backfill types include random and user defined. See docs for configuring the UR and look for the
- This version requires PredictionIO-0.9.7-aml from the ActionML repo here.
- Implements a moving time window if events: Now supports the
SelfCleanedDataSource trait. Adding params to the
DataSource part of
engine.json allows control of de-duplication, property event compaction, and a time window of event. The time window is used to age out the oldest events. Note: this only works with the ActionML fork of PredictionIO found in the repo mentioned above.
- Parameter changed:
backfillField: duration to accept Scala Duration strings. This will require changes to all engine.json files that were using the older # of seconds duration.
- Event-types used in queries: added support for indicator predictiveness testing with the MAP@k tool. This is so only certain mixes of user events are used at query time.
- Bug fix: which requires that the
typeName in engine.json is required be
"items", with this release the type can be any string.
- removed isEmpty calls that were taking an extremely long time to execute, results in considerable speedup. Now the vast majority of
pio train time is taken up by writing to Elasticsearch. This can be optimized by creating and ES cluster or giving ES lots of memory.
- a query with no item or user will get recommendations based on popularity
- a new integration test has been added
- a regression bug where some ids were being tokenized by Elasticsearch, leading to incorrect results, was fixed. NOTE: for users with complex ids containing dashes or spaces this is an important fix.
- a dateRange in the query now takes precedence to the item attached expiration and available dates.
- date ranges attached to items will be compared to the prediction servers current data if no date is provided in the query.
- date range filters implemented
- hot/trending/popular used for backfill and when no other recommendations are returned by the query
- filters/bias < 0 caused scores to be altered in v0.1.1 fixed in this version so filters have no effect on scoring.
- the model is now hot-swapped in Elasticsearch so no downtime should be seen, in fact there is no need to run
pio deploy to make the new model active.
- it is now possible to have an engine.json (call it something else) dedicated to recalculating the popularity model. This allows fast updates to popularity without recalculating the collaborative filtering model.
- Elasticsearch can now be in cluster mode
- ids are now exact matches, for v0.1.0 the ids had to be lower case and were subject to tokenizing analysis so using that version is not recommended.
- user and item based queries supported
- multiple usage events supported
- filters and boosts supported on item properties and on user or item based results.
- fast writing to Elasticsearch using Spark
- convention over configuration for queries, defaults make simple/typical queries simple and overrides add greater expressiveness.
This Software is licensed under the Apache Software Foundation version 2 license found here: http://www.apache.org/licenses/LICENSE-2.0