Hopsworks - Data-Intensive AI platform with a Feature Store

Overview

Give us a star if you appreciate what we do

What is Hopsworks?
Quick Start
Development and Operational ML on Hopsworks
Docs
Who’s behind Hopsworks?
Open-Source
Join the community
Contribute

What is Hopsworks?

Hopsworks and its Feature Store are an open source data-intensive AI platform used for the development and operation of machine learning models at scale.

Quick Start

bash <(curl -s https://repo.hops.works/installer/latest/hopsworks-installer.sh)

Recommended minimum specification.

The Hopsworks Installer takes roughly 1-2 hrs to complete, depending on your bandwidth.

Installer Quick Start with Azure CLI or GCP CLI

bash <(curl -s https://repo.hops.works/installer/latest/hopsworks-cloud-installer.sh)

If you have the Azure or GCP CLI utilities installed (on a Linux machine), then the Hopsworks Cloud Installer (Hopsworks-cloud-installer.sh) will both install Hopsworks and provision the VMs in one command.

Hopsworks IDE Plugin

To work with the Hopsworks IDE plugin for IntelliJ/PyCharm, you can install it directly from the plugins menu of the IDE or clone it and follow the README.

Build with Maven

mvn install

Maven uses yeoman-maven-plugin to build both the front-end and the backend. Maven first executes the Gruntfile in the yo directory, then builds the back-end in Java. The yeoman-maven-plugin copies the dist folder produced by grunt from the yo directory to the target folder of the backend.

You can also build Hopsworks without the frontend (for Java EE development and testing):

mvn install -P-web

Development and Operational ML on Hopsworks

You can develop and run Python, Spark, and Flink applications on Hopsworks - in Jupyter notebooks, as jobs, or even notebooks as jobs. You can build production pipelines with the bundled Airflow, and even run ML training pipelines with GPUs in notebooks on Airflow. You can train models on as many GPUs as are installed in a Hopsworks cluster and easily share them among users.

Docs

Hopsworks documentation includes a user-guide, Feature Store documentation, and an Administrator Guide. There is also dedicated documentation for the Hopsworks Feature Store.

Hopsworks REST API is documented with Swagger and hosted by SwaggerHub.

To build and deploy swagger on your own Hopsworks instance you can follow the instructions found in this guide.

Who’s behind Hopsworks?

Hopsworks started as an open-source collaborative project at KTH University, RISE, and has more recently been taken on by Logical Clocks. Several funding bodies have helped contribute to its development including: European Commission (FP7, H2020), EIT, SSF, Vinnova and Celtic-Next.

Open-Source

Hopsworks is available under the AGPL-V3 license. In plain English this means that you are free to use Hopsworks and even build paid services on it, but if you modify the source code, you should also release your changes and any systems built around it as AGPL-V3.

Join the community

Contribute

We are building the most complete and modular ML platform available in the market and we count on your support to continuously improve Hopsworks. Feel free to give us suggestions, report bugs and add features to our library anytime.

We’re the best in what we do and want our community to succeed as well.
Our many thanks to the top contributors of Hopsworks!

Enjoy!

Issues
  • [HOPSWORKS-1276] Anaconda import yml file not working, removal of explicitly enabling python kernel

    [HOPSWORKS-1276] Anaconda import yml file not working, removal of explicitly enabling python kernel

    …licit enabling of python kernel

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [x] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by robzor92 12
  • [HOPSWORKS-1070] Refactor HdfsLeDescriptorsFacade to return a random NN for bootstrapping the client

    [HOPSWORKS-1070] Refactor HdfsLeDescriptorsFacade to return a random NN for bootstrapping the client

    …ing the client

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [ ] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by robzor92 6
  • [HOPSWORKS-903] Jupyter cleanup and timer bugfixes

    [HOPSWORKS-903] Jupyter cleanup and timer bugfixes

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [x] Adds tests for the submitted changes (for bug fixes & features)
    • [x] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by robzor92 4
  • [HOPSWORKS-718] Add Feature Store to Hopsworks

    [HOPSWORKS-718] Add Feature Store to Hopsworks

    • Adds a REST API for the feature store service ([HOPSWORKS-720])
    • Adds a UI for the feature store service ([HOPSWORKS-719])
    • Adds tests for the feature store service
    • Adds managed training datasets to the feature store ([HOPSWORKS-839])

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [x] Adds tests for the submitted changes (for bug fixes & features)
    • [x] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue https://logicalclocks.atlassian.net/browse/HOPSWORKS-718

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...) Feature: Feature Store for HopsML

    • What is the new behavior (if this is a feature change)? A feature store service in hopsworks

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?) Hive becomes a first-class dependency to Hopsworks

    • Other information:

    Dependent on:

    • https://github.com/logicalclocks/hops-hadoop-chef/pull/119
    • https://github.com/logicalclocks/hopsworks-chef/pull/233
    • https://github.com/logicalclocks/spark-chef/pull/57

    Tests:

    https://github.com/logicalclocks/hops-testing/pull/248

    opened by Limmen 4
  • [HOPSWORKS-825] Remove member from project and jobs review

    [HOPSWORKS-825] Remove member from project and jobs review

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [x] Adds tests for the submitted changes (for bug fixes & features)
    • [ ] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

      • Data owner should be able to remove data scientists

      • Timers for jobs will be cancelled if the user is not a member of the project where the job is created

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by robzor92 4
  • [HOPSWORKS-1538] add endpoint to download the client archive for auto…

    [HOPSWORKS-1538] add endpoint to download the client archive for auto…

    … install on remote machine

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [ ] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue https://logicalclocks.atlassian.net/browse/HOPSWORKS-1538

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...) New feature

    • What is the new behavior (if this is a feature change)? add an endpoint to download the hopsfs and hive clients for remote access.

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by berthoug 3
  • [HOPSWORKS-1338] [featurestore] Configurable in-bucket path for exporting training datasets to S3

    [HOPSWORKS-1338] [featurestore] Configurable in-bucket path for exporting training datasets to S3

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [ ] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [ ] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue https://logicalclocks.atlassian.net/browse/HOPSWORKS-1338

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by moritzmeister 3
  • [HOPSWORKS-1284] add install lib without search

    [HOPSWORKS-1284] add install lib without search

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [x] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by ErmiasG 3
  • [HOPSWORKS-1124] optimize scp-deploy-frontend.sh

    [HOPSWORKS-1124] optimize scp-deploy-frontend.sh

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [ ] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue https://logicalclocks.atlassian.net/browse/HOPSWORKS-1124

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...) Improvement to the script.

    • What is the new behavior (if this is a feature change)? Reduced the time from ~90 seconds to ~20 seconds.

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by kai-chi 3
  • [HOPSWORKS-1008] added activation date to user admin and fix role assignment

    [HOPSWORKS-1008] added activation date to user admin and fix role assignment

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [x] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by ErmiasG 3
  • [HOPSWORKS-853] Feature Store Tour

    [HOPSWORKS-853] Feature Store Tour

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [x] Adds tests for the submitted changes (for bug fixes & features)
    • [x] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...) Feature

    • What is the new behavior (if this is a feature change)? [HOPSWORKS-853] Feature Store Tour

    • Adds feature store tour to hopsworks
    • The tour includes example notebooks and jobs for the feature store
    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?) No breaking changes

    • Other information:

    This PR depends on:

    • https://github.com/logicalclocks/spark-chef/pull/58
    • https://github.com/logicalclocks/hops-hadoop-chef/pull/120
    • https://github.com/logicalclocks/tensorflow-chef/pull/43
    • https://github.com/logicalclocks/hopsworks-chef/pull/236
    opened by Limmen 3
  • [HOPSWORKS-3066] [FeatureView] Add integration test for FeatureView

    [HOPSWORKS-3066] [FeatureView] Add integration test for FeatureView

    …source (#956)

    • init

    • small changes

    • additional tests

    • revert changes

    Co-authored-by: kenneth [email protected] (cherry picked from commit c9e77f26cd11794066b88b150e6bd0c4cfea1a8f)

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [ ] Passes the tests
    • [ ] HOPSWORKS JIRA issue has been opened for this PR
    • [ ] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by kennethmhc 0
  • [HOPSWORKS-3224] complex tags - free mode tag schemas

    [HOPSWORKS-3224] complex tags - free mode tag schemas

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [ ] Passes the tests
    • [x] HOPSWORKS JIRA issue has been opened for this PR
    • [x] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue https://hopsworks.atlassian.net/browse/HOPSWORKS-3224

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by o-alex 0
  • [HOPSWORKS-3209] id and meta expectationId field are not matching if meta already had expectationId

    [HOPSWORKS-3209] id and meta expectationId field are not matching if meta already had expectationId

    Make sure there is no duplicate PR for this issue

    • Please check if the PR meets the following requirements
    • [ ] Adds tests for the submitted changes (for bug fixes & features)
    • [ ] Passes the tests
    • [ ] HOPSWORKS JIRA issue has been opened for this PR
    • [ ] All commits have been squashed down to a single commit
    • Post a link to the associated JIRA issue

    • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

    • What is the new behavior (if this is a feature change)?

    • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

    • Other information:

    opened by moritzmeister 0
  • Bump kubernetes-client from 4.10.3 to 4.11.2

    Bump kubernetes-client from 4.10.3 to 4.11.2

    Bumps kubernetes-client from 4.10.3 to 4.11.2.

    Release notes

    Sourced from kubernetes-client's releases.

    4.11.2 (2021-02-09)

    Bugs

    • Fix #2715: CVE-2021-20218 vulnerable to a path traversal leading to integrity and availability compromise

    4.11.1 (2020-09-02)

    Bugs

    • Fix #2445: ConfigMap and other resources are replaced

    4.11.0 (2020-08-26)

    Bugs

    • Fix #2373: Unable to create a Template on OCP3
    • Fix #2308: Fix kubernetes client Config loading KUBECONFIG with external authentication command
    • Fix #2316: Cannot load resource from stream without apiVersion
    • Fix #2354: Fix NullPointerException in ResourceCompare when no resource is returned from fromServer.get()
    • Fix #2389: KubernetesServer does not use value from https in crud mode
    • Fix #2306: Make KubernetesServer CRUD mode work with informers
    • Fix #2418: CertificateSigningRequest doesn't implement Namespaced
    • Fix #2265: InAnyNamespace uses invalid api endpoint for SelfSubjectAccessReviews
    • Fix #2404: Readiness.isReady doesn't handle extensions/v1beta1 Deployment
    • Fix #2389: KubernetesServer JUnit rule ignores value of https when using crud mode

    Improvements

    • Fix #2331: Fixed documentation for namespaced informer for all custom types implementing Namespaced interface
    • Fix #2406: Add documentation for serializing resources to YAML

    Dependency Upgrade

    • Fix #2360: bump mockito-core from 3.4.0 to 3.4.2
    • Fix #2355: bump jandex from 2.1.3.Final to 2.2.0.Final
    • Fix #2353: chore: bump workflow action-setup versions + kubernetes to 1.18.6
    • Fix #2292: Update createOrReplace to do replace when create fails with conflict
    • Fix: Bump SnakeYaml to version 1.26 (as required for OSGi bundle for jackson-dataformat-yaml)
    • Fix #2401: bump maven-resources-plugin from 3.1.0 to 3.2.0
    • Fix #2405: bump mockito-core from 3.4.4 to 3.5.0

    New Features

    • CSI Volume Snapshot extension
    • Fix #2311: Add Support for creating bootstrap project template
    • Fix #2287: Add support for V1 and V1Beta1 CustomResourceDefinition
    • Fix #2319: Create Config without using auto-configure functionality or setting env variables
    • Fix #2284: Supports create and run a particular image in a pod operation using client
    • Fix #2321: Add Support for new resources in OpenShift Model

    Note: Some classes have been moved to other packages:

    • CustomResourceDefinition has been moved to io.fabric8.kubernetes.api.model.apiextensions.v1 and io.fabric8.kubernetes.api.model.apiextensions.v1beta1
    • SubjectAccessReview, SelfSubjectAccessReview, LocalSubjectAccessReview and SelfSubjectRulesReview have been moved to io.fabric8.kubernetes.api.model.authorization.v1 and io.fabric8.kubernetes.api.model.authorization.v1beta1
    • io.fabric8.tekton.pipeline.v1beta1.WorkspacePipelineDeclaration is now io.fabric8.tekton.pipeline.v1beta1.PipelineWorkspaceDeclaration
    Changelog

    Sourced from kubernetes-client's changelog.

    4.11.2 (2021-02-09)

    Bugs

    • Fix #2715: CVE-2021-20218 vulnerable to a path traversal leading to integrity and availability compromise

    4.7.2 (2021-02-09)

    Bugs

    • Fix #2715: CVE-2021-20218 vulnerable to a path traversal leading to integrity and availability compromise

    5.0.2 (2021-02-08)

    Bugs

    • Fix #2715: CVE-2021-20218 vulnerable to a path traversal leading to integrity and availability compromise

    4.13.2 (2021-02-05)

    Bugs

    • Fix #2715: CVE-2021-20218 vulnerable to a path traversal leading to integrity and availability compromise

    5.0.1 (2021-01-27)

    Bugs

    • Fix #2687: RawCustomResourceOperationsImpl ignores config
    • Fix #2612: add support for checking latest kubeconfig in Interceptor

    Improvements

    • Fix #2723: Dependency cleanup
    • Remove javax.annotation-api
    • Remove jaxb-api
    • Remove jackson-module-jaxb-annotations
    • Fix #2744: Automatically instantiates spec and status fields on CustomResource when possible. initSpec and initStatus methods are also provided to allow for overriding of the default implementation.

    Dependency Upgrade

    • Fix #2723: Upgrade sundrio.version to 0.24.1

    4.13.1 (2021-01-20)

    Bugs

    • Fix #2734: (Backport #2596) Add buildable ref for container, port and volume

    5.0.0 (2020-12-30)

    Bugs

    • Fix #2695: fromCustomResourceType should be of type CustomResource

    5.0.0-beta-1 (2020-12-21)

    ... (truncated)

    Commits
    • fee2cb0 [RELEASE] Updated project version to v4.11.2
    • fc3462b fix: CVE-2021-20218 adapt test to 4.11.x project structure
    • 690647c Merge branch '4.7.x' into 4.11.x
    • c9d712e [RELEASE] Updated project version to v4.7.2
    • f4d70bb fix: CVE-2021-20218 adapt test to 4.7.x project structure
    • 9273ec3 fix: CVE-2021-20218 vulnerable to a path traversal
    • 34b4483 [RELEASE] Updated project version to v4.11.1
    • 6dcebc9 Merge pull request #2453 from manusa/fix/config-map-replace
    • aa651fa fix: ConfigMap and other resources are replaced
    • c1e4862 Merge pull request #2429 from Shivkumar13/pr/fabricfixbaseoperation
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies java 
    opened by dependabot[bot] 0
  • Bump numpy from 1.19.0 to 1.22.0 in /hopsworks-IT/src/test/ruby/spec/auxiliary

    Bump numpy from 1.19.0 to 1.22.0 in /hopsworks-IT/src/test/ruby/spec/auxiliary

    Bumps numpy from 1.19.0 to 1.22.0.

    Release notes

    Sourced from numpy's releases.

    v1.22.0

    NumPy 1.22.0 Release Notes

    NumPy 1.22.0 is a big release featuring the work of 153 contributors spread over 609 pull requests. There have been many improvements, highlights are:

    • Annotations of the main namespace are essentially complete. Upstream is a moving target, so there will likely be further improvements, but the major work is done. This is probably the most user visible enhancement in this release.
    • A preliminary version of the proposed Array-API is provided. This is a step in creating a standard collection of functions that can be used across application such as CuPy and JAX.
    • NumPy now has a DLPack backend. DLPack provides a common interchange format for array (tensor) data.
    • New methods for quantile, percentile, and related functions. The new methods provide a complete set of the methods commonly found in the literature.
    • A new configurable allocator for use by downstream projects.

    These are in addition to the ongoing work to provide SIMD support for commonly used functions, improvements to F2PY, and better documentation.

    The Python versions supported in this release are 3.8-3.10, Python 3.7 has been dropped. Note that 32 bit wheels are only provided for Python 3.8 and 3.9 on Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. All 64 bit wheels are also linked with 64 bit integer OpenBLAS, which should fix the occasional problems encountered by folks using truly huge arrays.

    Expired deprecations

    Deprecated numeric style dtype strings have been removed

    Using the strings "Bytes0", "Datetime64", "Str0", "Uint32", and "Uint64" as a dtype will now raise a TypeError.

    (gh-19539)

    Expired deprecations for loads, ndfromtxt, and mafromtxt in npyio

    numpy.loads was deprecated in v1.15, with the recommendation that users use pickle.loads instead. ndfromtxt and mafromtxt were both deprecated in v1.17 - users should use numpy.genfromtxt instead with the appropriate value for the usemask parameter.

    (gh-19615)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies python 
    opened by dependabot[bot] 0
  • Bump mysql-connector-java from 8.0.21 to 8.0.28 in /hopsworks-IT

    Bump mysql-connector-java from 8.0.21 to 8.0.28 in /hopsworks-IT

    Bumps mysql-connector-java from 8.0.21 to 8.0.28.

    Changelog

    Sourced from mysql-connector-java's changelog.

    Changelog

    https://dev.mysql.com/doc/relnotes/connector-j/8.0/en/

    Version 8.0.29

    • Fix for Bug#21978230, COMMENT PARSING NOT PROPER IN PREPSTMT.EXECUTEBATCH().

    • Fix for Bug#81468 (23312764), MySQL server fails to rewrite batch insert when column name contains word select.

    • Fix for Bug#106435 (33850099), 8.0.28 Connector/J has regressive in setAutoCommit after Bug#104067 (33054827).

    • Fix for Bug#106240 (33781440), StringIndexOutOfBoundsException when VALUE is at the end of the query.

    • Fix for Bug#106397 (33893591), Contribution: fix: fix LocalizedErrorMessages.properties doc: less then -> ... Thanks to Jianjian Song for his contribution.

    • Fix for Bug#77924 (25710160), JDBC SOCKS SHOULD NOT PERFORM LOCAL DNS RESOLUTION.

    • Fix for Bug#82084 (23743938), YEAR DATA TYPE RETURNS INCORRECT VALUE FOR JDBC GETCOLUMNTYPE().

    • Fix for Bug#106441 (33850155), Add charset mapping for utf8mb3.

    • WL#15048, Upgrade Protocol Buffers dependency to protobuf-java-3.19.4.

    • Fix for Bug#106065 (33726184) Contribution: BigDecimal.toPlainString no need to check decimal exponent. Thanks to Baoyi Chen for his contribution.

    • Fix for Bug#106171 (33757217), Contribution: Remove unnecessary boxing in ResultSetImpl. Thanks to Ningpp Ning for his contribution.

    • Fix for Bug#25701740, STMT EXECUTION FAILS FOR REPLICATION CONNECTION WHEN USECURSORFETCH=TRUE.

    • Fix for Bug#33723611, getDefaultTransactionIsolation must return repeatable read.

    • Fix for Bug#38954 (11749415), DATA TRUNCATION WHILE USING BIT(1) IN STORED PROCEDURE WITH INOUT TYPE.

    • Fix for Bug#85317 (25672958), EXECUTE BATCH WILL THROW NULL POINTER EXCEPTION WHERE THE COLUMN IS BLOB!

    • Fix for Bug#105915 (33678490), Connector/J 8 server prepared statement precision loss in execute batch.

    • Fix for Bug#104349 (33563548), com.mysql.cj NPE.

    • Fix for Bug#62006 (16714956), JAVA.IO.NOTSERIALIZABLEEXCEPTION: JAVA.IO.STRINGREADER WHEN PROFILESQL=TRUE.

    • WL#14750, Better unification of query bindings.

    • WL#14834, Support for FIDO authentication.

    • WL#14835, Align TLS option checking across connectors.

    ... (truncated)

    Commits
    • 7ff2161 Updating copyright years
    • b13af38 Fix for DateTimeTest according to changes in MySQL server.
    • 5c7b775 Update in test for Bug#96900 (30355150).
    • e1169ee Fix for Bug#99260 (31189960), statement.setQueryTimeout,creates a database co...
    • 05778ef Fix for Bug#103324 (32770013), X DevAPI Collection.replaceOne() missing match...
    • 48219f2 Fix for Bug#105197 (33461744), Statement.executeQuery() may return non-naviga...
    • 24cf7e2 Fix for Bug#105323 (33507321), README.md contains broken links.
    • ad46620 Fix for Bug#96900 (30355150), STATEMENT.CANCEL()CREATE A DATABASE
    • 4d19ea1 Fix for Bug#104067 (33054827), No reset autoCommit after unknown issue occurs.
    • bc45d35 Fix for Bug#85223 (25656020), MYSQLSQLXML SETSTRING CRASH.
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies java 
    opened by dependabot[bot] 0
Releases(v2.5.0)
  • v2.5.0(Feb 13, 2022)

    Release notes - Hopsworks - Version 2.5.0

    Bug

    HOPSWORKS-2983 Projects can't be created without hops-system namespace in Kubernetes

    HOPSWORKS-2980 Allow multiple model files in a model artifact

    HOPSWORKS-2973 zip/unzip URI encoding calls system python, use hops-system python instead

    HOPSWORKS-2965 Remove pathlib, confluent-kafka, hops-petastorm, opencv-python as dependencies for hops-util-py

    HOPSWORKS-2963 NullPointerException when listing tags for model

    HOPSWORKS-2962 Components in Serving UI are not aligned

    HOPSWORKS-2953 init_prepared_statement throws NoneType exception

    HOPSWORKS-2952 update hsfs examples with train_split property

    HOPSWORKS-2951 hsml should also be monitored for minor releases

    HOPSWORKS-2950 Remove user, project and installation section from hopsworks documentation

    HOPSWORKS-2939 [tez] Tez master does not send application certificates with the create container request

    HOPSWORKS-2936 Pin ipython to 7.31.0

    HOPSWORKS-2935 jupyter_spec insufficient spark memory test should set python kernel to false

    HOPSWORKS-2924 cloud-metrics-daemon ownership is incorrect

    HOPSWORKS-2920 exclude org.slf4j:slf4j-log4j12 from zookeeper

    HOPSWORKS-2893 SYNC_BASE_ENV should also check for conflicts

    HOPSWORKS-2884 DefaultJobConfiguration update fails for PySpark/Spark JobType

    HOPSWORKS-2873 Log4j2 vulnerability mitigation

    HOPSWORKS-2869 Improve Spark/PySpark logging

    HOPSWORKS-2863 parallel conversion of same notebook - add execution id to make name unique

    HOPSWORKS-2856 OnlineFS user removal during project removal can result in collision

    HOPSWORKS-2839 Fix batch serving vectors sql syntax

    HOPSWORKS-2838 Training dataset get_query for offline storage not working on hive engine

    HOPSWORKS-2826 OnlineFS replays data on cluster restart

    HOPSWORKS-2818 Serialize Date objects correctly in Query Constructor

    HOPSWORKS-2817 settings.updateOauthLogoutRedirectUri add a path to the uri

    HOPSWORKS-2814 Enabling feature store service on an existing project doesn't create online users for existing members

    HOPSWORKS-2813 Should be able to scale out a running serving

    HOPSWORKS-2812 Kafka failing to query the database

    HOPSWORKS-2809 Fix IllegalAnnotationsException that is polluting the log

    HOPSWORKS-2805 Regex for validating library name does not allow square brackets

    HOPSWORKS-2800 onlinefs service fails silently to start if Kafka has not fully started yet

    HOPSWORKS-2781 Fix timestamp features for Hudi Feature Groups

    HOPSWORKS-2779 OAuth callback redirect not working when user is already logged in

    HOPSWORKS-2777 executions table grows infinitely and may kill RonDB when deleting a Job or Project

    HOPSWORKS-2770 Feature Group clear content endpoint rethinking

    HOPSWORKS-2721 Data validation inequality rules should compare columns but accept fractional values

    HOPSWORKS-2698 Update hops::fuse_mnt for logging and bug fixes

    HOPSWORKS-2592 Dataset href path is malformed

    HOPSWORKS-2590 Storage connector should not set empty values

    HOPSWORKS-2398 KVM ruby sorting int as string.

    HOPSWORKS-2239 NullPointerException when creating a feature group with Null description

    HOPSWORKS-2171 Online Feature Group creation without schema through UI fails with uninformative Error

    HOPSWORKS-2097 Materialize font awesome icons

    HOPSWORKS-2030 Hopsworks Airflow operator doesn't work with python jobs

    HOPSWORKS-1867 flyway_schema_history engine should be ndbcluster instead of innodb

    HOPSWORKS-1351 Services versions list needs to be updated

    Improvement

    HOPSWORKS-2955 default train_split property to "train"

    HOPSWORKS-2948 Simplify getting shared feature store by name on HSFS

    HOPSWORKS-2940 Use consul for livy.rsc.launcher.address configuration

    HOPSWORKS-2933 Improve HSFS Python upload memory usage

    HOPSWORKS-2926 Adding vscode specific folders to .gitignore

    HOPSWORKS-2925 Move SqlCondition and SqlFilterLogic Enums to Persistance Package to make them reusable

    HOPSWORKS-2916 use tf.io.parse_example instead of tf.io.parse_single_example

    HOPSWORKS-2902 Fix for user management new UI

    HOPSWORKS-2899 feature group data preview should default to online feature store

    HOPSWORKS-2891 Add tags as a subresource for Models

    HOPSWORKS-2888 Support for pyodbc in python environment

    HOPSWORKS-2887 OAuth not working if 2fa set to mandatory

    HOPSWORKS-2882 Vulnerability check failing in jenkins

    HOPSWORKS-2880 Update prometheus version

    HOPSWORKS-2872 Migrate auth and project

    HOPSWORKS-2865 Disallow TLSv1 and TLSv1.1 at kagent web interface

    HOPSWORKS-2853 Save OAuth id_token to be used for logout.

    HOPSWORKS-2850 Fine-grained activity logging for validations

    HOPSWORKS-2843 default data storage storage policy

    HOPSWORKS-2842 Improve recent notebooks endpoint

    HOPSWORKS-2836 Add retention policy for services logs

    HOPSWORKS-2828 Support serving scikit-learn models and custom predictor scripts with KFServing

    HOPSWORKS-2827 New module for managing model serving (HSML)

    HOPSWORKS-2820 [consul-chef] Keep existing search domain in resolv.conf

    HOPSWORKS-2819 Allow feature comparison in Query Filters

    HOPSWORKS-2810 Fixes for oauth new UI

    HOPSWORKS-2808 Fix home directory

    HOPSWORKS-2804 Remove zookeeper rubygems dependency

    HOPSWORKS-2803 Add prometheus rules for RonDb alerting

    HOPSWORKS-2799 Use logging module in hops-util-py

    HOPSWORKS-2797 Add model name and serving state filters to ServingService

    HOPSWORKS-2784 bump up dependency versions to fix vulnerabilities

    HOPSWORKS-2780 Configuration parameter for InitialLogFileGroup and InitialTablespace

    HOPSWORKS-2774 Rest endpoints for configuring remote authentication

    HOPSWORKS-2772 Alert on job fail not working

    HOPSWORKS-2771 Include vnd.schemaregistry.v1+json header for kafka registry POST endpoints

    HOPSWORKS-2766 Add template_fields to the Airflow HopsworksAbstractOperator

    HOPSWORKS-2759 Set HttpOnly and Secure flags for cookies

    HOPSWORKS-2745 New API for managing machine learning models

    HOPSWORKS-2703 Improve documentation for model serving

    HOPSWORKS-2633 Support for Istio authentication

    HOPSWORKS-2623 Add Feature normalizer functions as in-built Transformation Functions to HSFS

    HOPSWORKS-2617 add batch lookup API to retrieve batch of serving vectors from online feature store

    HOPSWORKS-2504 Move feature descriptions from Hive Metastore to Hopsworks

    HOPSWORKS-2395 Change s3 connector bucket name tooltip

    HOPSWORKS-1882 Sparkmagic should send certificate to Livy

    New Feature

    HOPSWORKS-2919 Refresh online storage connector if connection gets dropped

    HOPSWORKS-2917 Rest API for editing Hopsworks configuration

    HOPSWORKS-2829 Expandable links provenance graph

    HOPSWORKS-2811 Add ModelRegistry API to fully support shared models

    HOPSWORKS-2802 Make it possible to manually set oauth session end point even if provider metadata end point is supported

    HOPSWORKS-2750 Hopsworks GIT integration

    HOPSWORKS-2680 Save code that it's used to create a feature group

    HOPSWORKS-2382 Collect service logs on Elastic

    HOPSWORKS-1960 Add support for downloading a model using the python API

    Task

    HOPSWORKS-2954 Add Old Documentation header to mkdocs.yml to point to readthedocs

    HOPSWORKS-2930 Update schema and model_schema paths in test notebooks

    HOPSWORKS-2929 Add zip package to hopsworks::install recipe

    HOPSWORKS-2913 Bump Hops dependencies to 3.2.0.6

    HOPSWORKS-2909 Reduce hopsworks-ear size

    HOPSWORKS-2896 Run Airflow in Docker container

    HOPSWORKS-2895 Add endpoint to generate elk service log user jwt token

    HOPSWORKS-2894 adjust model_spec tests to check new input_example format

    HOPSWORKS-2868 Model schema and input example should be expandable

    HOPSWORKS-2860 Rename Signature to ModelSchema in ModelDTO

    HOPSWORKS-2837 Extend query field size for SQL based on-demand feature group

    HOPSWORKS-2835 Zookeeper recipe does not respect external_user option

    HOPSWORKS-2833 Add documentation template

    HOPSWORKS-2821 Add user scope API key for secrets APIs

    HOPSWORKS-2776 Bump hops dependencies to 3.2.0.6-SNAPSHOT

    HOPSWORKS-2773 Maven dependency refactoring and vulnerability fixes

    HOPSWORKS-2769 Fix error management for python library installation

    HOPSWORKS-2767 Update to Python 3.8

    HOPSWORKS-1755 Upgrade PyTorch to 1.5

    HOPSWORKS-1416 Support for bulk activating users in the Admin UI

    Source code(tar.gz)
    Source code(zip)
  • v2.4.0(Oct 26, 2021)

    Release notes - Hopsworks - Version 2.4.0

    Bug

    HOPSWORKS-2770 Feature Group clear content endpoint rethinking

    HOPSWORKS-2748 Kibana and Grafana consul health checks are wrong

    HOPSWORKS-2738 Artifact files should not have glassfish as owner

    HOPSWORKS-2730 Proxy servlet is leaking resources

    HOPSWORKS-2717 Avro fails with schema names starting with numeric characters

    HOPSWORKS-2710 Wrong permissions for Airflow subdirectories

    HOPSWORKS-2696 Cannot store features of tinyint offline type as avro serialization fails

    HOPSWORKS-2682 node_exporter is not added to kagent services file if kagent is not enabled

    HOPSWORKS-2669 Nullpointerexception when trying to get an environment that doesn't exists

    HOPSWORKS-2665 Use Hopsworks internal port when calling generate_service_jwt in chef

    HOPSWORKS-2663 Fix docker installation on ubuntu 18.04

    HOPSWORKS-2662 Fix angular-material version

    HOPSWORKS-2659 [hopsworks-chef] Chef guard has incorrect syntax

    HOPSWORKS-2656 Incorrect payload when creating sklearn servings with new artifacts

    HOPSWORKS-2644 Non HOPS_ADMIN users should not be able to access the cluster settings.

    HOPSWORKS-2642 EditableSelect listening to keyboard event on window.

    HOPSWORKS-2641 Project page not loading after login

    HOPSWORKS-2640 Cannot start Jupyter if Kafka is offline

    HOPSWORKS-2638 elastic clear cached scrolling contexts

    HOPSWORKS-2620 Add shared by and accepted by to dataset sharing requests

    HOPSWORKS-2213 feature group commit endpoints doesn't fall back to pagination defaults

    Improvement

    HOPSWORKS-2768 Add private_ips attribute in consul-chef metadata.rb

    HOPSWORKS-2736 Add versioning for KFServing tars in the archive

    HOPSWORKS-2731 On-demand feature group should return primary key information

    HOPSWORKS-2720 KfServing storage initializer should use Hopsworks internal port

    HOPSWORKS-2712 Add support for IN condition in query filter

    HOPSWORKS-2709 Add endpoint to retrieve the most recent ran jupyter notebooks

    HOPSWORKS-2708 Add option to disable Jupyter shutdown

    HOPSWORKS-2701 improve Jobs REST API filters

    HOPSWORKS-2695 Replace creator with UserDTO in FeaturestoreEntityDTO

    HOPSWORKS-2692 Disable Yarn application retry

    HOPSWORKS-2683 Anaconda environment should be exported after every operation

    HOPSWORKS-2681 Profiling optimization

    HOPSWORKS-2667 Disable projects with no feature store

    HOPSWORKS-2661 Sort Feature Group columns by hive metastore index

    HOPSWORKS-2660 No user feedback when jupyter is not launched due to cpu/memory constraints

    HOPSWORKS-2654 Attaching an expectation without features should try to use all the FG features

    HOPSWORKS-2653 Flyway dir already exists error in hopsworks::install

    HOPSWORKS-2606 compute training dataset statistics for train,test and validation separately

    HOPSWORKS-2598 Add application field in snowflake storage connector

    HOPSWORKS-2453 Feature validations - turn on by default

    HOPSWORKS-1832 [ePipe] Handle connection errors with NDB

    New Feature

    HOPSWORKS-2518 Point-In-Time Joins

    Task

    HOPSWORKS-2765 Remove Sqoop

    HOPSWORKS-2754 Bump hops dependencies to 3.2.0.5-RC0

    HOPSWORKS-2753 Install ca-certificates in base image as required by nodejs

    HOPSWORKS-2737 Pre-install flask in airflow-chef

    HOPSWORKS-2723 Remove AUDITOR role

    HOPSWORKS-2705 Deterministic UID/GID

    HOPSWORKS-2693 Bump minor version of service proxy

    HOPSWORKS-2689 Increase max file descriptors for mysql unit

    HOPSWORKS-2684 Remove ndb_backup from Hopsworks

    HOPSWORKS-2650 Bump hops dependencies to 3.2.0.5-SNAPSHOT

    HOPSWORKS-2155 Set up multi-repo documentation for docs.hopsworks.ai

    Source code(tar.gz)
    Source code(zip)
  • v2.3.0(Aug 24, 2021)

    Release notes - Hopsworks - Version 2.3.0

    Bug

    HOPSWORKS-2663 Fix docker installation on ubuntu 18.04

    HOPSWORKS-2639 NPE with no Databricks Clusters in Feature Store Integration

    HOPSWORKS-2634 Feature order not respected in training dataset prepared statement

    HOPSWORKS-2631 SYNC_BASE_ENV operation should also show error logs in case of failure

    HOPSWORKS-2628 OnlineFS schema of feature groups with appended features misses old features

    HOPSWORKS-2626 MySQL timezone issue

    HOPSWORKS-2625 Add status to dataset projectsSharedWith

    HOPSWORKS-2619 Add missing JVM imports in Livy for Spark 3

    HOPSWORKS-2618 NPE when force deleting a Project

    HOPSWORKS-2614 Training Dataset from Python with default connector fails

    HOPSWORKS-2609 Unshare a pending feature store fails

    HOPSWORKS-2594 Attaching Jupyter config when running notebook with Spark or Python kernel with git backend fails

    HOPSWORKS-2583 Experiment mnist notebook should work in airgapped environment

    HOPSWORKS-2582 Spark driver cores is incorrectly set to number of configured executor cores

    HOPSWORKS-2577 Python Feature Group Ingestion fails with additional write arguments

    HOPSWORKS-2574 Feature validation rule Level.Warning defaults to Error

    HOPSWORKS-2569 Allow capital letters keywords

    HOPSWORKS-2567 Nullpointerexception when retrieving validations for on-demand feature groups

    HOPSWORKS-2556 Jupyter should use default DockerJobConfiguration for Experiments and Spark

    HOPSWORKS-2555 KAFKA_BROKERS not set in Python kernel

    HOPSWORKS-2554 oauth2 get user bug

    HOPSWORKS-2553 Fix kfserving airgapped deployment

    HOPSWORKS-2546 Add RonDB services to kagent even if disabled

    HOPSWORKS-2538 Secrets API not setting projectId and visibility in certain cases

    HOPSWORKS-2536 Online FS Avro schema constructor decimals should take precision and scale

    HOPSWORKS-2526 add fg to prepared statement only if it contains features other than pk and targets.

    HOPSWORKS-2520 Installing python libraries with no version and from git should use --no-cache docker build option

    HOPSWORKS-2517 Job executions not filtered by submission time

    HOPSWORKS-2515 Set spark.serializer to Kryo for Databricks integration

    HOPSWORKS-2510 Updating validationType on feature group response contains old value

    HOPSWORKS-2503 Pin containerd version in ubuntu

    HOPSWORKS-2495 HSFS connection cannot not read JWT in Python jobs

    HOPSWORKS-2490 Disable Python Library Updates Monitor for snapshot versions

    HOPSWORKS-2386 Decimal types are converted to varbinary on the online feature grouip

    HOPSWORKS-2349 Hive InodeHelper doesn't work if path contains slahs (/)

    HOPSWORKS-2343 Validate storage connector type before creating a training dataset

    Improvement

    HOPSWORKS-2653 Flyway dir already exists error in hopsworks::install

    HOPSWORKS-2645 Support feature validation for on-demand FGs in hsfs

    HOPSWORKS-2643 Add option to enable hopsfs/rm audit logs

    HOPSWORKS-2636 Add option to specify Kubernetes node labels from cluster definition

    HOPSWORKS-2615 Add option to redirect stdout and stderr of Docker jobs

    HOPSWORKS-2607 Set NotValidBefore of issued X.509 a few minutes earlier

    HOPSWORKS-2605 Configuration parameter for Kubernetes maximum retry number

    HOPSWORKS-2604 add option to specify additional insecure docker registry

    HOPSWORKS-2597 Add hops Python helper methods for model and dataset modules

    HOPSWORKS-2584 Increase Schema Registry database schema column size

    HOPSWORKS-2578 Use a separate Data directory to store Hopsworks state

    HOPSWORKS-2575 Make Online FS Kafka topic partition number configurable

    HOPSWORKS-2560 Support Memory, Cores and GPU as a resource for model serving

    HOPSWORKS-2559 Add input paths option for Docker jobs and multiple commands

    HOPSWORKS-2552 Send monitoring URLs as part of execution dto

    HOPSWORKS-2551 Bump hops dependencies to 3.2.0.4-SNAPSHOT

    HOPSWORKS-2529 Package dockerfile:experimental in the base.tar for

    HOPSWORKS-2528 Add support to provide PIP extra-index-url in the cluster definition

    HOPSWORKS-2523 Use own java cookbook instead of upstream

    HOPSWORKS-2506 Feature Validation Improvements

    HOPSWORKS-2502 Add JOB audience to delete endpoint for training datasets

    HOPSWORKS-2492 Improve readiness check of KFServing inference services

    HOPSWORKS-2356 Alerting for hopsworks services

    HOPSWORKS-2167 Add inference logging sidecar to KFServing deployments

    New Feature

    HOPSWORKS-2381 Project level configuration for Jupyter notebooks and Jobs

    HOPSWORKS-2334 Support custom transformers in KFServing deployments

    HOPSWORKS-2319 Add API to interact with the secret store

    HOPSWORKS-2104 Model Serving with KFServing

    Task

    HOPSWORKS-2637 Bump hops dependencies to 3.2.0.4-RC0

    HOPSWORKS-2622 pin six to 1.15.0

    HOPSWORKS-2621 Bump pip in hops-system to last version that supports python 2.7

    HOPSWORKS-2601 Update swagger code to be compatible with the latest swagger hub and enable for demo cluster

    HOPSWORKS-2498 Remove filebeat beam recipes

    HOPSWORKS-2424 Attach transformation function to training dataset

    HOPSWORKS-2420 Bump hops dependencies to 3.2.0.3-SNAPSHOT

    HOPSWORKS-1942 Upgrade to Spark 3.x

    Source code(tar.gz)
    Source code(zip)
  • v2.2.0(Apr 9, 2021)

    Release notes - Hopsworks - Version 2.2.0

    Bug

    HOPSWORKS-2482 Read Hudi Feature Group from Hive Engine fails with unknown fg0

    HOPSWORKS-2481 Only default deployments must be enabled when KFServing is not installed

    HOPSWORKS-2473 dataset read acl list does not get cleared on dataset delete

    HOPSWORKS-2471 fix prepared statement parameter for string PKs

    HOPSWORKS-2447 Pip conflicts modal not closing when clicking confirm

    HOPSWORKS-2415 reset elastic client in case it uses the wrong certificate

    HOPSWORKS-2414 Verify that API token used when starting Jupyter with AutoPushOnShutdown enabled has write access to git repo

    HOPSWORKS-2352 Snowflake connector requires pytz <= 2021.0

    HOPSWORKS-2303 Change python pip search to use pypi simple index and JSON API

    HOPSWORKS-2293 Fix test for kvm

    Improvement

    HOPSWORKS-2484 Fix SSL/TLS settings for Glassfish network listeners

    HOPSWORKS-2460 Create admin end-point for user management

    HOPSWORKS-2433 Azure Root CA missing from glassfish trust store.

    HOPSWORKS-2432 TLS for Resource Manager UI

    HOPSWORKS-2423 Advertise Rondb managment node through consul

    HOPSWORKS-2422 Advertise Kafka brokers through Consul

    HOPSWORKS-2421 [ndb-chef] Fix config.ini for RonDB

    HOPSWORKS-2408 Feature store activity should return validation results

    HOPSWORKS-2405 Upgrade Python version we install in recipes

    HOPSWORKS-2404 remove metadata designer code

    HOPSWORKS-2396 Add AzureML Studio notebook

    HOPSWORKS-2368 Do not require API token for working with public GitHub and Gitlab repos using Jupyter

    HOPSWORKS-2367 Hopsworks Connector Docs

    HOPSWORKS-2366 Replace service with systemctl in recipes

    HOPSWORKS-2351 derived fg provenance tests - links/usage

    HOPSWORKS-2345 Get Kafka Broker Endpoints through REST Api

    HOPSWORKS-2336 It should not be mandatory to provide a version when installing a pip or conda library

    HOPSWORKS-2314 Snowflake connector

    HOPSWORKS-2247 Replace feature store job tracking with provenance

    HOPSWORKS-2139 Use Hudi commit ID as statistics timestamp

    HOPSWORKS-2071 Data validation v2

    HOPSWORKS-2052 Save preferences for dataset and experiments filter

    New Feature

    HOPSWORKS-2474 Docker job

    HOPSWORKS-2458 Online-FS Service Integration

    HOPSWORKS-2399 Add new frontend deployment

    HOPSWORKS-2384 Add support for coalesce in training dataset

    HOPSWORKS-2195 Attach Jupyter configuration to notebook as Xattr

    HOPSWORKS-1978 schematized tags

    Sub-task

    HOPSWORKS-2193 Add support for request batching

    HOPSWORKS-2185 Implement inference controller

    HOPSWORKS-2106 Implement storage initializer webhook for KFServing

    Task

    HOPSWORKS-2476 Bump hops dependencies to 3.2.0.3-RC0

    HOPSWORKS-2348 Disable storage and compute quota by default

    HOPSWORKS-2342 Add PyTorch Maggy hyperparameter optimization example notebook

    HOPSWORKS-2326 Display alert if a more recent version of hops or hsfs exists in python service

    HOPSWORKS-2317 Remove featurestore module from hops-util-py and hops-util

    HOPSWORKS-2305 Remove security questions

    Source code(tar.gz)
    Source code(zip)
  • v2.1.0(Feb 12, 2021)

    Release notes - Hopsworks - Version 2.1.0

    Bug

    HOPSWORKS-2359 Always show GPU configuration fields for Jobs and Jupyter HOPSWORKS-2358 Remove ROCM GPU monitoring code HOPSWORKS-2346 Identifying if a project is using the base image on ECR does not work HOPSWORKS-2338 handle empty tfrecord files in TFDataEngine HOPSWORKS-2333 create_training_dataset method throws Connector not found exception HOPSWORKS-2311 Template required ec2-init parameters for the unmanaged version HOPSWORKS-2307 Docker garbage collection should not remove base or python37 images for any hopsworks version HOPSWORKS-2302 Hide python pip search bar as underlying pip search is now deprecated HOPSWORKS-2294 Installing a python library from git should enclose the url in quotes HOPSWORKS-2283 Reading from the online feature store doesn't work on Databricks HOPSWORKS-2249 provenance elasticsearch - count and aggregations should be separate methods HOPSWORKS-2248 Python file path is not shown in Job details HOPSWORKS-2238 Feature Group schema primary key info not set for partition keys HOPSWORKS-2219 Remove NaN from statistics HOPSWORKS-2202 Docker registry caches deleted manifests HOPSWORKS-2199 Glassfish consul domain should be included in Kafka super.users HOPSWORKS-2198 Cannot get the schema of a shared kafka topic HOPSWORKS-2187 Add anaconda user to hadoop group only if the group exists. Create hops-system Anaconda env conditionally HOPSWORKS-2182 Disable unattended-upgrades in Ubuntu HOPSWORKS-2174 issues attaching xattr to shared files HOPSWORKS-2164 Add API Key annotation on the commits endpoint HOPSWORKS-2159 ruby tests fix HOPSWORKS-2158 Query constructor ignores more than 2 joining conditions HOPSWORKS-2151 Tag validation doesn't work for key only tags HOPSWORKS-2145 Flask (sklearn) race condition when reading pid file HOPSWORKS-2102 Conversion of Jupyter notebooks to Python programs should consider notebook kernel HOPSWORKS-1984 Exception when creating online feature group without primary key

    Task

    HOPSWORKS-2348 Disable storage and compute quota by default HOPSWORKS-2337 Upgrade hudi to 0.7.0 version HOPSWORKS-2327 Ui fixed for generating training datasets HOPSWORKS-2324 Display alert if project is using older docker image HOPSWORKS-2320 Set PARQUET as default for non-hudi offline feature groups HOPSWORKS-2287 Bump up docker.io version for ubuntu HOPSWORKS-2275 Remove InodeEncodingStatus Bean HOPSWORKS-2274 Feature store "NOT FOUND" errors should return 404 HOPSWORKS-2241 Consul security recipe should not try to get Hopsworks IP if not needed HOPSWORKS-2206 HSFS profile to install with and without Hive dependencies HOPSWORKS-2196 Remove hudiEnabled flag and sync endpoint from hopsworks HOPSWORKS-2189 Update spark-avro package for to_avro/from_avro methods HOPSWORKS-2188 Remove partition key requirements for HUDI feature groups and allow users to set precombine keys HOPSWORKS-2178 Remove leftover ldconfig in hops::install HOPSWORKS-2163 Add missing chef changes of HOPSWORKS-2057 in master HOPSWORKS-2144 Bump hops dependencies to 3.2.0.2-RC0 HOPSWORKS-2138 remove slider and LLAP HOPSWORKS-2132 Check that base environment does not contain broken requirements when building docker HOPSWORKS-2131 Add pip check command to Python service HOPSWORKS-2127 Bump TensorFlow to 2.4 and PyTorch to 1.7 HOPSWORKS-2103 Add documentation for connecting to the Hopsworks Kafka cluster from an external client

    Improvement

    HOPSWORKS-2341 allow hsfs to get all metadata from feature store within experiment function HOPSWORKS-2315 [hops-hadoop-chef] Template in Chef application key size HOPSWORKS-2280 Save statistics configuration also for training datasets HOPSWORKS-2277 Add VARBINARY(N) to proposed types for online features HOPSWORKS-2201 Move to delete API for removing images from local docker registry HOPSWORKS-2200 set fs.azure.skipUserGroupMetadataDuringInitialization to true for ADLSv2 HOPSWORKS-2183 Add feature group, training dataset, storage connector counters to FeaturestoreDTO summary HOPSWORKS-2180 Disable transactions when Spark writes to the online feature store HOPSWORKS-2177 extend on-demand feature groups parquet/orc/delta/hudi/csv tables HOPSWORKS-2176 Add API key annotations to provenance endpoint HOPSWORKS-2172 Reduce default varchar size for string features HOPSWORKS-2168 Add keyword support to feature groups and training datasets HOPSWORKS-2157 Store information about primary and precombine keys for time travel enabled feature groups separately from Hive Metastore HOPSWORKS-2143 experiments/models list for any user with access (including shared) HOPSWORKS-2129 Should run conda clean after installing a package HOPSWORKS-2120 enable full provenance by default HOPSWORKS-2119 Create an admin rest endpoint for user registration HOPSWORKS-2101 Add resource to download files using api key HOPSWORKS-2026 Add "converting notebook" job state HOPSWORKS-1817 Replace influxdb with prometheus + pushgateway for Spark applications HOPSWORKS-1591 UI improvements for storage connectors HOPSWORKS-1193 Make deleting a project more intuitive HOPSWORKS-288 Allow editing a job's configuration property without clearing the rest

    New Feature

    HOPSWORKS-2197 Add ADLS storage connector for the feature store HOPSWORKS-2184 Install Python libraries from Anaconda environment file HOPSWORKS-2165 Support installing python libraries using requirements.txt HOPSWORKS-2134 Extend Query constructor with filter capability HOPSWORKS-2118 Add download method to hops-util-py HOPSWORKS-2085 Add assume role support for redshift connector HOPSWORKS-1971 HSFS Python support

    Source code(tar.gz)
    Source code(zip)
  • v2.0.0(Feb 1, 2021)

    Release notes - Hopsworks - Version 2.0.0

    Task

    HOPSWORKS-2287 Bump up docker.io version for ubuntu HOPSWORKS-2186 Remove installation of slack operator in Airflow cookbook HOPSWORKS-2161 Option to disable just the Registration UI HOPSWORKS-2156 Pin all TensorFlow projects in Docker image HOPSWORKS-2126 Cleanup old hops-util(-py) feature store notebooks HOPSWORKS-2125 Bump hops dependencies to 3.2.0.1-RC0 HOPSWORKS-2109 Remove base_env from PythonDep HOPSWORKS-2094 Setup GPU monitoring without enabling it even when it's not a GPU deployment HOPSWORKS-2090 Bump up docker.io version for Ubuntu HOPSWORKS-2072 [hops-hadoop-chef] Recipe should check if Hops distro has been build with correct profiles HOPSWORKS-2043 Upgrade base and airflow conda environment to python 3.7 HOPSWORKS-2042 Unmanaged Hopsworks Image HOPSWORKS-2041 Bump Hops Hadoop version to 3.2.0.1-SNAPSHOT HOPSWORKS-2040 Docker registry does not need to be versioned HOPSWORKS-2039 Bump hops dependencies to 3.2.0.0-RC3 HOPSWORKS-2037 Rename map_fun and wrapper_fun to train_fn HOPSWORKS-1945 Store AWS access/secret key in the secret store. HOPSWORKS-1944 Split feature store tests in several categories HOPSWORKS-1387 Upgrade to NDB 8

    Bug

    HOPSWORKS-2202 Docker registry caches deleted manifests HOPSWORKS-2182 Disable unattended-upgrades in Ubuntu HOPSWORKS-2162 Airflow page does not open HOPSWORKS-2151 Tag validation doesn't work for key only tags HOPSWORKS-2150 S3 storage connector should not bother about secrets if iam role is configured HOPSWORKS-2149 Disable generating and sharing ssh key freely HOPSWORKS-2147 Add feature group api scope to statistics resource endpoint HOPSWORKS-2146 Resolve namenode consul name before setting feature group location HOPSWORKS-2145 Flask (sklearn) race condition when reading pid file HOPSWORKS-2128 Create api key not adding the right scope HOPSWORKS-2122 Backend not handling null TimeTravelFormat sent by client HOPSWORKS-2116 Add --network option to docker build HOPSWORKS-2095 Uninstalling conda libraries doesn't work in airgapped environments HOPSWORKS-2092 Overwriting a feature group by dropping/recreating loses default value of appended features HOPSWORKS-2081 Fix search for registration disabled HOPSWORKS-2080 Hops services should wait for Hopsworks before they start HOPSWORKS-2068 Singleton initialization failure HOPSWORKS-2067 shared featuregroup creation HOPSWORKS-2065 Restart systemd units after failure HOPSWORKS-2064 UI fixes for data preview HOPSWORKS-2062 ProjectException should set prefix usrMsg from the causing exception's messge HOPSWORKS-2061 Datasets browser order files by name not working while paginating HOPSWORKS-2058 Conda environments are not removed when deleting a project HOPSWORKS-2056 Change DNS server for docker containers in Localhost deployments HOPSWORKS-2054 Increment timeout when launching Jupyter locally HOPSWORKS-2053 Fix systemd dependencies HOPSWORKS-2051 Save preferences for jobs/page filter HOPSWORKS-2048 Jupyter-git missing configuration for git_backend HOPSWORKS-2047 Non LTS Nvidia Driver version doesn't work on Azure VMs HOPSWORKS-2038 Python client can't handle disabled online feature store HOPSWORKS-2015 Users should not be allowed to create Dataset 'Projects' HOPSWORKS-1988 IPynb converter fails for notebooks which contain special characters in the filename HOPSWORKS-1918 Spark executor memory not modifiable from UI when starting jupyter HOPSWORKS-1876 Data validation UI doesn't fill in Job parameters HOPSWORKS-1861 API keys not all selectable for Github Jupyter notebooks

    Improvement

    HOPSWORKS-2140 Increase retries for HDFS Mover in hops-hadoop-chef HOPSWORKS-2130 Refactor on-demand feature groups for HSFS HOPSWORKS-2124 Remove R installation from spark::yarn HOPSWORKS-2123 Add option to disable datasource validation when deploying Hopsworks HOPSWORKS-2113 Add endpoint to list a user's projects HOPSWORKS-2112 Return the user activation time as part of the userProfileDTO HOPSWORKS-2093 remove env/pkgs dir symlinks in conda installation HOPSWORKS-2091 Label/Prediction feature metadata for training datasets HOPSWORKS-2084 Remove default_storage option from feature groups HOPSWORKS-2077 add support for reading csv files as tf data object and rename feeder to tf_data HOPSWORKS-2046 Kafka broker should use zookeeper fqdn instead of IP HOPSWORKS-2045 artifact usage HOPSWORKS-2044 Add option to return an error message if a user starts a YARN job but no nodemanagers are available HOPSWORKS-2025 Delete a job's executions HOPSWORKS-2010 Upgrade hudi to 0.6.1-SNAPSHOT for hsfs HOPSWORKS-1986 Add redshift/snowflake examples and libraries HOPSWORKS-1982 Deequ statistics for Feature Groups/Training Datasets HOPSWORKS-1878 Add default args for Jobs HOPSWORKS-1847 online feature store storage connectors should not be stored in the database HOPSWORKS-1833 Move beam related files from conf directory to lib HOPSWORKS-1769 enable publishing models to other projects HOPSWORKS-1374 Add Hudi API to HSFS and commits to the FeatureStore metadata

    New Feature

    HOPSWORKS-2014 Manage AWS roles per application in Hopsworks HOPSWORKS-2009 Append Features to Feature Group schema HOPSWORKS-1853 Support installation of custom egg, wheel and from git sources HOPSWORKS-1751 Query replay for training datasets

    Sub-task

    HOPSWORKS-1580 Online Feature Store JDBC password retrieval from UI

    Source code(tar.gz)
    Source code(zip)
  • v1.4.1(Oct 9, 2020)

        Release notes - Hopsworks - Version 1.4.0
    

    Improvement

    New Feature

    Bug

    • [HOPSWORKS-1421] - Hopsworks ignores chef parameter to set default max number of user projects
    • [HOPSWORKS-1504] - Workflow builder should sanitize job names
    • [HOPSWORKS-1568] - Installing Node.js must work for airgapped deployments
    • [HOPSWORKS-1623] - Overwrite config.ini for ndb-chef
    • [HOPSWORKS-1633] - Sparkmagic links to Spark UI and driver logs are broken
    • [HOPSWORKS-1634] - MaggyFacade throws getSingleResult for findByAppId in parallel experiments
    • [HOPSWORKS-1671] - increase systemctl timeout for ndbmtd
    • [HOPSWORKS-1672] - Featurestore Descriptions containing ' cannot be handled by hive
    • [HOPSWORKS-1740] - HopsKafkaAdminClient initialisation fails if kafka is down/unreachable and then never recovers
    • [HOPSWORKS-1768] - Restrict the usage of scan operations
    • [HOPSWORKS-1782] - Set C_INCLUDE_PATH and CPLUS_INCLUDE_PATH when installing libraries in Anaconda
    • [HOPSWORKS-1793] - Increase certificate keysize
    • [HOPSWORKS-1794] - Make Hopsworks logo consistent with the branding
    • [HOPSWORKS-1795] - Close connections in AirflowDagFacade
    • [HOPSWORKS-1796] - hops-util(-py) should use consul domain to reach Hopsworks
    • [HOPSWORKS-1797] - Hopsworks should listen on an internal port with certificate signed by HopsCA
    • [HOPSWORKS-1799] - Is kubernetes enabled is not evaluated correctly in front-end after variables service refactoring
    • [HOPSWORKS-1800] - cache provenance-epipe scan operations & fixes
    • [HOPSWORKS-1805] - Discover docker registry through consul
    • [HOPSWORKS-1806] - Clean up docker images from registry
    • [HOPSWORKS-1807] - Hadoop 3 canot be installed without hopsworks
    • [HOPSWORKS-1808] - Fix Python environment upgrades from 1.3 to 1.4
    • [HOPSWORKS-1810] - fix the base image
    • [HOPSWORKS-1811] - manage docker registry with kagent.
    • [HOPSWORKS-1813] - Put back conda environment import
    • [HOPSWORKS-1819] - Query constructor should return the online query equivalent
    • [HOPSWORKS-1820] - Use Miniconda in Docker image
    • [HOPSWORKS-1823] - [hops-util-py] Add metric_key as an argument to distributed training
    • [HOPSWORKS-1824] - Fix chrome driver version
    • [HOPSWORKS-1826] - Add set -e to Docker script
    • [HOPSWORKS-1836] - Pin nbdime-jupyterlab version
    • [HOPSWORKS-1840] - Sklearn serving PermissionError
    • [HOPSWORKS-1842] - Library installation/uninstallation should increment Docker image version for the project
    • [HOPSWORKS-1850] - Consul health check for Logstash uses private IP
    • [HOPSWORKS-1855] - Set PATH and LD_LIBRARY_PATH in Dockerfile
    • [HOPSWORKS-1856] - sklearn iris classifier import joblib broken
    • [HOPSWORKS-1858] - [chef] consul does not create hops gid needed by docker
    • [HOPSWORKS-1859] - make dn and nn ports configurable
    • [HOPSWORKS-1862] - Fix cyclic dependency error
    • [HOPSWORKS-1864] - Revert HopsFS data dir location to be compatible with HopsFS upgrades
    • [HOPSWORKS-1865] - Schema problem when updating from 1.2 to 1.3
    • [HOPSWORKS-1869] - Add 1.3 migration in expat-site.xml.rb in chef
    • [HOPSWORKS-1871] - Add admin DNs to all elasticsearch nodes
    • [HOPSWORKS-1875] - Kubernetes master recipe should run after Consul default
    • [HOPSWORKS-1879] - update gpu machine - docker packages
    • [HOPSWORKS-1880] - Can not create project with a member
    • [HOPSWORKS-1885] - hopsworks ee should use hops ee
    • [HOPSWORKS-1886] - Service start fail if consul not ready.
    • [HOPSWORKS-1887] - Hive warehouse should not be set as DB policy when HopsFS-S3 is enabled.
    • [HOPSWORKS-1890] - Bump Tez minor version to account for Hadoop 3.2 dependencies
    • [HOPSWORKS-1895] - Spark History server should listen to 0.0.0.0
    • [HOPSWORKS-1900] - epipe mishandling of large xattr when logged xattr size differs from current xattr size
    • [HOPSWORKS-1901] - JWTRequired filter not working if ApiKey is set
    • [HOPSWORKS-1904] - Livy needs access to users' certificates
    • [HOPSWORKS-1905] - reset_session at the beginning of maggy_spec
    • [HOPSWORKS-1911] - TensorFlow version in variables should be 2.2
    • [HOPSWORKS-1913] - Can't write to S3 bucket from a Docker container
    • [HOPSWORKS-1914] - Docker image name regex sanitizer should be a shared attribute in Chef
    • [HOPSWORKS-1915] - resolv.conf is regenerated on reboot on CentOS
    • [HOPSWORKS-1917] - Fix the notebook names in the ruby tests after tf 2.2 upgrade of hops examples
    • [HOPSWORKS-1919] - Consule race condition
    • [HOPSWORKS-1920] - Airflow Job Sensor does not pass the API key to connection hook
    • [HOPSWORKS-1928] - Add fs when returning HopsFS training datasets to the clients
    • [HOPSWORKS-1929] - enforce unique names in training dataset splits
    • [HOPSWORKS-1931] - online fs grants not updated correctly when changing user roles
    • [HOPSWORKS-1933] - Services hang up startup for 10min if consul failed to start
    • [HOPSWORKS-1938] - Jupyter fails to start as it does not find the launcher script
    • [HOPSWORKS-1953] - update expat to use hops 3.2.0.0-SNAPSHOT
    • [HOPSWORKS-1959] - kube-hops::master should run after consul::default
    • [HOPSWORKS-1963] - Fix download_url in setup-chef
    • [HOPSWORKS-1970] - navigating to featurestore fg/td/f page does not enable all functionality
    • [HOPSWORKS-1977] - use kagent/enabled, not kagent/test
    • [HOPSWORKS-1980] - aux is a reserved keyword in windows and cannot be used for directories
    • [HOPSWORKS-1983] - datanode not starting if bind-utils not installed on centos
    • [HOPSWORKS-1985] - delete from shared dataset with full path fails
    • [HOPSWORKS-1989] - Python jobs sometimes finishes immediately and no logs are available
    • [HOPSWORKS-1993] - Upgrade Petastorm from 0.7.6 to 0.9.4
    • [HOPSWORKS-1995] - Move docker image regex sanitizer from hopsworks to conda chef
    • [HOPSWORKS-1998] - Project ID in Jupyter configuration is incorrectly templated with ","
    • [HOPSWORKS-2002] - Chef fixes for upgrading 1.3 -> 1.4
    • [HOPSWORKS-2004] - get_latest_featuregroup_version not updating cache
    • [HOPSWORKS-2012] - Fix typo in the link to Hopsworks Build status in README.md
    • [HOPSWORKS-2018] - non admin hops users cannot view spark executors logs
    • [HOPSWORKS-2020] - Available GPUs not calculated correctly
    • [HOPSWORKS-2021] - TensorBoard too slow loading many experiment runs and times out
    • [HOPSWORKS-2024] - Fix consul hostname resolution on ubuntu containers
    • [HOPSWORKS-2028] - Starting two jobs with appPath pointing to .ipynb fails if started directly after each other
    • [HOPSWORKS-2036] - TensorFlow should not be imported in hops-util-py util module
    • [HOPSWORKS-2047] - Non LTS Nvidia Driver version doesn't work on Azure VMs
    • [HOPSWORKS-2048] - Jupyter-git missing configuration for git_backend
    • [HOPSWORKS-2059] - Base directory when starting Jupyter in python mode not set correctly

    Task

    Sub-task

        Release notes - Hopsworks - Version 1.4.1
    

    Bug

    • [HOPSWORKS-2057] - Unique constraint violation when installing multiple llibraries and update conda docker gc
    • [HOPSWORKS-2065] - Restart systemd units after failure
    Source code(tar.gz)
    Source code(zip)
  • v1.3.0(Jun 3, 2020)

        Release notes - Hopsworks - Version 1.3.0
    

    Bug

    • [HOPSWORKS-1382] - Placeholders for auditing REST API requests
    • [HOPSWORKS-1471] - Increase MaxNoOfTriggers and MaxNoOfFiredTriggers to delete large no of FKs
    • [HOPSWORKS-1545] - Handle cases where Elasticsearch is down
    • [HOPSWORKS-1546] - ruby tests
    • [HOPSWORKS-1556] - Kafka controller fixes
    • [HOPSWORKS-1557] - Dataset menu hidden as browser is not scrollable
    • [HOPSWORKS-1558] - dataset - share with cluster - fails
    • [HOPSWORKS-1559] - Truncate conda commands before upgrading to 1.1.0
    • [HOPSWORKS-1560] - Fix migration of datasets (HOPSWORKS-295)
    • [HOPSWORKS-1561] - Restrict access to elasticsearch data directory to elastic user
    • [HOPSWORKS-1563] - Schema of a shared topic is shown in one line
    • [HOPSWORKS-1565] - Removing a shared dataset removes it also from the source project
    • [HOPSWORKS-1566] - provenance elastic client - index not found exception not handled
    • [HOPSWORKS-1568] - Installing Node.js must work for airgapped deployments
    • [HOPSWORKS-1569] - Can't deploy with disable password login flag set to true
    • [HOPSWORKS-1578] - epipe operation fail on missing index
    • [HOPSWORKS-1579] - Align services on the project service nav bar
    • [HOPSWORKS-1581] - [hops-hadoop-log] Touch hadoop.log with correct ownership and permissions
    • [HOPSWORKS-1583] - Incorrect path for PYSPARK_ARCHIVES_PATH in spark-env.sh
    • [HOPSWORKS-1589] - Cannot upload to a shared dataset
    • [HOPSWORKS-1590] - Kafka advertised listeners external port conflict among brokers
    • [HOPSWORKS-1596] - Incorrect sklearn logs format
    • [HOPSWORKS-1599] - Cannot start/stop services from admin UI due to missing CommunicationController property
    • [HOPSWORKS-1600] - Services Status enum missing INIT state of kagent
    • [HOPSWORKS-1601] - Append projectId to topic name in kafka tour to avoid clashes with other topics
    • [HOPSWORKS-1603] - Kagent JWTHelper should retry in case of hopsworks authentication failure
    • [HOPSWORKS-1604] - In hops python library, the s3 connector tries to read parquet files as type avro
    • [HOPSWORKS-1609] - [ePipe] Fix version conflicts in Elasticsearch
    • [HOPSWORKS-1612] - Can't view directories with name url encoded
    • [HOPSWORKS-1616] - Hopsworks ear download URL is incorrect when installing with Kubernetes
    • [HOPSWORKS-1619] - Can't zip/unzip directories with name containing url encoded chars
    • [HOPSWORKS-1631] - [tensorflow-chef] Kernel-devel not installed correctly on some Centos flavors
    • [HOPSWORKS-1636] - [hops-hadoop-chef] Configuration key in force renew-service-tokens script is wrong
    • [HOPSWORKS-1637] - Papermill creates connection file without kernel_id in the filename which breaks notebook versioning
    • [HOPSWORKS-1643] - Filebeat kagent multiline property is not configured for new ELK 7.2
    • [HOPSWORKS-1645] - duplicate class in hopswoks-common
    • [HOPSWORKS-1646] - Leftover state when updating experiments table
    • [HOPSWORKS-1650] - Search does not work when in a service other than Data Sets
    • [HOPSWORKS-1653] - Error from Provenance when sharing dataset to target project name including upper case characters
    • [HOPSWORKS-1654] - Feature store page stuck in loading view if project name contains upper case letters
    • [HOPSWORKS-1663] - ePipe crashes after rerunning featurestore tour job
    • [HOPSWORKS-1664] - Shared dataset showing up in move and copy modals without project-origin prefix
    • [HOPSWORKS-1669] - Only chief worker should return wrapper return value in distributed training
    • [HOPSWORKS-1679] - Pin version of Java used in Berksfile
    • [HOPSWORKS-1682] - Downloading a file results in renamed file with the full path prefixed in the filename
    • [HOPSWORKS-1684] - [hops-util-py] nvidia-ml-py should be included in install_requires, remove tensorflow-gpu
    • [HOPSWORKS-1685] - Livy server should start after ResourceManager
    • [HOPSWORKS-1686] - Calling create training dataset with non existing storage connector doesn't fail
    • [HOPSWORKS-1690] - nvml_monitor fails when devices does not support device calls
    • [HOPSWORKS-1696] - Increase timeout for listing conda environments when creating projects
    • [HOPSWORKS-1699] - OOM Exceptions when parsing feature store metadata in hops-util-py
    • [HOPSWORKS-1703] - Create training dataset endpoint does not return full entity of the new training dataset
    • [HOPSWORKS-1704] - fetching schema and data preview not working in featurestore shared with another project
    • [HOPSWORKS-1706] - Use system's nameserver for DNS service discovery
    • [HOPSWORKS-1707] - Activating users using User-Admin broken
    • [HOPSWORKS-1708] - When sharing a featurestore dataset training dataset should also be shared
    • [HOPSWORKS-1710] - Add permission inheritance configuration to hive-site
    • [HOPSWORKS-1712] - epipe provenance - failure on project creation failure/cleanup
    • [HOPSWORKS-1713] - Fix integration test
    • [HOPSWORKS-1714] - Set default HopsFS umask to 0027
    • [HOPSWORKS-1715] - Create feature group from the UI does not work
    • [HOPSWORKS-1717] - New project members are not added to shared datasets' groups
    • [HOPSWORKS-1719] - Logs/Spark directory has 750 permissions, only project owner can see application logs
    • [HOPSWORKS-1720] - Better handling of Null when getting a non existing variable from the API
    • [HOPSWORKS-1721] - add visibility level to variables
    • [HOPSWORKS-1722] - Test missing JWT annotation.
    • [HOPSWORKS-1723] - Deleting a training dataset from featurestore UI does not delete the parent folder of the dataset
    • [HOPSWORKS-1727] - download nccl from s3 url broken
    • [HOPSWORKS-1729] - Disable SELinux for Community Edition
    • [HOPSWORKS-1731] - Data scientists cannot install Python libs due to kibana permissions error
    • [HOPSWORKS-1734] - Prometheus retention time too high for development
    • [HOPSWORKS-1737] - Fix search for non demo projects
    • [HOPSWORKS-1740] - HopsKafkaAdminClient initialisation fails if kafka is down/unreachable and then never recovers
    • [HOPSWORKS-1744] - Multiple Consul masters are in bootstrap mode
    • [HOPSWORKS-1749] - update featurestore search tests to new tag endpoints
    • [HOPSWORKS-1757] - Update Maggy ablation example notebook with fix for TF pickling issue
    • [HOPSWORKS-1760] - using RELEASE for mvn artifact breaks build
    • [HOPSWORKS-1764] - Ensure base url is reset after setting it to empty.
    • [HOPSWORKS-1767] - Race condition in certificate removal
    • [HOPSWORKS-1770] - epipe - reindex featurestore (featuregroups/training datasets)
    • [HOPSWORKS-1774] - [hops-util-py] Experiment directory should be cleaned up if Spark job is rerun
    • [HOPSWORKS-1775] - Bump experiments and models test job memory configuration
    • [HOPSWORKS-1777] - Hopsworks airflow operator should check also for the am status
    • [HOPSWORKS-1793] - Increase certificate keysize
    • [HOPSWORKS-1799] - Is kubernetes enabled is not evaluated correctly in front-end after variables service refactoring
    • [HOPSWORKS-1818] - Kakfa broker env variable not available in jobs

    Sub-task

    New Feature

    Improvement

    • [HOPSWORKS-1409] - Make storage policy configurable
    • [HOPSWORKS-1503] - REST API - Admin - Services
    • [HOPSWORKS-1532] - No need to mount an S3 bucket for the certificates in Databricks
    • [HOPSWORKS-1552] - Add S3/Redshift feature store get started notebooks
    • [HOPSWORKS-1562] - Improve databricks docs for the feature store
    • [HOPSWORKS-1567] - Add a confirmation when sharing a kafka topic with another project
    • [HOPSWORKS-1584] - add inferSchema when importing csv feature groups from S3
    • [HOPSWORKS-1588] - Fixing default values for parameters in Hops & NDB cookbooks
    • [HOPSWORKS-1617] - Improve hopsworks-installer.sh for single-host cloud installations (non localhost)
    • [HOPSWORKS-1622] - Add proxy attribute for pip.conf
    • [HOPSWORKS-1629] - Add hopsworks read the docs link on the sidebar
    • [HOPSWORKS-1647] - hops-util-py should allow disabling of Hostname verification in the featurestore.connect method
    • [HOPSWORKS-1662] - Parameterize JWT token location
    • [HOPSWORKS-1668] - Add API key annotations for StorageConnectorService to make requests from AWS
    • [HOPSWORKS-1670] - Replace withColumn with select when attaching prov metadata to feature groups
    • [HOPSWORKS-1676] - Make maggy version configurable in chef
    • [HOPSWORKS-1678] - Extend training dataset endpoint to save train-test split information as metadata
    • [HOPSWORKS-1689] - Allow maggy to be installed by git URL
    • [HOPSWORKS-1698] - Fix hops-hadoop-chef to fix logging Issue with ClusterJ
    • [HOPSWORKS-1718] - Start service health checks with passing state
    • [HOPSWORKS-1726] - Remove need to define consul::default in karamel cluster defns
    • [HOPSWORKS-1728] - Register Logstash with Consul
    • [HOPSWORKS-1733] - Separate HopsFS mysql user from Hopsworks MySQL user
    • [HOPSWORKS-1743] - Report Hopsworks unit tests in Jenkins
    • [HOPSWORKS-1746] - Full screen data preview for feature store
    • [HOPSWORKS-1758] - Make email case insensitive
    • [HOPSWORKS-1763] - Improve error message for Sparkmagic when driver is not reachable
    • [HOPSWORKS-1766] - Increased default no of threads for NDB
    • [HOPSWORKS-1781] - Reuse Service discovery client
    • [HOPSWORKS-1790] - Kube-hops master untaint fails arbitrarily

    Task

    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Feb 10, 2020)

        Release notes - Hopsworks - Version 1.2.0
    

    Bug

    • [HOPSWORKS-1283] - Zip does not work when there is a space in the file path
    • [HOPSWORKS-1379] - Timers should be rescheduled when failed and handle exceptions
    • [HOPSWORKS-1382] - Placeholders for auditing REST API requests
    • [HOPSWORKS-1507] - [ndb-chef] List only TCP and UDP sockets when starting MGM server
    • [HOPSWORKS-1509] - Allow all OU in nodes DN in elasticsearch configuration
    • [HOPSWORKS-1511] - pci-tools package not available and torchvision broken for 0.4.1
    • [HOPSWORKS-1515] - airflow webserver , scheduler and sqoop should not be enabled when service.enabled is false
    • [HOPSWORKS-1534] - Kibana error when recreating the project enviroment
    • [HOPSWORKS-1541] - [ePipe] Skip non searchable fields for schemabased exmetadata
    • [HOPSWORKS-1543] - [ePipe] Retry in case of conflict when adding datasets and projects
    • [HOPSWORKS-1545] - Handle cases where Elasticsearch is down
    • [HOPSWORKS-1550] - Do not upgrade pip binary in tensorflow default recipe
    • [HOPSWORKS-1553] - provenance cleaner timer
    • [HOPSWORKS-1556] - Kafka controller fixes
    • [HOPSWORKS-1559] - Truncate conda commands before upgrading to 1.1.0
    • [HOPSWORKS-1560] - Fix migration of datasets (HOPSWORKS-295)
    • [HOPSWORKS-1561] - Restrict access to elasticsearch data directory to elastic user
    • [HOPSWORKS-1563] - Schema of a shared topic is shown in one line
    • [HOPSWORKS-1564] - Job args expat tries to set null instead of empty string
    • [HOPSWORKS-1565] - Removing a shared dataset removes it also from the source project
    • [HOPSWORKS-1578] - epipe operation fail on missing index
    • [HOPSWORKS-1589] - Cannot upload to a shared dataset
    • [HOPSWORKS-1590] - Kafka advertised listeners external port conflict among brokers
    • [HOPSWORKS-1594] - HOPSWORKS-1554 breaks conda search in Hopsworks
    • [HOPSWORKS-1599] - Cannot start/stop services from admin UI due to missing CommunicationController property
    • [HOPSWORKS-1600] - Services Status enum missing INIT state of kagent
    • [HOPSWORKS-1601] - Append projectId to topic name in kafka tour to avoid clashes with other topics
    • [HOPSWORKS-1603] - Kagent JWTHelper should retry in case of hopsworks authentication failure
    • [HOPSWORKS-1610] - Prometheus alerting-rules files owned by root

    Improvement

    • [HOPSWORKS-836] - Send message in Hopsworks to a user that has been added to a project.
    • [HOPSWORKS-1257] - sudoers script should be owned by root and 700
    • [HOPSWORKS-1306] - [ePipe] Add support for HA Elasticsearch
    • [HOPSWORKS-1308] - [ePipe] Rewrite and fix the metrics code
    • [HOPSWORKS-1481] - Add endpoints to retrieve feature groups and training datasets by name
    • [HOPSWORKS-1483] - Refactor Admin API - Hosts
    • [HOPSWORKS-1494] - Feature store UI improvements
    • [HOPSWORKS-1527] - Check TensorBoard readiness using http in Experiments service and for running applications
    • [HOPSWORKS-1532] - No need to mount an S3 bucket for the certificates in Databricks
    • [HOPSWORKS-1538] - Automatically setup hive and hopsfs on databricks cluster
    • [HOPSWORKS-1544] - Add endpoint to retrieve featurestore by name
    • [HOPSWORKS-1554] - Add support for proxy configuration in condarc
    • [HOPSWORKS-1562] - Improve databricks docs for the feature store

    Task

    • [HOPSWORKS-1330] - Remove deprecated LocalFsService (Charon) codebase
    • [HOPSWORKS-1354] - Import(upload) template saves files in /Projects/Uploads/<Sample.json>
    • [HOPSWORKS-1407] - Change error message when failing to export project-user certificates
    • [HOPSWORKS-1522] - Update support link in Hopsworks and docs
    • [HOPSWORKS-1525] - [hops-hadoop-chef] Add option to allow Yarn auto discover HW capabilities
    • [HOPSWORKS-1528] - Remove unnecessary code to set kibana default index
    • [HOPSWORKS-1536] - Right-click rename does not work in Datasets browser
    • [HOPSWORKS-1540] - Upgrade PyTorch to 1.4
    • [HOPSWORKS-1549] - [kagent-chef] Update Hops CA bundle after host certificate rotation
    • [HOPSWORKS-1551] - Dynamically create Airflow DB connection only if Airflow is installed
    • [HOPSWORKS-1575] - Replace hops.readthedocs with hopsworks.readthedocs
    • [HOPSWORKS-1577] - [ndb-chef] Restrict MySQL connections to localhost for localhost installations
    • [HOPSWORKS-1582] - Add a new db user for Kafka
    • [HOPSWORKS-1611] - Tensorflow::default fails to install avro-python3 due to upstream changes
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 10, 2020)

        Release notes - Hopsworks - Version 1.1.0
    

    Bug

    • [HOPSWORKS-1214] - Airflow HopsworksLaunchOperator does not handle job state App Master Failed
    • [HOPSWORKS-1245] - [ePipe] Fixes for xattrs handling to be work with the changes introduced in HOPS-1434 and HOPS-1451
    • [HOPSWORKS-1246] - [ePipe] Check for the existence of elastic fields before delete
    • [HOPSWORKS-1296] - [ePipe] log replay fails on xattr op
    • [HOPSWORKS-1305] - Fix broken Anaconda export yml
    • [HOPSWORKS-1318] - Featurestore should throw an error if you try to create the same featuregroup twice
    • [HOPSWORKS-1323] - [ndb-chef] Fix bug in NDB cluster-init.sh
    • [HOPSWORKS-1325] - Online Feature store tests fail when project name contains uppercase letters
    • [HOPSWORKS-1342] - Add parameter for s3 encryption type in core-site.xml
    • [HOPSWORKS-1353] - Flink timer should handle case when services are not installed
    • [HOPSWORKS-1385] - HA elastic does not have a synchronization point between starting nodes and creating indices
    • [HOPSWORKS-1398] - User activation date is irrational
    • [HOPSWORKS-1408] - AirflowManager kills Hopsworks deployments if Airflow directory does not exist
    • [HOPSWORKS-1412] - Fix create featuregroup with partition keys
    • [HOPSWORKS-1413] - AWS S3 does not accept empty Authorization header when downloading from a public bucket
    • [HOPSWORKS-1425] - Update stack to handle hopsfs paths starting by hopsfs instead of hdfs
    • [HOPSWORKS-1426] - Avoid using 100% containers for yarn containers
    • [HOPSWORKS-1428] - hops_airflow fails to install pymssql 3.0.3, pin it to 2.1.4
    • [HOPSWORKS-1431] - Better handling of Jupyter/Git failures
    • [HOPSWORKS-1432] - Elasticsearch node should always use its private ip
    • [HOPSWORKS-1434] - Scale jvm -Xmx for Spark and Jupyter based on allocated memory
    • [HOPSWORKS-1442] - Update statistics job shows app file as undefined
    • [HOPSWORKS-1443] - [featurestore] Spark returning Infinity values for mean statistics of string type features
    • [HOPSWORKS-1451] - Storage connector ignored when creating a training dataset from the UI
    • [HOPSWORKS-1456] - [hops-hadoop-chef] Hopsworks security related properties are not templated correctly
    • [HOPSWORKS-1457] - Correct systemd scripts
    • [HOPSWORKS-1458] - spark should not bring any hops jar with it
    • [HOPSWORKS-1461] - The JWT signing key should be decoded first before using it to sign jwt
    • [HOPSWORKS-1463] - Hops ndb version not taken in account when downloading dal
    • [HOPSWORKS-1464] - fix chef for HOPS-1542
    • [HOPSWORKS-1484] - Shared feature stores do not appear in the UI
    • [HOPSWORKS-1485] - Sparkmagic upgrades not working
    • [HOPSWORKS-1488] - Deep Learning tour project doesn't create Kafka inferenceschema
    • [HOPSWORKS-1489] - Increase default Heap size and max shards per node for Elasticsearch
    • [HOPSWORKS-1490] - Handle Kibana failures in Hopsworks Kibana client
    • [HOPSWORKS-1496] - Elastic::default does not template correctly hosts in elasticsearch.yml
    • [HOPSWORKS-1497] - hops-util-py incorrectly sets Kafka config
    • [HOPSWORKS-1498] - hops-util-py can't find attributes in KafkaTopicDTO
    • [HOPSWORKS-1507] - [ndb-chef] List only TCP and UDP sockets when starting MGM server
    • [HOPSWORKS-1508] - [kagent-chef] Update kibana x.509 certificate when host certificate is rotated
    • [HOPSWORKS-1510] - Remove the code to delete index patterns since we delete kibana project index
    • [HOPSWORKS-1516] - [ePipe] handle the case if XAttrAddAll is received for a deleted inode
    • [HOPSWORKS-1518] - [livy-chef] Create root Hops directory
    • [HOPSWORKS-1519] - The subject of ELK JWT token should be always lowercase
    • [HOPSWORKS-1521] - Dataset browser refresh button not working
    • [HOPSWORKS-1523] - Kibana opendistro_security plugin is not idempotent
    • [HOPSWORKS-1524] - [hops-hadoop-chef] Hops services startup should be dependent on Glassfish
    • [HOPSWORKS-1526] - Can't update job when doing insert into featuregroup
    • [HOPSWORKS-1564] - Job args expat tries to set null instead of empty string

    Sub-task

    New Feature

    Improvement

    • [HOPSWORKS-1248] - [ePipe] Adapt the new dataset tracking introduced in HOPS-1483
    • [HOPSWORKS-1254] - [ePipe] handle elastic search failures
    • [HOPSWORKS-1310] - Refactor Kafka API - ACLs
    • [HOPSWORKS-1324] - [ePipe] Adapt to the new dataset table schema
    • [HOPSWORKS-1332] - [featurestore] improve queryplanner docs/examples
    • [HOPSWORKS-1356] - Enable paraller executions for jobs
    • [HOPSWORKS-1380] - [featurestore] Configure Feature Group statistics during creation
    • [HOPSWORKS-1384] - Refactor Kafka API - Schemas
    • [HOPSWORKS-1391] - Extend Pandas and Numpy Helper Classes
    • [HOPSWORKS-1395] - Hopsworks CA REST endpoint to download Hops CRL instead of Glassfish serving the file
    • [HOPSWORKS-1404] - Prometheus should handle the case in which recipes are missing.
    • [HOPSWORKS-1415] - Admin should be able to reset users' passwords
    • [HOPSWORKS-1418] - Bump PyTorch, Numpy, Torchvision versions
    • [HOPSWORKS-1423] - [cloud-sdk] Import feature group job not failing in client when storage connector does not exist
    • [HOPSWORKS-1424] - [featurestore] Handle upper/lower case for feature group names
    • [HOPSWORKS-1435] - [featurestore] Unify feature store entity max length validation in backend
    • [HOPSWORKS-1436] - Accept DB name when setting up Hive connection in hops-util-py
    • [HOPSWORKS-1440] - Recipes should be able to update Hive/Spark jars without updating the distribution
    • [HOPSWORKS-1465] - Make feature store documentation easier to find
    • [HOPSWORKS-1469] - REST API v2 - Admin - Users
    • [HOPSWORKS-1517] - Double // in download url for Kubenetes artifacts

    Task

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Oct 28, 2019)

        Release notes - Hopsworks - Version 1.0.0
    

    Bug

    • [HOPSWORKS-432] - Clean Kafka ACLs when unsharing a topic or deleting a project.
    • [HOPSWORKS-1095] - Localhost (127.0.0.1) cluster definition support
    • [HOPSWORKS-1114] - Rework password recovery
    • [HOPSWORKS-1146] - Kagent stops reporting if env creation fails with non default repo
    • [HOPSWORKS-1152] - Dataset browser not displaying contents of shared datasets with naming conflicts
    • [HOPSWORKS-1172] - Upload progress not visible in the beginning of uploading large files
    • [HOPSWORKS-1175] - yarnapp/user attribute not propagated correctly to yarn-site.xml
    • [HOPSWORKS-1176] - Security risk for default admin password pops up even if the password has been changed
    • [HOPSWORKS-1178] - [airflow-chef] Add default timezone for Airflow
    • [HOPSWORKS-1180] - Chef upgrade mixes kafka brokers ids
    • [HOPSWORKS-1185] - Jupyter not working when Hopsworks runs behind a LB/reverse proxy w/ TLS termination
    • [HOPSWORKS-1190] - [featurestore] missing argument when writing petastorm datasets
    • [HOPSWORKS-1195] - Bad handling of Job resource missing
    • [HOPSWORKS-1197] - Fix broken admin and agent tests
    • [HOPSWORKS-1198] - Cleanup hadoop sbin scripts
    • [HOPSWORKS-1199] - ErrorCode range mismatch and TestRestCodes not testing all ErrorCodes
    • [HOPSWORKS-1203] - [ePipe] Fixes for JSON handling in the XAttr table
    • [HOPSWORKS-1207] - Hopsworks broken on Ubuntu 18.04
    • [HOPSWORKS-1209] - filebeat_kagent does not start with kagent start-all services script
    • [HOPSWORKS-1210] - Jupyter now requires the python kernel.json to be under python{2, 3} dir.
    • [HOPSWORKS-1215] - Use OSProcessExecutor for Python library search
    • [HOPSWORKS-1216] - tensorflow-chef install recipe CentOS hiccups
    • [HOPSWORKS-1218] - Python service not working when searching or installing libraries containing 'http'
    • [HOPSWORKS-1225] - add attribute for url to download spark in spark-chef
    • [HOPSWORKS-1228] - Remove livy session name from sparkmagic conf file
    • [HOPSWORKS-1233] - [ePipe] hivecleaner fail during project removal
    • [HOPSWORKS-1234] - Remove sudo command from sudoers scripts
    • [HOPSWORKS-1236] - Project cleanup does not work after a failed project creation
    • [HOPSWORKS-1246] - [ePipe] Check for the existence of elastic fields before delete
    • [HOPSWORKS-1251] - images not rendering in Jupyter notebooks
    • [HOPSWORKS-1252] - Airflow HopsworksLaunchOperator waits for wrong Job statuses
    • [HOPSWORKS-1253] - [ePipe] Rename operation should read the inodes info from the DB
    • [HOPSWORKS-1263] - Project cleanup fails if X.509 to be revoked does not exist
    • [HOPSWORKS-1268] - Sanitize python library name/version/reposity/url
    • [HOPSWORKS-1271] - Bump HopsHive to use ORC 1.5.5
    • [HOPSWORKS-1276] - Anaconda import yml file not working, removal of explicit enabling of python kernel
    • [HOPSWORKS-1278] - [ePipe] The tailer should guarantee the order of epochs during recovery and while waiting for events
    • [HOPSWORKS-1279] - [ePipe] Check for the existence of the xattr parent field before delete
    • [HOPSWORKS-1280] - [hops-util-py] add_module not working in python kernel
    • [HOPSWORKS-1282] - Exclude netty from Hudi hdfs dependency
    • [HOPSWORKS-1285] - Livy is missing Yarn GENERATING_SECURITY_MATERIAL state
    • [HOPSWORKS-1290] - [hops-util-py] numpy_helper and pandas_helper documentation not rendering
    • [HOPSWORKS-1297] - Model Serving view not sorted
    • [HOPSWORKS-1298] - Cannot remove job associated with featuregroups
    • [HOPSWORKS-1300] - [kagent-chef] Change deprecated node.set on install recipe
    • [HOPSWORKS-1315] - Hopsworks wrongly sets umask for HDFS client to 000
    • [HOPSWORKS-1326] - setup-chef is broken
    • [HOPSWORKS-1328] - [airflow-chef] Package libjpeg-devel is no longer available
    • [HOPSWORKS-1329] - [hopsworks-cloud-sdk] Fix connecting from Databricks
    • [HOPSWORKS-1336] - [kagent] Host private key is not converted to PKCS#1 if it's an upgrade
    • [HOPSWORKS-1343] - Airflow Hopsworks connection hook ignores configured connections
    • [HOPSWORKS-1350] - [featurestore] default training dataset connector not found
    • [HOPSWORKS-1355] - Conda solver is problematic
    • [HOPSWORKS-1357] - flyway_schema_history fix for versions < 0.6 doesn't work after 1.0.0
    • [HOPSWORKS-1358] - Handle glassfish upgrade from 4.1.2.174 to 4.1.2.181
    • [HOPSWORKS-1359] - MySQL server reinitialized during update to 1.0.0
    • [HOPSWORKS-1360] - Data validation JS controller does not properly include mainCtrl
    • [HOPSWORKS-1365] - When upgrading ignore anaconda base env's lib dir of previous installations
    • [HOPSWORKS-1368] - NullPointerException when deleting a project without OnlineFeaturestore
    • [HOPSWORKS-1370] - Online featurestore need mysql driver in the spark classpath

    Sub-task

    New Feature

    Improvement

    • [HOPSWORKS-453] - Spark jobs should use local spark jars instead of hdfs archive
    • [HOPSWORKS-781] - [ePipe] Integrate hive cleaner into ePipe
    • [HOPSWORKS-782] - [ePipe] Synchronize recovery tailer and table tailers to avoid inconsistencies during recovery
    • [HOPSWORKS-1054] - Reimplement the schemaless metadata API to use the extended attributes added in HOPS-1312
    • [HOPSWORKS-1063] - Cleanup NDB configuration in ndb-chef
    • [HOPSWORKS-1124] - optimize scp-deploy-frontend.sh
    • [HOPSWORKS-1144] - Update conda distribution binary
    • [HOPSWORKS-1162] - Make hops home readable for everybody
    • [HOPSWORKS-1163] - Set the Jupyter token explicitly
    • [HOPSWORKS-1164] - Refactor Hopswork to prepare support for non-local Jupyter
    • [HOPSWORKS-1179] - [hops-examples] Add Maggy MNIST example notebook to DL tour
    • [HOPSWORKS-1189] - Remove kagent/network/interface attribute
    • [HOPSWORKS-1196] - Feature Store V2
    • [HOPSWORKS-1200] - upgrade apexcharts
    • [HOPSWORKS-1204] - [ePipe] Use configuration files instead of command line parameters
    • [HOPSWORKS-1208] - Replace Jupyter Notebook with JupyterLab
    • [HOPSWORKS-1217] - Configure Livy to not upload internal libraries to stagingDir
    • [HOPSWORKS-1219] - [ePipe] Update CMakeLists to use C++14, Boost 1.70, and CMake 3.5.0
    • [HOPSWORKS-1226] - [ePipe] Use Boost:Beast http client library instead of Libcurl
    • [HOPSWORKS-1227] - forward kafka authorizer logs to a separate file
    • [HOPSWORKS-1230] - Support changing the schema of an existing Kafka topic
    • [HOPSWORKS-1237] - Improve the Installation / Configuration Documentation
    • [HOPSWORKS-1242] - [Featurestore] update hops-petastorm to track petastorm 0.7.6
    • [HOPSWORKS-1247] - Download restricted artifacts when installing Hopsworks EE
    • [HOPSWORKS-1256] - Add flag to disable creation of users, groups and user-group mapping
    • [HOPSWORKS-1269] - [featurestore] make S3 connector compatible with IAM role
    • [HOPSWORKS-1284] - Add support for installing python libraries by name and/or version
    • [HOPSWORKS-1299] - Forbid S3 ft connector access/secret key if using instance role
    • [HOPSWORKS-1307] - [Karamel] Fixes to release Karamel 0.6
    • [HOPSWORKS-1312] - Airflow operator to fetch Feature store validation result
    • [HOPSWORKS-1321] - [featurestore] Utility function to list feature names in feature group
    • [HOPSWORKS-1322] - [hopsworks-cloud-sdk] Configure credential per IAM role not per project and IAM role
    • [HOPSWORKS-1337] - [featurestore] Utility function to list names of features in training dataset
    • [HOPSWORKS-1366] - Upgrade default maggy version

    Task

    Source code(tar.gz)
    Source code(zip)
  • v0.10.0(Jul 2, 2019)

        Release notes - Hopsworks - Version 0.10.0
    

    Bug

    • [HOPSWORKS-197] - Turning on and off tours is a terrible user experience
    • [HOPSWORKS-715] - Hopsworks singletons are not single
    • [HOPSWORKS-770] - Disable http port and TLS 1.0 on glassfish
    • [HOPSWORKS-849] - kagent - conda can get stuck
    • [HOPSWORKS-903] - Jupyter cleanup and timer bugfixes
    • [HOPSWORKS-904] - Users should be able to specify version of Python library to install when behind PyPi proxy
    • [HOPSWORKS-905] - remote_material_references is not cleaned up during project deletion
    • [HOPSWORKS-913] - Feature store quota missing from admin UI in hopsworks
    • [HOPSWORKS-917] - Non HA admin UI when HA namenodes
    • [HOPSWORKS-920] - Job monitor thread gets stuck if job is killed while running
    • [HOPSWORKS-927] - TensorBoard could not bind to an unsupported address family
    • [HOPSWORKS-928] - [Featurestore] Bug: nested spark schemas are not automatically translatable to Hive
    • [HOPSWORKS-929] - [Featurestore] Don't allow hyphen in featuregroup names because Hive do not allow it
    • [HOPSWORKS-930] - [Featurestore] bug, in API retrieval code in feature-registry featurestore is undefined
    • [HOPSWORKS-932] - Jobs UI date selector should persist user-selected values
    • [HOPSWORKS-937] - Adjust Cgroup configuration to account for longer Cgroup deletion and make CPU utilization configurable
    • [HOPSWORKS-938] - Adding pyFiles to Jupyter does not add it to python path
    • [HOPSWORKS-941] - Fix dataset access request and related tests
    • [HOPSWORKS-945] - Zipping and unzipping is broken with Hops TLS
    • [HOPSWORKS-948] - [hops-hadoop-chef] Copy hadoop logs utility is not removed from crontab
    • [HOPSWORKS-949] - [airflow-chef] Install recipe does not create root hops installation directory
    • [HOPSWORKS-951] - sqoop and airflow bugfixes
    • [HOPSWORKS-953] - [kagent-chef] kagent falsely assumes service is dead when it restarts
    • [HOPSWORKS-955] - [airflow-chef] Use assigned roles in JWT to determine if a user is admin
    • [HOPSWORKS-958] - AirflowJWTManager randomly does not renew tokens
    • [HOPSWORKS-962] - copy_to_hdfs is broken in hops-util-py
    • [HOPSWORKS-965] - Certificate enddate should always be UTC
    • [HOPSWORKS-968] - Dataset browser pagination and select all not working
    • [HOPSWORKS-970] - upgrade chef ulimit cookbook
    • [HOPSWORKS-972] - Missing sasl/sasl.h breaks PyHopsHive installation
    • [HOPSWORKS-973] - Dela install should create the hops group before adding members to it
    • [HOPSWORKS-974] - Blank padded day in rspec breaks tests
    • [HOPSWORKS-975] - [airflow-chef] Change restart policy of airflow scheduler
    • [HOPSWORKS-976] - Airflow file manager does not refresh secret directory when navigate to another project
    • [HOPSWORKS-977] - Pin tornado version to 5.1.1
    • [HOPSWORKS-978] - dela::install should create /srv/hops/ before creating sub directories
    • [HOPSWORKS-979] - Cannot change default 'admin' password of [email protected]
    • [HOPSWORKS-980] - HOPSWORKS-950 breaks hops-util-py in python 2.7 environments
    • [HOPSWORKS-983] - [ePipe] Wrong type for long values in query condition handling
    • [HOPSWORKS-987] - User admin to slow and user search returning duplicated results
    • [HOPSWORKS-988] - Launching Jupyter server gradually gets slower and times out
    • [HOPSWORKS-990] - uploading big files from web ui can cause 'invalidated token'
    • [HOPSWORKS-992] - [kagent-chef] Invalid command in kagent's service status script
    • [HOPSWORKS-993] - [Feturestore] tf record schema infer for float arrays is not working correctly
    • [HOPSWORKS-994] - TensorBoard in experiments service fails to start if tensorflow-gpu is in the project environment
    • [HOPSWORKS-996] - Jobs monitor threads may saturate the system
    • [HOPSWORKS-1000] - Parameter servers in ParameterServerStrategy do not return Spark task when all workers are finished
    • [HOPSWORKS-1003] - Jupyter convert notebook should inject SparkSession and handle spaces in notebook name
    • [HOPSWORKS-1004] - YarnJobMonitor does not have a full list of Yarn running states
    • [HOPSWORKS-1007] - Install version specific kernel packages, not generic ones
    • [HOPSWORKS-1009] - Non-websocket requests in Jupyter notebook are blocked due to cross-origin policy
    • [HOPSWORKS-1011] - hops-metadata-dal-impl-ndb schema update renames hdfs_inode_attributes to hdfs_directory_with_quota_feature
    • [HOPSWORKS-1012] - Metadata designer does not work
    • [HOPSWORKS-1013] - Descrease websocket_ping_interval to avoid proxied websocket connections being dropped
    • [HOPSWORKS-1014] - ui bugs
    • [HOPSWORKS-1017] - Date in field "feature store created" is NaN
    • [HOPSWORKS-1022] - NOT_START is not a final state for Log Aggregation
    • [HOPSWORKS-1030] - [hops-util-py] copy_to_local bugfixes and removal of localize
    • [HOPSWORKS-1031] - Show enable conda message for PySpark Jobs if conda is disabled
    • [HOPSWORKS-1036] - JobScheduler gets expunged when trying to start an execution for a job while the job is still running
    • [HOPSWORKS-1040] - assign unique ids to growl
    • [HOPSWORKS-1041] - Uploading template on Metadata designer does not work
    • [HOPSWORKS-1042] - Python kernels with capital letters are filtered out by Jupyter Notebook
    • [HOPSWORKS-1043] - addUserToGroup throws exception if user is added twice
    • [HOPSWORKS-1047] - Cannot see datasets if an HiveDb has been shared with the project
    • [HOPSWORKS-1048] - Kagent elastic logs fails to be created for projects with capital letters
    • [HOPSWORKS-1049] - io.hops.util.exceptions.FeaturestoreNotFound logged/thrown during the Kafka tour
    • [HOPSWORKS-1051] - [airflow-chef] Colliding JWT libraries
    • [HOPSWORKS-1056] - [ePipe] wrong key type for inode id
    • [HOPSWORKS-1058] - [ePipe] wrong event ordering during recovery
    • [HOPSWORKS-1060] - tensorflow and torch lib conflicts
    • [HOPSWORKS-1061] - SegFault when importing tensorflow with petastorm in the wrong order
    • [HOPSWORKS-1066] - [hops-hadoop-chef] Recipes should not try to template JWT tokens when Hopsworks is not available
    • [HOPSWORKS-1067] - Hopsworks does not build with cluster profile enabled
    • [HOPSWORKS-1069] - Fix chef attr for hopsworks/https/port
    • [HOPSWORKS-1071] - spark.executor.instances should default to spark.dynamicAllocation.minExecutors for Spark Dynamic configurations
    • [HOPSWORKS-1075] - [kagent-chef] Fail fast if FQDN is longer than 63 characters
    • [HOPSWORKS-1077] - Review overridable Spark configuration properties for SparkJobConfiguration
    • [HOPSWORKS-1086] - Fix nvidia install provider
    • [HOPSWORKS-1087] - Fix nested transaction exception when starting jupyter
    • [HOPSWORKS-1088] - FeaturegroupController doesn't handle certificates correctly
    • [HOPSWORKS-1090] - TensorBoardController nested transaction exception
    • [HOPSWORKS-1093] - Using hops-util inside a transformation in spark job throws NoClassDefFoundError
    • [HOPSWORKS-1101] - Fix JWT tests after changing default user account
    • [HOPSWORKS-1106] - Conda search versions not ordered
    • [HOPSWORKS-1112] - Fix ssl-server.xml for client machines.
    • [HOPSWORKS-1113] - Remove PIP upgrade in tensorflow::install
    • [HOPSWORKS-1116] - hopsworks_port conf has been removed, use hopsworks_endpoint instead
    • [HOPSWORKS-1121] - Use pip binary in base environment for search
    • [HOPSWORKS-1122] - Invalid JobConfiguration when EXPERIMENT is set as default
    • [HOPSWORKS-1125] - Pip libraries with no release date
    • [HOPSWORKS-1126] - [hops-util-py] avro parsing only works in python 2.7
    • [HOPSWORKS-1127] - [hops-util-py] rest error messages are not parsed correctly
    • [HOPSWORKS-1136] - Off by one error when templating Hopsworks schema upgrade files
    • [HOPSWORKS-1138] - Transaction batchsizes should not be hardcoded in hdfs-site.xml
    • [HOPSWORKS-1147] - Search results for hive datasets are not clickable
    • [HOPSWORKS-1148] - Job duration increasing forever
    • [HOPSWORKS-1149] - CA module should not have its own persistence.xml file
    • [HOPSWORKS-1150] - Users should not be allowed to start jobs with no yarn quota.
    • [HOPSWORKS-1165] - JAXB Date serializer for JWTResource should include TZ info
    • [HOPSWORKS-1168] - GPU warning being shown in Spark Static and Spark Dynamic views
    • [HOPSWORKS-1169] - [hops-util-py] CollectiveAllReduceStrategy and MirroredStrategy should have a 'chief' executor in cluster spec
    • [HOPSWORKS-1173] - Expose tez and slider user in hive-chef's metadata.rb

    Sub-task

    New Feature

    Improvement

    • [HOPSWORKS-566] - Move Hive scratchdir cleaner from hiveCleaner to Hopsworks
    • [HOPSWORKS-716] - Unify Spark Configuration for Jobs service and Jupyter
    • [HOPSWORKS-759] - selenium integration test
    • [HOPSWORKS-875] - Add PyTorch, Torchvision and Matplotlib to python base environment
    • [HOPSWORKS-897] - Hopsworks-ca should set expiration date not validity in days
    • [HOPSWORKS-911] - [Featurestore] Cache metadata on client
    • [HOPSWORKS-912] - [Featurestore] Support image and avro data types for training datasets
    • [HOPSWORKS-916] - kagent reporting error message for failed Conda commands
    • [HOPSWORKS-926] - Airflow should install dependencies based on the platform_family not platform
    • [HOPSWORKS-942] - [Featurestore] Enable to specify decimal precision
    • [HOPSWORKS-950] - Make Pandas dataframes and numpy arrays easier to read/write from HopsFS
    • [HOPSWORKS-956] - Hops Python library: hops.hdfs.copy_to_local() returns immediately if files are already local and unchanged.
    • [HOPSWORKS-963] - Make tls module in hops-util-py usable from a python notebook
    • [HOPSWORKS-967] - hops artifacts access
    • [HOPSWORKS-991] - Add new window button to experiments
    • [HOPSWORKS-997] - Set envs_dirs and pkgs_dirs in .condarc to handle anaconda version upgrades
    • [HOPSWORKS-999] - AirflowJWTManager recovery
    • [HOPSWORKS-1001] - Experiments service refactor certificates cleanup
    • [HOPSWORKS-1002] - MirroredStrategy support for multi-worker
    • [HOPSWORKS-1006] - Membrane proxy should not rename threads
    • [HOPSWORKS-1008] - Add activated date to user administration
    • [HOPSWORKS-1024] - [featurestore] Optimize metadata for clients
    • [HOPSWORKS-1025] - add Hive Partitioning support for feature groups
    • [HOPSWORKS-1029] - [Featurestore] Revise project roles permissions to feature store
    • [HOPSWORKS-1033] - Redesign service JWTs
    • [HOPSWORKS-1039] - [Featurestore] Make featurestore/training datasets folder more visible
    • [HOPSWORKS-1064] - After HOPS-1215, small files are a storage type
    • [HOPSWORKS-1070] - Refactor HdfsLeDescriptorsFacade to return a random NN for bootstrapping the client
    • [HOPSWORKS-1074] - Bump up airborne dependency
    • [HOPSWORKS-1083] - Python3.6 environment should be created by default
    • [HOPSWORKS-1091] - Add systemd dependencies to unit files to enable clean host restarts
    • [HOPSWORKS-1097] - [Featurestore] Parameterize dataset-dir in tour job
    • [HOPSWORKS-1099] - Get leader NN http address automatically from hdfs_le_descriptors
    • [HOPSWORKS-1100] - [Featurestore] update hops-petastorm to track petastorm 0.7.4
    • [HOPSWORKS-1104] - [featurestore] visualization of feature stats in %%local
    • [HOPSWORKS-1115] - Remove redundant code from Hadoop clients factory
    • [HOPSWORKS-1129] - Use stereotypes instead of alternatives for serving integration
    • [HOPSWORKS-1131] - [featurestore] verify that sparkSQL session has hive enabled
    • [HOPSWORKS-1134] - Add extra check when initializing Hopsworks db
    • [HOPSWORKS-1153] - [sparkmagic] Progress bar for multiple experiments from same app_id
    • [HOPSWORKS-1157] - Increase flyway migrate timeout for hops migrations

    Task

    Source code(tar.gz)
    Source code(zip)
  • v0.9.1(Apr 4, 2019)

        Release notes - Hopsworks - Version 0.9.1
    

    Bug

    • [HOPSWORKS-197] - Turning on and off tours is a terrible user experience
    • [HOPSWORKS-903] - Jupyter cleanup and timer bugfixes
    • [HOPSWORKS-913] - Feature store quota missing from admin UI in hopsworks
    • [HOPSWORKS-920] - Job monitor thread gets stuck if job is killed while running
    • [HOPSWORKS-927] - TensorBoard could not bind to an unsupported address family
    • [HOPSWORKS-928] - [Featurestore] Bug: nested spark schemas are not automatically translatable to Hive
    • [HOPSWORKS-929] - [Featurestore] Don't allow hyphen in featuregroup names because Hive do not allow it
    • [HOPSWORKS-930] - [Featurestore] bug, in API retrieval code in feature-registry featurestore is undefined
    • [HOPSWORKS-932] - Jobs UI date selector should persist user-selected values
    • [HOPSWORKS-938] - Adding pyFiles to Jupyter does not add it to python path
    • [HOPSWORKS-941] - Fix dataset access request and related tests
    • [HOPSWORKS-945] - Zipping and unzipping is broken with Hops TLS
    • [HOPSWORKS-949] - [airflow-chef] Install recipe does not create root hops installation directory
    • [HOPSWORKS-951] - sqoop and airflow bugfixes
    • [HOPSWORKS-953] - [kagent-chef] kagent falsely assumes service is dead when it restarts
    • [HOPSWORKS-955] - [airflow-chef] Use assigned roles in JWT to determine if a user is admin
    • [HOPSWORKS-958] - AirflowJWTManager randomly does not renew tokens
    • [HOPSWORKS-968] - Dataset browser pagination and select all not working
    • [HOPSWORKS-974] - Blank padded day in rspec breaks tests
    • [HOPSWORKS-975] - [airflow-chef] Change restart policy of airflow scheduler
    • [HOPSWORKS-976] - Airflow file manager does not refresh secret directory when navigate to another project
    • [HOPSWORKS-977] - Pin tornado version to 5.1.1
    • [HOPSWORKS-978] - dela::install should create /srv/hops/ before creating sub directories
    • [HOPSWORKS-979] - Cannot change default 'admin' password of [email protected]
    • [HOPSWORKS-987] - User admin to slow and user search returning duplicated results
    • [HOPSWORKS-988] - Launching Jupyter server gradually gets slower and times out
    • [HOPSWORKS-990] - uploading big files from web ui can cause 'invalidated token'
    • [HOPSWORKS-992] - [kagent-chef] Invalid command in kagent's service status script
    • [HOPSWORKS-993] - [Feturestore] tf record schema infer for float arrays is not working correctly
    • [HOPSWORKS-996] - Jobs monitor threads may saturate the system
    • [HOPSWORKS-1004] - YarnJobMonitor does not have a full list of Yarn running states
    • [HOPSWORKS-1007] - Install version specific kernel packages, not generic ones
    • [HOPSWORKS-1012] - Metadata designer does not work
    • [HOPSWORKS-1017] - Date in field "feature store created" is NaN
    • [HOPSWORKS-1022] - NOT_START is not a final state for Log Aggregation

    Task

    • [HOPSWORKS-959] - Remove apache2 dependency from cookbooks
    • [HOPSWORKS-984] - Remove commons-fileupload dependency from kmon and admin modules
    Source code(tar.gz)
    Source code(zip)
  • v0.9.0(Feb 13, 2019)

        Release notes - Hopsworks - Version 0.9.0
    

    Bug

    • [HOPSWORKS-546] - Multiple Jupyter notebooks opened for the same user cause PersistenceException
    • [HOPSWORKS-727] - Modify influxdb template to account for fqdn
    • [HOPSWORKS-846] - Wrong metrics in multiple NN setup
    • [HOPSWORKS-865] - Pip search fallback mechanism should catch Exception on the REST request
    • [HOPSWORKS-868] - Remove usages of case node['platform'] to allow RHEL installation
    • [HOPSWORKS-871] - GPUs can get stuck, need nvidia-smi command to restart them
    • [HOPSWORKS-876] - Add nvidia/driver_version to metadata.rb
    • [HOPSWORKS-888] - featurestore: Infer tf-record-schema does not work with bigint types
    • [HOPSWORKS-891] - [Featurestore]: bug, can't infer join-string in edge-case
    • [HOPSWORKS-895] - [Featurestore] bug, get_project_featurestore can return wrong result if project name contains upper-case letters
    • [HOPSWORKS-896] - [kagent-chef] Fix broken Anaconda GC command
    • [HOPSWORKS-899] - deep learning tour broken after running feature store tour
    • [HOPSWORKS-902] - tensorboard.sh script uses hopworks users as group in chmod
    • [HOPSWORKS-907] - Crypto material lock gets vanished in CertificateMaterializer
    • [HOPSWORKS-910] - Lock down hive-site.xml
    • [HOPSWORKS-925] - Hopsworks common dependency on jwt causes persistence unit issues
    • [HOPSWORKS-933] - Enable/Disable Airflow microservice for projects

    Sub-task

    • [HOPSWORKS-752] - Add support for running notebooks (.ipynb) in Job service
    • [HOPSWORKS-893] - Remove Zeppelin service icon from Hopsworks

    New Feature

    Improvement

    • [HOPSWORKS-755] - Fix minor bugs/semantic issues in hops-util-py
    • [HOPSWORKS-855] - Display readable error in case of naming conflict when sharing datasets
    • [HOPSWORKS-870] - Support PIP configuration for internal PyPi servers/proxies
    • [HOPSWORKS-872] - Job's configuration should be editable
    • [HOPSWORKS-873] - Change featurestore.create_training_dataset semantics to allow overwrite
    • [HOPSWORKS-879] - Add flag to not install epel-repo on Centos/Rhel
    • [HOPSWORKS-880] - Make sure umask is 022 before creating envs/pip install
    • [HOPSWORKS-887] - Add new variable to enable/disable downloading files
    • [HOPSWORKS-894] - InferenceController implementations should use connection pooling
    • [HOPSWORKS-906] - Use experiments service in feature store tour
    • [HOPSWORKS-921] - Increase timeout for project re-index in epipe

    Task

    • [HOPSWORKS-838] - Upgrade Spark to 2.4.0
    • [HOPSWORKS-866] - Support JobName in addition to JobId as arguments in the Feature Store API
    • [HOPSWORKS-867] - Move from Oracle JDK to OpenJDK from the package manager
    • [HOPSWORKS-898] - Add rpc size related configs in Livy
    • [HOPSWORKS-918] - Bump Hops depdendency in Hopsworks to 2.8.2.7
    • [HOPSWORKS-919] - [hopsworks-chef] Create Airflow connection pool in Glassfish on updates
    • [HOPSWORKS-922] - (chef) Update kafka attributes
    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(Jan 23, 2019)

        Release notes - Hopsworks - Version 0.8.1
    

    Bug

    • [HOPSWORKS-727] - Modify influxdb template to account for fqdn
    • [HOPSWORKS-865] - Pip search fallback mechanism should catch Exception on the REST request
    • [HOPSWORKS-868] - Remove usages of case node['platform'] to allow RHEL installation
    • [HOPSWORKS-871] - GPUs can get stuck, need nvidia-smi command to restart them
    • [HOPSWORKS-876] - Add nvidia/driver_version to metadata.rb
    • [HOPSWORKS-878] - Moving files in Hopsworks datasets browser does not work
    • [HOPSWORKS-883] - [conda-chef] Installation of kagent_utils fail
    • [HOPSWORKS-885] - Jobs service date field should account for clock skew

    Improvement

    • [HOPSWORKS-870] - Support PIP configuration for internal PyPi servers/proxies

    Task

    • [HOPSWORKS-867] - Move from Oracle JDK to OpenJDK from the package manager
    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Jan 7, 2019)

        Release notes - Hopsworks - Version 0.8.0
    

    Bug

    • [HOPSWORKS-845] - IllegalStateException in TensorBoardProxyServlet
    • [HOPSWORKS-850] - Experiments dashboard should be created when jobs service is added
    • [HOPSWORKS-851] - Convert .ipynb to .py uses wrong jupyter binary path
    • [HOPSWORKS-857] - Spark executor UI not displaying whe high number of executors
    • [HOPSWORKS-861] - Deep Learning demo shows "Jupyter not installed" on Jupyter page

    Sub-task

    New Feature

    Task

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Dec 20, 2018)

        Release notes - Hopsworks - Version 0.7.0
    

    Bug

    • [HOPSWORKS-713] - Tame setup-chef
    • [HOPSWORKS-801] - Jupyter application view/kill buttons not visible in low resolutions
    • [HOPSWORKS-803] - Add Elastic dependency to Berksfile in hops-hadoop-chef
    • [HOPSWORKS-805] - Add rm dependency to nm in hops-hadoop-chef karamelfile
    • [HOPSWORKS-807] - fix yarn.nodemanager.aux-services in mapred-site.xml
    • [HOPSWORKS-808] - fix hops cluster definition in karamel-chef
    • [HOPSWORKS-812] - TensorBoard poller running too often
    • [HOPSWORKS-815] - Mail sent from dataset request addresses wrong user
    • [HOPSWORKS-820] - Only one instance of JAXBContext should be created per JobConfiguration.
    • [HOPSWORKS-823] - Jupyter cleanup for Project deletion never worked
    • [HOPSWORKS-824] - [conda-chef] Template conda repo only if provided for hops-system environment
    • [HOPSWORKS-825] - Review projectmember removal and jobs
    • [HOPSWORKS-830] - conda hops-system env fails the upgrade from 0.6.* to 0.7.0
    • [HOPSWORKS-833] - NPE when getting the state of CertificateMaterializer
    • [HOPSWORKS-843] - TensorBoard cleans up certificates twice
    • [HOPSWORKS-847] - Add Sparkmagic support for utf-8 chars in configuration file

    Sub-task

    Improvement

    Task

    Source code(tar.gz)
    Source code(zip)
  • v0.6.1(Nov 21, 2018)

  • v0.6.0(Nov 13, 2018)

        Release notes - Hopsworks - Version 0.6.0
    

    Bug

    • [HOPSWORKS-224] - Vizops driver and executor tabs do not display data
    • [HOPSWORKS-369] - Restrict characters allowed in project names
    • [HOPSWORKS-429] - Fix configuration of TFSpark in JobService
    • [HOPSWORKS-526] - checkstyle rules out of date for Java 8
    • [HOPSWORKS-587] - Chef recipes bug fixes and common install version
    • [HOPSWORKS-594] - Fix compilation bug in tensorflow::default
    • [HOPSWORKS-599] - Kafka tour broken because of HOPSWORKS-587
    • [HOPSWORKS-602] - Update Hopsworks settings in cluster definition breaks flyway migrate
    • [HOPSWORKS-604] - Review OpenMPI and CUDA upgrade
    • [HOPSWORKS-605] - Chef: fix remote path for hops-examples and hops-util
    • [HOPSWORKS-608] - Put back the code for downloading hops-tensorflow.jar
    • [HOPSWORKS-610] - The project logging to ops_log should be done after the project is assigned an inode
    • [HOPSWORKS-614] - [kagent-chef] Be consistent on the separator in config.ini
    • [HOPSWORKS-615] - Kagent-chef upgrade broken
    • [HOPSWORKS-617] - Hopsworks-579 breaks build with -Pcluster profile
    • [HOPSWORKS-619] - nvidia-smi might not be available in hops::install to generate /tmp/num_gpus
    • [HOPSWORKS-620] - Hops cookbook broken for configuration update
    • [HOPSWORKS-624] - Proxy not handling capital letters in url
    • [HOPSWORKS-632] - Regex too strict for yarn quota updates in admin ui
    • [HOPSWORKS-635] - Enable logging for Livy
    • [HOPSWORKS-636] - Only install nccl2 if chef attribute cuda/download_terms/accept is 'true'
    • [HOPSWORKS-646] - Inconsistent metadata due to parallel data readers
    • [HOPSWORKS-647] - Unshare dataset UI not working
    • [HOPSWORKS-650] - Job-monitor ui broken
    • [HOPSWORKS-652] - [hopslog-chef] Change primary group of yarn user to hadoop
    • [HOPSWORKS-653] - Driver and Executor links broken in Executors tab in Spark UI
    • [HOPSWORKS-660] - Zip/Unzip changes ownership for files in same directory as the unzipped folder or zipped file
    • [HOPSWORKS-663] - Admin UI is broken because of missing methods in Hosts
    • [HOPSWORKS-666] - MMLSpark breaks reproducible environments
    • [HOPSWORKS-668] - ProjectController ignores failures for adding Jupyter dataset and Hive service
    • [HOPSWORKS-670] - Conda search broken on master
    • [HOPSWORKS-675] - Handling of agent heartbeat could result in transactions timing-out
    • [HOPSWORKS-677] - [hops-hadoop-chef] Hopsworks host is not interpolated in yarn-site.xml
    • [HOPSWORKS-681] - Wrong filename for certificates (.cert.pem appended twice)
    • [HOPSWORKS-682] - Installed Python libraries do not change status
    • [HOPSWORKS-684] - TensorBoard broken for single Experiment in JobUI
    • [HOPSWORKS-691] - "Unhealthy HDFS blocks" display badly in admin pannel when running multiple NN
    • [HOPSWORKS-696] - /previewFile in DataSetService does not filter out control codes from JSON response
    • [HOPSWORKS-701] - JobService proxy should not set host config for client
    • [HOPSWORKS-704] - TensorBoard not showing events in Experiments service
    • [HOPSWORKS-707] - Lib status for anaconda envs still is broken
    • [HOPSWORKS-731] - Add additional Distributed Training configuration check to JupyterConfigFilesGenerator
    • [HOPSWORKS-746] - TfLibMapping missing TensorFlow 1.8
    • [HOPSWORKS-753] - Change default location of Unix MySQL socket for Rspec
    • [HOPSWORKS-756] - Jupyter NPE when starting it before Python service is enabled
    • [HOPSWORKS-760] - elastic-chef don't use the elasticsearch lwrp to create the user
    • [HOPSWORKS-766] - If hopsworks.version is 0.6.0 replace flyway_schema_history table
    • [HOPSWORKS-768] - appservice tests fail randomly
    • [HOPSWORKS-772] - Chef replace provider uses local path
    • [HOPSWORKS-773] - Usernames should not contain any special char
    • [HOPSWORKS-777] - Hopsworks administrator should be able to delete Projects
    • [HOPSWORKS-779] - [ePipe] Fix a race condition while reading from ndb
    • [HOPSWORKS-783] - INFO log severity in UserStatusValidator is polluting the logs
    • [HOPSWORKS-786] - Fix file path for 0.6.0 flyway migration table
    • [HOPSWORKS-794] - flyway schema history should be replaced only if the last migration was 0.5.0
    • [HOPSWORKS-795] - Jupyter does not have a preselected mode set
    • [HOPSWORKS-797] - Swagger cannot generate json
    • [HOPSWORKS-799] - Lowercase project name for TfServing logs

    Sub-task

    • [HOPSWORKS-244] - Replace deprecated _ttl field in elasticsearch
    • [HOPSWORKS-251] - Replace log4j socketAppender for Logstash
    • [HOPSWORKS-299] - Add functionality to cancel job schedule
    • [HOPSWORKS-509] - REST API v2 - Refactor error handling
    • [HOPSWORKS-558] - Upgrade hopslog-chef and elastic-chef
    • [HOPSWORKS-559] - Upgrade Hopsworks to work with latest elasticsearch
    • [HOPSWORKS-560] - Upgrade ePipe to work with latest elasticsearch
    • [HOPSWORKS-584] - REST Api for Tensorflow Serving
    • [HOPSWORKS-694] - Notebooks for Kafka tour (Python/Scala)
    • [HOPSWORKS-698] - Add Kafka Python documentation to hops-doc
    • [HOPSWORKS-699] - Add kafka python dependencies to anaconda base environment
    • [HOPSWORKS-737] - Update dela config template according to updated dropwizard version
    • [HOPSWORKS-738] - Remove configurable attributes that can be retrieved from kagent attributes (ex. mysql ip/port)
    • [HOPSWORKS-741] - Upgrade livy to work with Spark 2.3.2
    • [HOPSWORKS-758] - Bump py4j version in Hopsworks and make it configurable at runtime
    • [HOPSWORKS-784] - Job status stuck in "Starting ApplicationMaster"
    • [HOPSWORKS-789] - hops-util-py should account for elastic index being lowercase
    • [HOPSWORKS-790] - KibanaProxyServlet and TensorBoardService lowercase projectname

    New Feature

    Improvement

    • [HOPSWORKS-256] - Project name should be equal regardless of capitalization
    • [HOPSWORKS-357] - Move public dataset size to details
    • [HOPSWORKS-451] - karamel override attributes for node groups
    • [HOPSWORKS-452] - Add support for rebuilding Elasticsearch index in ePipe
    • [HOPSWORKS-564] - Refactor ePipe to easily enable or disable different watch units
    • [HOPSWORKS-565] - Update ePipe start command
    • [HOPSWORKS-578] - supply custom tensorlfow wheel to chef for installation
    • [HOPSWORKS-579] - Add to hopsworks-ca ability to sign csr using multiple CAs
    • [HOPSWORKS-591] - Admin UI (Conda): Show disk usage for conda, allow cleanup
    • [HOPSWORKS-595] - [kagent] Persistent state store for kagent
    • [HOPSWORKS-601] - [kagent] Remove Chef attributes from Python code
    • [HOPSWORKS-607] - Improve TensorFlow demo tour
    • [HOPSWORKS-611] - Unset MetaEnabled on the dataset before deleting to avoid redundant calls to Elasticsearch
    • [HOPSWORKS-613] - Add support for application certificate versioning
    • [HOPSWORKS-631] - Integrate Pydoop HDFS API into Hops API
    • [HOPSWORKS-643] - Refactor TfServing code to allow pluggable providers
    • [HOPSWORKS-644] - Kagent improvements: multiple params from single recipe & my_hostname function
    • [HOPSWORKS-645] - [hops-hadoop-chef] Enable encryption for the block transfer
    • [HOPSWORKS-648] - Conda envs Garbage Collection
    • [HOPSWORKS-667] - Unify Job logs and Jupyter Elasticsearch index and Kibana objects creation
    • [HOPSWORKS-674] - Parallelize materialization of certificates
    • [HOPSWORKS-676] - Make SparkUI logs intuitive
    • [HOPSWORKS-690] - Add button to stop particular Livy session in Jupyter dashboard
    • [HOPSWORKS-697] - Refine structure of keystores and truststores in Hops
    • [HOPSWORKS-705] - Extend TfLibMapping to support ongoing environment creation
    • [HOPSWORKS-709] - Link to hops-util-py docs in Python service
    • [HOPSWORKS-725] - [kagent-chef] Make Conda GC blacklist more generic
    • [HOPSWORKS-726] - [hops-hadoop-chef] Update Hopsworks CA endpoints in yarn-site.xml
    • [HOPSWORKS-736] - dela updates and fixes
    • [HOPSWORKS-757] - Chef improvements for upgrading ELK stack
    • [HOPSWORKS-761] - Run Expat at the end of the installation
    • [HOPSWORKS-771] - Allow memory quota configuration in yarn

    Task

    Source code(tar.gz)
    Source code(zip)
Owner
Logical Clocks AB
Developers of Hopsworks
Logical Clocks AB
An open source python library for automated feature engineering

"One of the holy grails of machine learning is to automate more and more of the feature engineering process." ― Pedro Domingos, A Few Useful Things to

alteryx 6.2k Jun 20, 2022
Unofficial third-party implementation of FFD (fast feature detector) published in IEEE TIP 2020.

fast_feature_detector Unofficial third-party implementation of FFD (fast feature detector) published in IEEE TIP 2020. Caution I have not got any perm

kamino410 12 Feb 17, 2022
FG-Net: Fast Large-Scale LiDAR Point Clouds Understanding Network Leveraging Correlated Feature Mining and Geometric-Aware Modelling

FG-Net: Fast Large-Scale LiDAR Point Clouds Understanding Network Leveraging Correlated Feature Mining and Geometric-Aware Modelling Comparisons of Running Time of Our Method with SOTA methods RandLA and KPConv:

Kangcheng LIU 68 Jun 23, 2022
LIDAR(Livox Horizon) point cloud preprocessing, including point cloud filtering and point cloud feature extraction (edge points and plane points)

LIDAR(Livox Horizon) point cloud preprocessing, including point cloud filtering and point cloud feature extraction (edge points and plane points)

hongyu wang 10 Apr 12, 2022
Implementation of Sift feature detection and matching in C + +

Sift-In-CPP This is SIFT feature detection and matching implemented in C + + Environment version information: VS2017、Opencv3.4.3 Reference link: 1.htt

null 1 Nov 19, 2021
Distributed machine learning platform

Veles Distributed platform for rapid Deep learning application development Consists of: Platform - https://github.com/Samsung/veles Znicz Plugin - Neu

Samsung 897 May 28, 2022
Machine Learning Platform for Kubernetes

Reproduce, Automate, Scale your data science. Welcome to Polyaxon, a platform for building, training, and monitoring large scale deep learning applica

polyaxon 3.1k Jun 23, 2022
MediaPipe offers cross-platform, customizable ML solutions for live and streaming media.

Cross-platform, customizable ML solutions for live and streaming media.

Google 17.7k Jun 25, 2022
ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

ONNX Runtime is a cross-platform inference and training machine-learning accelerator compatible with deep learning frameworks, PyTorch and TensorFlow/Keras, as well as classical machine learning libraries such as scikit-learn, and more.

Microsoft 7k Jun 25, 2022
from Microsoft STL, but multi-platform

mscharconv <charconv> from Microsoft STL, but multi-platform. Tested with MSVC, gcc, and clang on Windows, Linux, and macOS (will likely work on other

Borislav Stanimirov 35 Jun 8, 2022
Zero-latency convolution on Bela platform

bela-zlc Zero-latency convolution on Bela platform | Report | Video | Overview Convolution has many applications in audio, such as equalization and ar

Christian J. Steinmetz 19 Jun 25, 2022
Read and write rosbag on a platform without ROS installed, using MQTT for message delivery.

EasyRosBag x86_64: Test on Ubuntu18.04 arm64 : Test on Ubuntu21.04(raspberry Pi4) Introducton ROS(Robot Operation System) is too fat!!! We do not ins

afei 9 May 23, 2022
A C++-based, cross platform ray tracing library

Visionaray A C++ based, cross platform ray tracing library Getting Visionaray The Visionaray git repository can be cloned using the following commands

Stefan Zellmann 404 Jun 11, 2022
ncnn is a high-performance neural network inference framework optimized for the mobile platform

ncnn ncnn is a high-performance neural network inference computing framework optimized for mobile platforms. ncnn is deeply considerate about deployme

Tencent 14.8k Jun 22, 2022
A Cross platform implement of Wenet ASR. It's based on ONNXRuntime and Wenet. We provide a set of easier APIs to call wenet models.

RapidASR: a new member of RapidAI family. Our visio is to offer an out-of-box engineering implementation for ASR. A cpp implementation of recognize-on

RapidAI-NG 70 Jun 20, 2022
RapidOCR - A cross platform OCR Library based on PaddleOCR & OnnxRuntime

RapidOCR (捷智OCR) 简体中文 | English 目录 RapidOCR (捷智OCR) 简介 近期更新 ?? 2021-12-18 update 2021-11-28 update 2021-11-13 update 2021-10-27 update 2021-09-13 upda

RapidAI-NG 419 Jun 26, 2022
ClanLib is a cross platform C++ toolkit library.

ClanLib ClanLib is a cross platform toolkit library with a primary focus on game creation. The library is Open Source and free for commercial use, und

Kenneth Gangstø 299 Jun 21, 2022
Insight Toolkit (ITK) is an open-source, cross-platform toolkit for N-dimensional scientific image processing, segmentation, and registration

ITK: The Insight Toolkit C++ Python Linux macOS Windows Linux (Code coverage) Links Homepage Download Discussion Software Guide Help Examples Issue tr

Insight Software Consortium 1k Jun 26, 2022
Gesture Recognition Toolkit (GRT) is a cross-platform, open-source, C++ machine learning library designed for real-time gesture recognition.

Gesture Recognition Toolkit (GRT) The Gesture Recognition Toolkit (GRT) is a cross-platform, open-source, C++ machine learning library designed for re

Nicholas Gillian 777 Jun 19, 2022