Learning embeddings for classification, retrieval and ranking.

Overview

StarSpace

StarSpace is a general-purpose neural model for efficient learning of entity embeddings for solving a wide variety of problems:

  • Learning word, sentence or document level embeddings.
  • Information retrieval: ranking of sets of entities/documents or objects, e.g. ranking web documents.
  • Text classification, or any other labeling task.
  • Metric/similarity learning, e.g. learning sentence or document similarity.
  • Content-based or Collaborative filtering-based Recommendation, e.g. recommending music or videos.
  • Embedding graphs, e.g. multi-relational graphs such as Freebase.
  • Image classification, ranking or retrieval (e.g. by using existing ResNet features).

In the general case, it learns to represent objects of different types into a common vectorial embedding space, hence the star ('*', wildcard) and space in the name, and in that space compares them against each other. It learns to rank a set of entities/documents or objects given a query entity/document or object, which is not necessarily the same type as the items in the set.

See the paper for more details on how it works.

News

  • StarSpace is available in Python: check out the Building StarSpace section for details.
  • Support reading from compressed file: check out the Compressed File section for more details.
  • New license and patents: now StarSpace is under MIT license. Checkout LICENSE for details.
  • StarSpace training is much faster now with mini batch training (setting batch size by "-batchSize" argument). Details in #190.
  • We added support for real-valued input and label weights: checkout the File Format and ImageSpace section for more details on how to use weights in input and label.

Requirements

StarSpace builds on modern Mac OS, Windows, and Linux distributions. Since it uses C++11 features, it requires a compiler with good C++11 support. These include :

  • (gcc-4.6.3 or newer), (Visual Studio 2015), or (clang-3.3 or newer)

Compilation is carried out using a Makefile, so you will need to have a working make.

You need to install Boost library and specify the path of boost library in makefile in order to run StarSpace. Basically:

$wget https://dl.bintray.com/boostorg/release/1.63.0/source/boost_1_63_0.zip
$unzip boost_1_63_0.zip
$sudo mv boost_1_63_0 /usr/local/bin

Optional: if one wishes to run the unit tests in src directory, google test is required and its path needs to be specified in 'TEST_INCLUDES' in the makefile.

Building StarSpace

In order to build StarSpace on Mac OS or Linux, use the following:

git clone https://github.com/facebookresearch/Starspace.git
cd Starspace
make

In order to build StarSpace on Windows, open the following in Visual Studio:

MVS\StarSpace.sln

In order to build StarSpace python wrapper, please refer README inside the directory python.

File Format

StarSpace takes input files of the following format. Each line will be one input example, in the simplest case the input has k words, and each labels 1..r is a single word:

word_1 word_2 ... word_k __label__1 ... __label__r

This file format is the same as in fastText. It assumes by default that labels are words that are prefixed by the string __label__, and the prefix string can be set by "-label" argument.

In order to learn the embeddings, do:

$./starspace train -trainFile data.txt -model modelSaveFile

where data.txt is a training file containing utf-8 encoded text. At the end of optimization the program will save two files: model and modelSaveFile.tsv. modelSaveFile.tsv is a standard tsv format file containing the entity embedding vectors, one per line. modelSaveFile is a binary file containing the parameters of the model along with the dictionary and all hyper parameters. The binary file can be used later to compute entity embedding vectors or to run evaluation tasks.

In the more general case, each label also consists of words:

word_1 word_2 ... word_k <tab> label_1_word_1 label_1_word_2 ... <tab> label_r_word_1 .. 

Embedding vectors will be learned for each word and label to group similar inputs and labels together.

In order to learn the embeddings in the more general case where each label consists of words, one needs to specify the -fileFormat flag to be 'labelDoc', as follows:

$./starspace train -trainFile data.txt -model modelSaveFile -fileFormat labelDoc

We also extend the file format to support real-valued weights (in both input and label space) by setting argument "-useWeight" to true (default is false). If "-useWeight" is true, we support weights by the following format

word_1:wt_1 word_2:wt_2 ... word_k:wt_k __label__1:lwt_1 ...    __label__r:lwt_r

e.g.,

dog:0.1 cat:0.5 ...

The default weight is 1 for any word / label that does not contain weights.

Compressed File

StarSpace can also read from compressed file (currently only support gzip files). You can skip this part if you do not plan to use compressed input files. To run StarSpace with compressed input, first compile StarSpace using makefile_compress instead of makefile:

make -f makefile_compress

Then in the train config, specify

./starspace -trainFile input -compressFile gzip -numGzFile 10 ...

It assumes that there are input files with names

input00.gz, input01.gz, ..., input09.gz 

and reads from those files.

To prepare data in this format, one can use the standard 'split' function to first split input file into multiple chunks, then compress them. For instance:

split -d -l xxx original_input.txt input && gzip input*

Training Mode

To explain how it works in different train modes, we call the input of a particular example as "LHS" (stands for left-hand-side) and the label as "RHS" (stands for right-hand-side). StarSpace supports the following training modes (the default is the first one):

  • trainMode = 0:
    • Each example contains both input and labels.
    • If fileFormat is 'fastText' then the labels are individuals features/words specified (e.g. with a prefix label, see file format above).
    • Use case: classification tasks, see tagspace example below.
    • If fileFormat is 'labelDoc' then the labels are bags of features, and one of those bags is selected (see file format, above).
    • Use case: retrieval/search tasks, each example consists of a query followed by a set of relevant documents.
  • trainMode = 1:
    • Each example contains a collection of labels. At training time, one label from the collection is randomly picked as the RHS, and the rest of the labels in the collection become the LHS.
    • Use case: content-based or collaborative filtering-based recommendation, see pagespace example below.
  • trainMode = 2:
    • Each example contains a collection of labels. At training time, one label from the collection is randomly picked as the LHS, and the rest of the labels in the collection become the RHS.
    • Use case: learning a mapping from an object to a set of objects of which it is a part, e.g. sentence (from within document) to document.
  • trainMode = 3:
    • Each example contains a collection of labels. At training time, two labels from the collection are randomly picked as the LHS and RHS.
    • Use case: learn pairwise similarity from collections of similar objects, e.g. sentence similiarity.
  • trainMode = 4:
    • Each example contains two labels. At training time, the first label from the collection will be picked as the LHS and the second label will be picked as the RHS.
    • Use case: learning from multi-relational graphs.
  • trainMode = 5:
    • Each example contains only input. At training time, it generates multiple training examples: each feature from input is picked as the RHS, and other features surronding it (up to distance ws) are picked as the LHS.
    • Use case: learn word embeddings in unsupervised way.

Example use cases

TagSpace word / tag embeddings

Setting: Learning the mapping from a short text to relevant hashtags, e.g. as in this paper. This is a classical classification setting.

Model: the mapping learnt goes from bags of words to bags of tags, by learning an embedding of both. For instance, the input “restaurant has great food <\tab> #restaurant <\tab> #yum” will be translated into the following graph. (Nodes in the graph are entities for which embeddings will be learned, and edges in the graph are relationships between the entities).

word-tag

Input file format:

restaurant has great food #yum #restaurant

Command:

$./starspace train -trainFile input.txt -model tagspace -label '#'

Example scripts:

We apply the model to the problem of text classification on AG's News Topic Classification Dataset. Here our tags are news article categories, and we use the [email protected] metric to measure classification accuracy. This example script downloads the data and run StarSpace model on it under the examples directory:

$bash examples/classification_ag_news.sh

PageSpace user / page embeddings

Setting: On Facebook, users can fan (follow) public pages they're interested in. When a user fans a page, the user can receive all things the page posts on Facebook. We want to learn page embeddings based on users' fanning data, and use it to recommend users new pages they might be interested to fan (follow). This setting can be generalized to other recommendation problems: for instance, embedding and recommending movies to users based on movies watched in the past; embed and recommend restaurants to users based on the restaurants checked-in by users in the past, etc.

Model: Users are represented as the bag of pages that they follow (fan). That is, we do not learn a direct embedding of users, instead, each user will have an embedding which is the average embedding of pages fanned by the user. Pages are embedded directly (with a unique feature in the dictionary). This setup can work better in the case where the number of users is larger than the number of pages, and the number of pages fanned by each user is small on average (i.e. the edges between user and page is relatively sparse). It also generalizes to new users without retraining. However, the more traditional recommendation setting can also be used.

user-page

Each user is represented by the bag-of-pages fanned by the user, and each training example is a single user.

Input file format:

page_1 page_2 ... page_M

At training time, at each step for each example (user), one random page is selected as a label and the rest of bag of pages are selected as input. This can be achieved by setting flag -trainMode to 1.

Command:

$./starspace train -trainFile input.txt -model pagespace -label 'page' -trainMode 1

Example scripts:

To provide an example script, we choose the Last.FM (http://www.lastfm.com) dataset from HectRec 2011 and model it similarly as in the PageSpace setting: user is represented by the bag-of-artitsts listened by the user.

 $bash examples/recomm_user_artists.sh

DocSpace document recommendation

Setting: We want to embed and recommend web documents for users based on their historical likes/click data.

Model: Each document is represented by a bag-of-words of the document. Each user is represented as a (bag of) the documents that they liked/clicked in the past. At training time, at each step one random document is selected as the label and the rest of the bag of documents are selected as input.

user-doc

Input file format:

roger federer loses <tab> venus williams wins <tab> world series ended
i love cats <tab> funny lolcat links <tab> how to be a petsitter  

Each line is a user, and each document (documents separated by tabs) are documents that they liked. So the first user likes sports, and the second is interested in pets in this case.

Command:

./starspace train -trainFile input.txt -model docspace -trainMode 1 -fileFormat labelDoc

GraphSpace: Link Prediction in Knowledge Bases

Setting: Learning the mapping between entities and relations in Freebase. In freebase, data comes in the format

(head_entity, relation_type, tail_entity)

Performing link prediction can be formalized as filling in incomplete triples like

(head_entity, relation_type, ?) or (?, relation_type, tail_entity)

Model: We learn the embeddings of all entities and relation types. For each relation_type, we learn two embeddings: one for predicting tail_entity given head_entity, one for predicting head_entity given tail_entity.

multi-rel

Example scripts:

This example script downloads the Freebase15k data from here and runs the StarSpace model on it:

$bash examples/multi_relation_example.sh

SentenceSpace: Learning Sentence Embeddings

Setting: Learning the mapping between sentences. Given the embedding of one sentence, one can find semantically similar/relevant sentences.

Model: Each example is a collection of sentences which are semantically related. Two are picked at random using trainMode 3: one as the input and one as the label, other sentences are picked as random negatives. One easy way to obtain semantically related sentences without labeling is to consider all sentences in the same document are related, and then train on those documents.

sentences

Example scripts:

This example script downloads data where each example is a set of sentences from the same Wikipedia page and runs the StarSpace model on it:

$bash examples/wikipedia_sentence_matching.sh

To run the full experiment on Wikipedia Sentence Matching presented in this paper, use this script (warning: it takes a long time to download data and train the model):

$bash examples/wikipedia_sentence_matching_full.sh

ArticleSpace: Learning Sentence and Article Embeddings

Setting: Learning the mapping between sentences and articles. Given the embedding of one sentence, one can find the most relevant articles.

Model: Each example is an article which contains multiple sentences. At training time, one sentence is picked at random as the input, the remaining sentences in the article becomes the label, other articles are picked as random negatives (trainMode 2).

Example scripts:

This example script downloads data where each example is a Wikipedia article and runs the StarSpace model on it:

$bash examples/wikipedia_article_search.sh

To run the full experiment on Wikipedia Article Search presented in this paper, use this script (warning: it takes a long time to download data and train the model):

$bash examples/wikipedia_article_search_full.sh

ImageSpace: Learning Image and Label Embeddings

With the most recent update, StarSpace can also be used to learn joint embeddings with images and other entities. For instance, one can use ResNet features (the last layer of a pre-trained ResNet model) to represent an image, and embed images with other entities (words, hashtags, etc.). Just like other entities in Starspace, images can be either on the input or the label side, depending on your task.

Here we give an example using CIFAR-10 to illustrate how we train images with other entities (in this example, image class): we train a ResNeXt model on CIFAR-10 which achieves 96.34% accuracy on test dataset, and use the last layer of ResNeXt as the features for each image. We embed 10 image classes together with image features in the same space using StarSpace. For an example image from class 1 with last layer (0.8, 0.5, ..., 1.2), we convert it to the following format:

d0:0.8  d1:0.5   ...    d1023:1.2   __label__1

After converting train and test examples of CIFAR-10 to the above format, we ran this example script:

$bash examples/image_feature_example_cifar10.sh

and achieved 96.40% accuracy on an average of 5 runs.

Full Documentation of Parameters

Run "starspace train ..."  or "starspace test ..."

The following arguments are mandatory for train: 
  -trainFile       training file path
  -model           output model file path

The following arguments are mandatory for test: 
  -testFile        test file path
  -model           model file path

The following arguments for the dictionary are optional:
  -minCount        minimal number of word occurences [1]
  -minCountLabel   minimal number of label occurences [1]
  -ngrams          max length of word ngram [1]
  -bucket          number of buckets [2000000]
  -label           labels prefix [__label__]. See file format section.

The following arguments for training are optional:
  -initModel       if not empty, it loads a previously trained model in -initModel and carry on training.
  -trainMode       takes value in [0, 1, 2, 3, 4, 5], see Training Mode Section. [0]
  -fileFormat      currently support 'fastText' and 'labelDoc', see File Format Section. [fastText]
  -validationFile  validation file path
  -validationPatience    number of iterations of validation where does not improve before we stop training [10]
  -saveEveryEpoch  save intermediate models after each epoch [false]
  -saveTempModel   save intermediate models after each epoch with an unique name including epoch number [false]
  -lr              learning rate [0.01]
  -dim             size of embedding vectors [100]
  -epoch           number of epochs [5]
  -maxTrainTime    max train time (secs) [8640000]
  -negSearchLimit  number of negatives sampled [50]
  -maxNegSamples   max number of negatives in a batch update [10]
  -loss            loss function {hinge, softmax} [hinge]
  -margin          margin parameter in hinge loss. It's only effective if hinge loss is used. [0.05]
  -similarity      takes value in [cosine, dot]. Whether to use cosine or dot product as similarity function in  hinge loss.
                   It's only effective if hinge loss is used. [cosine]
  -p               normalization parameter: we normalize sum of embeddings by deviding Size^p, when p=1, it's equivalent to taking average of embeddings; when p=0, it's equivalent to taking sum of embeddings. [0.5]
  -adagrad         whether to use adagrad in training [1]
  -shareEmb        whether to use the same embedding matrix for LHS and RHS. [1]
  -ws              only used in trainMode 5, the size of the context window for word level training. [5]
  -dropoutLHS      dropout probability for LHS features. [0]
  -dropoutRHS      dropout probability for RHS features. [0]
  -initRandSd      initial values of embeddings are randomly generated from normal distribution with mean=0, standard deviation=initRandSd. [0.001]
  -trainWord       whether to train word level together with other tasks (for multi-tasking). [0]
  -wordWeight      if trainWord is true, wordWeight specifies example weight for word level training examples. [0.5]
  -batchSize       size of mini batch in training. [5]

The following arguments for test are optional:
  -basedoc         file path for a set of labels to compare against true label. It is required when -fileFormat='labelDoc'.
                   In the case -fileFormat='fastText' and -basedoc is not provided, we compare true label with all other labels in the dictionary.
  -predictionFile  file path for save predictions. If not empty, top K predictions for each example will be saved.
  -K               if -predictionFile is not empty, top K predictions for each example will be saved.
  -excludeLHS      exclude elements in the LHS from predictions

The following arguments are optional:
  -normalizeText   whether to run basic text preprocess for input files [0]
  -useWeight       whether input file contains weights [0]
  -verbose         verbosity level [0]
  -debug           whether it's in debug mode [0]
  -thread          number of threads [10]

Note: We use the same implementation of word n-grams for words as in fastText. When "-ngrams" is set to be larger than 1, a hashing map of size specified by the "-bucket" argument is used for n-grams; when "-ngrams" is set to 1, no hash map is used, and the dictionary contains all words within the minCount and minCountLabel constraints.

Utility Functions

We also provide a few utility functions for StarSpace:

Show Predictions for Queries

A simple way to check the quality of a trained embedding model is to inspect the predictions when typing in an input. To build and use this utility function, run the following commands:

make query_predict
./query_predict <model> k [basedocs]

where "<model>" specifies a trained StarSpace model and the optional K specifies how many of the top predictions to show (top ranked first). "basedocs" points to the file of documents to rank, see also the argument of the same name in the starspace main above. If "basedocs" is not provided, the labels in the dictionary are used instead.

After loading the model, it reads a line of entities (can be either a single word or a sentence / document), and outputs the predictions.

Nearest Neighbor Queries

Another simple way to check the quality of a trained embedding model is to inspect nearest neighbors of entities. To build and use this utility function, run the following commands:

make query_nn
./query_nn <model> [k]

where "<model>" specifies a trained StarSpace model and the optional K (default value is 5) specifies how many nearest neighbors to search for.

After loading the model, it reads a line of entities (can be either a single word or a sentence / document), and output the nearest entities in embedding space.

Print Ngrams

As the ngrams used in the model are not saved in tsv format, we also provide a separate function to output n-grams embeddings from the model. To use that, run the following commands:

make print_ngrams
./print_ngrams <model>

where "<model>" specifies a trained StarSpace model with argument -ngrams > 1.

Print Sentence / Document Embedding

Sometimes it is useful to print out sentence / document embeddings from a trained model. To use that, run the following commands:

make embed_doc
./embed_doc <model> [filename]

where "<model>" specifies a trained StarSpace model. If filename is provided, it reads each sentence / document from file, line by line, and outputs vector embeddings accordingly. If the filename is not provided, it reads each sentence / document from stdin.

Citation

Please cite the arXiv paper if you use StarSpace in your work:

@article{wu2017starspace,
  title={StarSpace: Embed All The Things!},
  author = {{Wu}, L. and {Fisch}, A. and {Chopra}, S. and {Adams}, K. and {Bordes}, A. and {Weston}, J.},
  journal={arXiv preprint arXiv:{1709.03856}},
  year={2017}
}

Contact

Issues
  • Any way to reduce data file size?

    Any way to reduce data file size?

    My input space has ~1000 features, and my training data files are topping 5GB just for 200,000 samples. Since the training data are provided as the -trainFile argument, rather than read from stdin, I'm not sure how I can get StarSpace to read compressed files. Is there anything I'm missing, there? Any support for file compression or more compact formats?

    opened by dhmay 23
  • Performance comparison with fastText

    Performance comparison with fastText

    Hi, a note on performance,

    I am testing the same input file with fastText & StarSpace,

    fastText running time is ~45 mins with params:

    ./fasttext cbow -minCount 100 -dim 100  -input input.txt.txt -output cbow_mincount_dim_100
    

    StarSpace running time is ~5hrs with params:

    ./starspace train -trainFile input.txt -model sp_100_10 -trainMode 5 -dim 10 -minCount 100 -loss hinge -verbose true
    

    Input file in lines is 15.2m, vocabulary size with minCount 100 is 95K words

    Is this expected behaviour or am I missing something obvious here?

    Thanks!

    opened by kperi 19
  • Example for PageSpace user / page embeddings

    Example for PageSpace user / page embeddings

    Could you please provide an example script with sample (anonymized of course) data of the PageSpace user / page embeddings in order to test the recommendations task.

    opened by loretoparisi 18
  • Add weight importance to file format

    Add weight importance to file format

    In many cases, it makes sense to say that an observation is more important than another one. Right now, there is no obvious way to provide this information. Would it be possible to add such feature? It would look like an optional key word to add to each sample (for instance __weight__:2.5). The effect would be to increase the loss (and the correction) for that sample.

    What do you think?

    opened by pommedeterresautee 17
  • Python wrapper for starSpace

    Python wrapper for starSpace

    Hi, I have wrote a python wrapper for starSpace. It uses pybind11 can be installed with conan package manager (conan is required in the dev. env.). To build wrapper library, CMake is required.

    build.sh inside python directory will build the wrapper and run test script.

    CLA Signed 
    opened by freakeinstein 13
  • How does DocSpace document recommendation works? Github documentation is very confusing.

    How does DocSpace document recommendation works? Github documentation is very confusing.

    This might be a dump question. But I wish the Starspace documentation was as simple as Fasttext's documentation.

    I'm trying to train a document recommender with starspace. For that, I have followed this section. Below is my problem in hand: I wanted to cheat the recommendation system, to recommend some utterance from a conversation context. For that, I have downloaded Cornell Movie-Dialogs Corpus and formatted the dialogues to match the context as detailed below.

    Each line in the training input file is a context of a conversation (here 'context' mocks 'user' as in the documentation page). A context is represented by the bag dialogues between two people in that context (here 'bag of dialogues' mocks 'bag of documents' as in the documentation page). And each dialogue is a bag of words.

    I have used this command for training - starspace train -trainFile input.txt -model docspace -trainMode 1 -fileFormat labelDoc.

    So, at training time, at each step one random 'dialogue' is selected as the label and the rest of the bag of 'dialogues' are selected as input. I hope it will learn the conversation context here.

    The problem I'm facing are happening here is the prediction process. Truth is, I'm not sure how to make predictions, so I have tried what I understood from the documentation. Below is what I tried to make predictions:

    I have followed this section. Executed query_predict <model> k [basedocs].

    • Firstly, I'm not sure what the content or format of basedocs should be. So, I have written each dialogue one by one without considering any context (splitting dialogues in a single conversation into separate lines).
    • With above point done, I have given some input when prompted. What the prediction method did there is, finding the most similar sentence. But that's not a recommendation system should be doing. Right? It should be giving the left out item. And I'm confused how could I give multiple dialogues within a context (or while rephrasing - how to give a user for getting recommendation? Giving a user means giving a bag of documents he likes, right?). I have tried it as well by separating each dialogues within a context by delimiter. I'm not sure if that's correct.

    Anyway, I guess the cheat is not working because of these,

    • My assumption about the context is wrong (yes its wrong by 50% because there's no sequential learning is done, but I expects any of the opt out dialogues instead of giving the exact/best matching sentence)
    • I'm formatting the 'basedocs' wrongly
    • query_predict is not meant to be a tool to use in this situation.

    Can someone help me to reduce my confusions and pin point the actual reason behind this behavior?

    Thanks for your time.

    opened by freakeinstein 11
  • Remove the Boost Lib Dependency

    Remove the Boost Lib Dependency

    Would it be possibile to remove the boost dependency? Is the matrix.hpp the only numeric library needed? If so, could be embedded.

    By the way, I have upgraded boost on macos:

    [[email protected]:mbploreto Starspace]$ brew upgrade boost
    Updating Homebrew...
    ==> Upgrading 1 outdated package, with result:
    boost 1.65.1
    ==> Upgrading boost 
    ==> Downloading https://homebrew.bintray.com/bottles/boost-1.65.1.sierra.bottle.tar.gz
    ######################################################################## 100,0%
    ==> Pouring boost-1.65.1.sierra.bottle.tar.gz
    🍺  /usr/local/Cellar/boost/1.65.1: 12,679 files, 401.1MB
    

    and after setup BOOST_DIR = /usr/local/Cellar/boost/1.65.1/ in the makefile I get:

    In file included from src/proj.cpp:11:
    In file included from src/proj.h:14:
    src/matrix.h:26:10: fatal error: 'boost/numeric/ublas/matrix.hpp' file not found
    #include <boost/numeric/ublas/matrix.hpp>
             ^
    1 error generated.
    make: *** [proj.o] Error 
    
    opened by loretoparisi 10
  • Starspace crashes while testing model

    Starspace crashes while testing model

    While running ./examples/classification_ag_news.sh, ./examples/wikipedia_article_search.sh, I find Starspace is crashing while testing.

    My system details -

    [email protected]:/mnt/hdd/sakets/facebook/Starspace$ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS" [email protected]:/mnt/hdd/sakets/facebook/Starspace$ uname -a Linux mobile-graphics-ml 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

    Attaching the log file with shows the error at -

    starspace: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 *(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. ./examples/classification_ag_news.sh: line 75: 29692 Aborted (core dumped) ./starspace test -model "${MODELDIR}"/ag_news -testFile "${DATADIR}"/ag_news.test -ngrams 1 -dim 10 -label "label" -thread 10 -similarity "dot" -trainMode 0 -verbose true

    error.zip

    opened by disdi 10
  • Multilabel training Failed

    Multilabel training Failed

    Hello,

    I have dataset with 90k sentences with 7 tags and I am trying to use multi label classify them. However, it doesn't work.

    The file format is:

    Some sample text #tag1 #tag2
    Another sample text #tag5 #tag3
    ..
    

    The command for training is: ./starspace train -trainFile starspace_input.txt -model tagspace -label '#' -trainMode 0 output of training:

    Epoch: 100.0%  lr: 0.008000  loss: 0.003489  eta: 0h1m  tot: 0h0m20s  (20.0%)
     ---+++                Epoch    0 Train error : 0.00351631 +++--- ☃
    Training epoch 1: 0.008 0.002
    Epoch: 100.0%  lr: 0.006000  loss: 0.002564  eta: <1min   tot: 0h0m40s  (40.0%)
     ---+++                Epoch    1 Train error : 0.00240930 +++--- ☃
    Training epoch 2: 0.006 0.002
    Epoch: 100.0%  lr: 0.004000  loss: 0.002277  eta: <1min   tot: 0h0m59s  (60.0%)
     ---+++                Epoch    2 Train error : 0.00222635 +++--- ☃
    Training epoch 3: 0.004 0.002
    Epoch: 100.0%  lr: 0.002021  loss: 0.002060  eta: <1min   tot: 0h1m20s  (80.0%)
     ---+++                Epoch    3 Train error : 0.00206080 +++--- ☃
    Training epoch 4: 0.002 0.002
    Epoch: 100.0%  lr: -0.000000  loss: 0.001748  eta: <1min   tot: 0h1m40s  (100.0%)
     ---+++                Epoch    4 Train error : 0.00182090 +++--- ☃
    

    The command for testing: ./starspace test -testFile starspace_input.txt -model tagspace -predictionFile tags_output.txt The out file is:

    Example 0:
    LHS:
    Some sample text
    RHS:
    #tag1
    Predictions:
    (++) [-0.405902]        #tag1
    (--) [-0.424607]        #tag2
    (--) [-0.424702]        #tag3
    (--) [-0.426362]        #tag4
    (--) [-0.431557]        #tag5
    (--) [-0.435428]        #tag6
    (--) [-0.437012]        #tag7 
    

    I have two issue the first one: 1- Why the model only mark one tag and not all tags for each sentences ? 2- Why the prediction is totally wrong despite the fact that the program tells me training error is "0.00182090" ? I have manually checked them and the propability is extremely wrong.

    opened by agemagician 9
  • how to embed document space

    how to embed document space

    Could you please provide an advice how to embed document space like Doc2vec. I want to embed document space and make sure which document is most similar to one document. I trained my textdata formatted by fasttext. So can I use trained model like a Doc2vec model? ( xxxxxx.mostsimilar(["I like dog"]))

    opened by KazuakiAihara 9
  • How are document (vectors) represented in document classification

    How are document (vectors) represented in document classification

    In training mode 0, I understand that starspace creates word ngrams from all the words in the training file for generating vectors and stores these vectors in the .tsv file. And then from these words, it creates a document embedding vector for the respective document; compares it with the label embedding and minimises the similarity objective.

    How are the document vectors generated from the word vectors here?

    I ran a small experiment using the python wrapper:

    arg = sw.args()
    arg.trainFile = './cooking.txt'
    arg.testFile = './cooking.se.txt'
    arg.trainMode = 0
    arg.ngrams = 1
    arg.verbose=2
    arg.normalizeText = True
    arg.p = 0
    
    sp = sw.starSpace(arg)
    sp.init()
    sp.train()
    
    sp.saveModel('cooking-model')
    sp.saveModelTsv('cooking-model.tsv')
    
    sp.initFromSavedModel('cooking-model')
    sp.initFromTsv('cooking-model.tsv')
    
    sp.evaluate()
    v = np.array(sp.getDocVector('this is', ' '))
    w = np.array(sp.getDocVector('is this', ' '))
    np.allclose(v, w) --> True
    
    t = np.array(sp.getDocVector('this', ' '))
    s = np.array(sp.getDocVector('is', ' '))
    np.allclose((t+s), v) --> False
    

    I know that the vectors are created after undergoing some operation for creating document vectors but I am not sure which one. I tried min, max, avg but they're not resulting in the same document vector v .

    Any help would be highly appreciated!

    opened by tharangni 8
  • my Facebook account was disabled for enable my disable account profile

    my Facebook account was disabled for enable my disable account profile

    Hii facebook team. This is urgent i need your help .my facebook account was disabled when i tried to fill form. It show too much time has passed we cnt review... I am real. User i dnt have fake name or something else i am not able to contact facebook through this feed back form i am requesting you plz restore my facebook account my disable facebook email. Is below plz review..

    opened by Bilal1422 0
  • Upgrade pybind

    Upgrade pybind

    Conan's minimal pybind version seems to be this one in the PR and also due to some internal changes it can't be found if we specify the stable channel (they changed from bintray).

    CLA Signed 
    opened by tgalery 2
  • Must provide base labels when label is featured. on query_nn on trainDoc=1

    Must provide base labels when label is featured. on query_nn on trainDoc=1

    As what @ledw said in #153, I used wikipedia_sentence_matching.sh and change the trainDoc=3 to trainDoc=1 and successfully trained the model

    But I cannot run the query_nn, I use command

     ./query_nn /tmp/starspace/models/wikipedia_sentence_match `
    

    and got

      Start to load a trained starspace model.
      STARSPACE-2018-2
      
      Model loaded.
      Must provide base labels when label is featured.
    

    I also tried

        ./query_nn /tmp/starspace/models/wikipedia_sentence_match -basedoc /tmp/starspace/data/wikipedia_test_basedocs_tm3.txt  -fileFormat labelDoc
    

    but no success.

    What is the correct command for query_nn?

    Thanks

    opened by luthfianto 0
  • loadBaseDocs() is called twice in query_predict utility

    loadBaseDocs() is called twice in query_predict utility

    If model is trained with trainMode=0 and fileFormat="labelDoc", and the query_predict utility is started with savedModel (instead of model.tsv), loadBaseDocs() will be called twice, at L33 and L41 respectively. In such a situation, duplicated results will be returned by query_predict, due to labelDoc file is loaded twice.

    $ ./bin/query_predict model/ss_model 6 data/basedoc.data
    

    NOTE: "model/ss_model" is the savedModel name and "data/basedoc.data" is the basedoc file name.

    opened by slvher 0
Owner
Facebook Research
Facebook Research
A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Light Gradient Boosting Machine LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed a

Microsoft 14k Jul 31, 2022
Fast, differentiable sorting and ranking in PyTorch

Torchsort Fast, differentiable sorting and ranking in PyTorch. Pure PyTorch implementation of Fast Differentiable Sorting and Ranking (Blondel et al.)

Teddy Koker 612 Jul 30, 2022
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

This is the Vowpal Wabbit fast online learning code. Why Vowpal Wabbit? Vowpal Wabbit is a machine learning system which pushes the frontier of machin

Vowpal Wabbit 8k Aug 4, 2022
Training and Evaluating Facial Classification Keras Models using the Tensorflow C API Implemented into a C++ Codebase.

CFace Training and Evaluating Facial Classification Keras Models using the Tensorflow C API Implemented into a C++ Codebase. Dependancies Tensorflow 2

null 8 Nov 23, 2021
MobileNet Image Classification with ESP32-CAM and Edge Impulse (TinyML)

MobileNet Image Classification on ESP32-CAM and Edge Impulse (TinyML) This example is for running a MobileNet neural network model on a 10-dollar Ai-T

Alan Wang 13 Jul 27, 2022
fast face classification

Fast Face Classification (F²C)—— An Efficient Training Approach for Very Large Scale Face Recognition Training on ultra-large-scale datasets is time-c

null 63 Aug 4, 2022
Using image classification with ConvMixer

Japanese Handwriting Classification with Fragment Shaders NOTE: This was built and tested with Unity 2019.4.31f1 using built-in render pipeline, there

null 8 Feb 21, 2022
DeepI2P - Image-to-Point Cloud Registration via Deep Classification. CVPR 2021

#DeepI2P: Image-to-Point Cloud Registration via Deep Classification Summary Video PyTorch implementation for our CVPR 2021 paper DeepI2P. DeepI2P solv

Li Jiaxin 124 Jul 29, 2022
A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms.

iNeural A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms. What is a Neural Network? Work on

Fatih Küçükkarakurt 5 Apr 5, 2022
Deep Scalable Sparse Tensor Network Engine (DSSTNE) is an Amazon developed library for building Deep Learning (DL) machine learning (ML) models

Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine DSSTNE (pronounced "Destiny") is an open source software library for training and deploying

Amazon Archives 4.4k Jul 30, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Aug 1, 2022
Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

JoliBrain 2.4k Aug 2, 2022
In-situ data analyses and machine learning with OpenFOAM and Python

PythonFOAM: In-situ data analyses with OpenFOAM and Python Using Python modules for in-situ data analytics with OpenFOAM 8. NOTE that this is NOT PyFO

Argonne Leadership Computing Facility - ALCF 105 Aug 5, 2022
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

The Apache Software Foundation 20k Aug 5, 2022
A lightweight C++ machine learning library for embedded electronics and robotics.

Fido Fido is an lightweight, highly modular C++ machine learning library for embedded electronics and robotics. Fido is especially suited for robotic

The Fido Project 412 Jun 25, 2022
learn300 is challenge taken to keep hunger of learning things alive and to push myself every day!

learn300 This repository contains all information shared publicly during my learn300 challenge. Learn300 is a challenge to keep myself pushing and to

seg_fault 19 Feb 13, 2022
Deep Learning in C Programming Language. Provides an easy way to create and train ANNs.

cDNN is a Deep Learning Library written in C Programming Language. cDNN provides functions that can be used to create Artificial Neural Networks (ANN)

Vishal R 11 Apr 18, 2022
Triton - a language and compiler for writing highly efficient custom Deep-Learning primitives.

Triton - a language and compiler for writing highly efficient custom Deep-Learning primitives.

OpenAI 3.8k Aug 1, 2022