Here I try to fill this gap by taking some data and drawing some plots.

Data source

First of all, we need 2 things: data to analyse and ideas of what data we need. Ideally, the latter should be the first, but in this case the order is more spiral than straight: some ideas may lead to new data sources, some data sources can inspire/provoke new ideas. The basic requirements to data were information about genre, year, author.

I started from structured data sources, namely Wikidata, because I heard that, after the shutdown of Freebase, it has been merged into Wikidata and Freebase contained a lot of data about books. However, Wikidata turned out to contain only 28463 novels, 3329 science fiction and 1722 fantasy items, check it by this query to wikidata.

DBpedia showed even worse results: 1654 Science_fiction and 1341 Fantasy_literature, see corresponding query.

Therefore I returned to unpleasant choice of sites to crawl and scrape data from. Goodreads, probably the most full database about books, like IMDB for movies, prohibits crawling in their robots.txt and TOS. fantasticfiction.com looked promising, but its coverage is also not so good. After all, I decided to go with Fantlab - from my point of view, it has the fullest coverage of speculative fiction, especially, but not only Russian authors. As it appears, there is even an API, but I preferred to simply crawl and scrape the data.

Crawling and scraping

Since I had no experience in scraping, I’ve chosen the most popular framework - scrapy. Non-obvious things I learned from using it:

1. In order to serialize encoded Russian text, scrapy should be at least 1.20 and FEED_EXPORT_ENCODING = 'utf-8' should be added to settings (thanks to this recent SO answer)
2. XPath selectors can be copied from browser (in Chrome: Right click - Inspect - Copy - Copy XPath). I didn’t find this advice anywhere - maybe, because it is actually obvious.
3. You can use ItemLoaders to nicely preprocess/postprocess extracted info and you can grab items from multiple pages/requrests, but not both at the same time - at least, I didn’t find the way to do it nicely.

(Since I wanted to get info about genres and this info is shown only when enough users voted for the same genre, I crawled only those works having at least one rating.)

Data preparation

We start from reading of serialized csv into pandas dataframe; we filter away works written before 1900 year. and we convert genres into immutable set of strings.

import pandas as pd
import numpy as np

def genres_split(s):
    return frozenset(s.split(','))

def year_converter(s):
    try:
        years = int(s)
        if 1900 < years < 2017:
            return years
    except ValueError:
        return np.nan
    
filename='/home/nik/workspace/fiction-miner/fantlab_annot.csv'
df = pd.read_csv(filename, converters={"genres": genres_split, "year": year_converter}, index_col='id')
df.head(3)

We have genres as immutable set of strings – it violates 1st normal form, so we perform usual one-hot encoding and, in the same time, we keep only part of genres and translate them from Russian to English.

main_genres_dict = {
    'Фантастика': 'Science fiction',
    'Фэнтези': 'Fantasy',
    'Магический реализм': 'Magic realism',
    'Мистика': 'Mystic fiction',
    'Хоррор/Ужасы': 'Horror fiction',
    'Сказка/Притча': 'Fairytale / parable',
    'Детектив': 'Detective fiction',
    'Боевик': 'Action fiction',
    'Триллер': 'Thriller',
    'Любовный роман': 'Romance novel',
    'Историческая проза': 'Historical fiction',
    'Сюрреализм': 'Surrealism',
    'Постмодернизм': 'Postmodernism',
    'Реализм': 'Realism',
    'Психоделика': 'Psyhodelic',
    }
scifi_subgenres = {
    '«Твёрдая» НФ': 'Hard SF',
    'Гуманитарная («мягкая») НФ': 'Soft SF',
    'Космоопера': 'Spaceopera',
    'Киберпанк': 'Cyberpunk',
    'Планетарная фантастика': 'Planet SF',
    'Таймпанк': 'Timepunk',
    'Хроноопера': 'Chronoopera',
    'Постапокалиптика': 'Postap',
    'Роман-катастрофа': 'Disaster novel',
    'Утопия': 'Utopia',
    'Антиутопия': 'Antiutopia',
    }
fantasy_subgenres = {
    'Эпическое фэнтези': 'Epic Fantasy',
    'Героическое фэнтези': 'Heroic Fantasy',
    'Городское фэнтези': 'Urban Fantasy',
    'Dark Fantasy': 'Dark Fantasy',
    'Технофэнтези': 'Technofantasy',
    'Science Fantasy': 'Science Fantasy',
    '«Дотолкиновское» фэнтези': 'Pretolkien Fantasy',
    '«Классическое» фэнтези': 'Classic Fantasy',
    'Артуриана': 'Arhturiana',
    'Анималистическое': 'Animalistic',
    'Мифологическое': 'Mythological Fantasy',
    }
# genres_dict = {**main_genres_dict, **scifi_subgenres, fantasy_subgenres}
# genres_set = frozenset.union(*df.genres)
def encode_genres(df, genres_dict):
    genres_set = genres_dict.keys()
    for genre in genres_set:
        genre_lab = genres_dict[genre]
        #taken from http://datascience.stackexchange.com/a/11799
        df[genre_lab] = df.apply(lambda x: int(genre in x['genres']), axis='columns')

        df[genre_lab+'_rat'] = df.apply(lambda x: x['rating'] if x[genre_lab]==1 else np.nan, axis=1)
        df[genre_lab+'_rat_count'] = df.apply(lambda x: x['rating_count'] if x[genre_lab]==1 else np.nan, axis=1)
        print(genre_lab + ": \t\t" + str(len(df[df[genre_lab] == 1])))
print('  Main genres')
encode_genres(df, main_genres_dict)
print('  SciFi subgenres')
encode_genres(df, scifi_subgenres)
print('  Fantasy subgenres')
encode_genres(df, fantasy_subgenres)
df.head(3)

  Main genres
Mystic fiction: 		1844
Fairytale / parable: 		1044
Fantasy: 		4635
Psyhodelic: 		263
Romance novel: 		276
Realism: 		4473
Action fiction: 		689
Thriller: 		283
Science fiction: 		12765
Postmodernism: 		209
Detective fiction: 		1273
Historical fiction: 		539
Surrealism: 		415
Magic realism: 		761
Horror fiction: 		1413
  SciFi subgenres
Hard SF: 		2815
Cyberpunk: 		269
Planet SF: 		651
Utopia: 		109
Spaceopera: 		1043
Soft SF: 		8957
Chronoopera: 		1021
Antiutopia: 		498
Timepunk: 		108
Postap: 		756
Disaster novel: 		218
  Fantasy subgenres
Science Fantasy: 		369
Pretolkien Fantasy: 		70
Epic Fantasy: 		443
Technofantasy: 		247
Dark Fantasy: 		359
Arhturiana: 		39
Heroic Fantasy: 		3364
Mythological Fantasy: 		213
Animalistic: 		57
Classic Fantasy: 		129
Urban Fantasy: 		649

In addition to the explicit info like year, genre, author, rating, etc., I determine 2 more types of data based on authors’ names:

1. Authors’ nationality: Russian or Foreign
2. Authors’ gender: Male or Female

Authors’ nationality

In order to estimate the author’s nationality, I simply check if the name contains at least one English character: I base my solution on the observation that names for foreign authors contains their original names in parentheses. Of course, there may be errors of both type, for example pseudonyms of Russian authors or missed original name of foreign one, but I belive that in general the error rate would be high enough.

import re
def containsEn(s):
    return bool(re.search('[a-zA-Z]', s))
print(containsEn('Майкл (Michael)'))
print(containsEn('Петр (псевдоним Ивана)'))

True
False

df_en = df[df.apply(lambda x: containsEn(x['author_name']), axis=1)]
df_rus = df[df.apply(lambda x: not containsEn(x['author_name']), axis=1)]
print(df.shape)
print(df_en.shape)

(75418, 120)
(36021, 120)

Authors’ gender

In order to estimate the author’s gender, I split full name into name parts and find intersection with prepared sets of male and female names. These lists were constructed from gazeteers of Gate 8 (both English and Russian lists) and some random site for oneiromancy, name choice and so on.

def read_in_set(filename, prefix='/home/nik/workspace/fiction-miner/names/'):
    with open(prefix + filename) as f:
        content = f.readlines()
    res = set([x.strip() for x in content if len(x.strip()) > 1])
    return res
# taken from GATE 8 gazeteers and http://www.sonnik-online.net/imena/_mujskie.html
names_female_ru = read_in_set('first_names_female.lst')
names_male_ru = read_in_set('first_names_male.lst')
names_female_en = read_in_set('person_female_ext.lst')
names_male_en = read_in_set('person_male_ext.lst')
names_female = set.union(names_female_ru, names_female_en)
names_male = set.union(names_male_ru, names_male_en)
# since my list contains forms of male names like Alexandra, lets filter them out
names_male = names_male.difference(names_female)

import string
def check_gender(s, names_set):
    translator = str.maketrans('', '', string.punctuation)
    words_in_s = set(s.translate(translator).split())
    return bool(words_in_s.intersection(names_set))

def isMale(s):
    return check_gender(s, names_male)

def isFemale(s):
    return check_gender(s, names_female)

print(isMale('Вера Петрова'))
print(isMale('Владимиp Ильин')) #incorrect layout
print(isMale('Владимир Ильин'))
print(isMale('Александра Олайва (Alexandra Oliva)')) #because of Oliva
print(isFemale('Алексей В. Андреев'))
print(isFemale('Вера Петрова'))
print(isMale('Алан Глинн (Alan Glynn)'))
print(isMale('Дэвид С. Гарнетт (David S. Garnett)'))
print(isFemale('John Doe'))

False
False
True
True
False
True
True
True
False

df_female = df[df.apply(lambda x: isFemale(x['author_name']), axis=1)]
print('females: ' + str(df_female.shape))
df_male = df[df.apply(lambda x: isMale(x['author_name']), axis=1)]
print('males: ' + str(df_male.shape))

females: 12367
males: 57734

df_unknown = df[df.apply(lambda x: not isMale(x['author_name']) and not isFemale(x['author_name']), axis=1)]
print('undefined: ' + str(df_unknown.shape))
print(np.unique(df_unknown['author_name'])[:5])
df_both = df[df.apply(lambda x: isMale(x['author_name']) and isFemale(x['author_name']), axis=1)]
print('both: ' + str(df_both.shape))
print(np.unique(df_both['author_name'])[:10])

undefined: (6884, 120)
['Zотов' 'А. Ли Мартинес (A. Lee Martinez)'
 'А. Н. Л. Манби (A. N. L. Munby)' 'Аврам Дэвидсон (Avram Davidson)'
 'Агоп Мелконян (Агоп Мъгърдич Мелконян)']
both: (1567, 120)
['Алан Глинн (Alan Glynn)' 'Александр и Людмила Белаш'
 'Александра Олайва (Alexandra Oliva)'
 'Александра Харви (Alyxandra Harvey)' 'Брижит Обер (Brigitte Aubert)'
 'Вероника Рот (Veronica Roth)' 'Габриэлла Пирс (Gabriella Pierce)'
 'Гай Гэвриел Кей (Guy Gavriel Kay)'
 'Гейл Карсон Ливайн (Gail Carson Levine)' 'Грэм Джойс (Graham Joyce)']

As we can see, most ‘hermaphrodite’ authors are those having surname equal to the name of the opposite gender, e.g. Alyxandra Harvey.

Genres development

Lets analyse the genres development over years. Area charts seem to be the most appropriate here, because they connect volume with square or areas.

Note that we use ‘wiggle’ modification, that is why it centers arount zero instead of building peaked mountains, read docs of matplotlib for details.

import matplotlib.pyplot as plt
%matplotlib notebook
# %matplotlib inline
from bokeh.palettes import colorblind, d3, brewer

def draw_plot(df2plot, genres_to_show, title, roll_window=2):
    fig, ax = plt.subplots(figsize=(11, 6))
    stackplot(df2plot, genres_to_show, ax, title, roll_window)

def stackplot(df2plot, genres_to_show, ax, title, roll_window):
    ax.set_title(title)
    palette = d3['Category10'][len(genres_to_show)]
    x = np.unique(df2plot['year'].dropna().values)
    y_all = []
    for i, genre in enumerate(genres_to_show):
        y = df2plot.groupby('year')[genre].sum()
        y_avg = pd.Series.rolling(y, center=False,window=roll_window).mean()
        y_all.append(y_avg)
    polys = ax.stackplot(x, y_all, labels=genres_to_show, linewidth=0, colors=palette, baseline='wiggle')
    ax.set_xticks(np.arange(1900,2020,10))
    ax.grid()
    ax.set_axis_bgcolor((0.8,0.8,0.8))
    ax.legend(loc='lower left', prop={'size':13})
    return ax

top_genres = [  #i.e those having at least 1000 items
    'Science fiction',
    'Fantasy',
    'Mystic fiction',
    'Horror fiction',
    'Fairytale / parable', 
    'Detective fiction', 
    'Realism', 
]
draw_plot(df, top_genres, 'Genres development over years', 2)

Note also the averaging (roll_window param), which is designed to smooth natural sparseness of the data; without it (i.e. with roll_window=1) random peaks hide trends, which is especially actual for further plots containg data subsets.

sf_subgenres = [
    'Hard SF',
    'Soft SF',
    'Spaceopera',
    'Cyberpunk',
    'Planet SF',
    'Timepunk',
    'Chronoopera',
    'Postap',
#     'Disaster novel',
    'Utopia',
    'Antiutopia',
]
draw_plot(df, sf_subgenres, 'Science fiction subgenres development over years')

fantasy_subgenres = [
    'Epic Fantasy',
    'Heroic Fantasy',
    'Urban Fantasy',
    'Dark Fantasy',
    'Technofantasy',
    'Science Fantasy',
#     'Pretolkien Fantasy',
    'Classic Fantasy',
#     'Arhturiana',
#     'Animalistic',
    'Mythological Fantasy',
]
draw_plot(df, fantasy_subgenres, 'Fantasy subgenres development over years')

#draw_plot(df_rus, top_genres, 'Genres development over years (Russian authors)', 2)

Compare Russian and Foreign authors

def compare2plots_row(df2plot1, df2plot2, genres_to_show, title1, title2, roll_window=2):
    fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(11, 10))
    stackplot(df2plot1, genres_to_show, axes[0], title1, roll_window)
    stackplot(df2plot2, genres_to_show, axes[1], title2, roll_window)

compare2plots_row(df_rus, df_en, top_genres, 'Genres (Russian authors)', 'Genres (Foreign authors)')

We can see the lag for Foreign authors: at the first plot volume becomes to shrink in 2015, while at the second one - already in 2010. If we further analyze subgenres, we find that most volume of Russian authors plot concentrates in last 15 years, while Foreign authors plot has most mass in the middle of XX century. It can be explained by the hypothesis that nearer works (both in time and nationality) cause more attention of the readers and thus have more representation on these plots.

compare2plots_row(df_rus, df_en, sf_subgenres, 'SciFi subgenres (Russian authors)', 'SciFi subgenres (Foreign authors)') 
compare2plots_row(df_rus, df_en, fantasy_subgenres, 'Fantasy subgenres (Russian authors)', 'Fantasy subgenres (Foreign authors)') 

It is interesting that Tehcnofantasy and Science Fantasy are much more popular in case of Russian authors, unlike Epic Fantasy and Dark Fantasy.

Compare Female and Male authors

compare2plots_row(df_female, df_male, top_genres, 'Genres (Female authors)', 'Genres (Male authors)')

compare2plots_row(df_female, df_male, sf_subgenres, 'SciFi subgenres (Female authors)', 'SciFi subgenres (Male authors)') 
compare2plots_row(df_female, df_male, fantasy_subgenres, 'Fantasy subgenres (Female authors)', 'Fantasy subgenres (Male authors)') 

Here we can note that women write less Hard SciFi and Dark Fantasy, as well as books in sum, than men.

Rating plots

def lineplot(df2plot, genres_to_show, title, roll_window=3, col_suffix='_rat'):
    fig, ax = plt.subplots(figsize=(12, 6))
    ax.set_title(title)
    x = np.unique(df2plot['year'].dropna().values)
    for genre in genres_to_show:
        y = df2plot.groupby('year')[genre+col_suffix].mean()
        y_avg = pd.Series.rolling(y, center=False,window=roll_window).mean()
        ax.plot(x, y_avg, label=genre)
    # plt.plot(x, y, label='overall')
    ax.set_xticks(np.arange(1900,2020,10))
    ax.grid()
    ax.set_axis_bgcolor((0.8,0.8,0.8))
    ax.legend(loc='lower left',prop={'size':13})

lineplot(df, top_genres, 'Rating by genres', 5)

Plots for subgenres are too sparse, even with 5-window smoothing:

lineplot(df, sf_subgenres, 'Rating by SciFi subgenres', 5)
lineplot(df, fantasy_subgenres, 'Rating by Fantasy subgenres', 5)

Rating counts depend on famous works like ‘Lord of the Rings’ too much:

lineplot(df, top_genres, 'Popularity (rating counts) by genres (without smoothing)', 1, '_rat_count')

Finally, lets compare authors by rating:

def add_line(df2plot, ax, label,roll_window):
    x = np.unique(df2plot['year'].dropna().values)
    y = df2plot.groupby('year')['rating'].mean()
    y_avg = pd.Series.rolling(y, center=False,window=roll_window).mean()
    ax.plot(x, y_avg, label=label)

def lineplot_compare(df2plot1, df2plot2, label1, label2, title, roll_window):
    fig, ax = plt.subplots()
    ax.set_title(title)
    add_line(df2plot1, ax, label1,roll_window)
    add_line(df2plot2, ax, label2,roll_window)
    ax.set_xticks(np.arange(1900,2020,10))
    ax.grid()
    ax.set_axis_bgcolor((0.8,0.8,0.8))
    ax.legend(loc='upper left',prop={'size':13})

lineplot_compare(df_rus, df_en, 'Russian', 'Foreign', 'Compare rating of Russian and Foreign authors', 3)

lineplot_compare(df_female, df_male, 'Female', 'Male', 'Compare rating of Female and Male authors', 3)

P.S. In the nearest future I’m going to play with annotations; for example, use them to predict and compare genres.

P.P.S. All code will be available on github soon.

Assume that you have built you project (written in Scala, but it will be relevant only for one point, see below) with gradle and now you want to upload jar, so that anyone can use it by maven/gradle/sbt/whatever – even ant, altough I used it with manually collected jars only.

It is well known that there are 2 main repos: mavenCentral and jcenter. They remind me Python 2 and Python 3, respectively: everyone says that Python 3 is better, but if you really want your lib to be used by everyone, use Python 2 – or, ideally, write “code that runs under both Python2 and 3” (so that you have to take worst from both worlds).

Fortunately, bintray team provides easy way to achieve both repos without much efforts.

Unfortunately, their original documentation describes evident cases, but omits what is really incomprehensible, e.g. they link to the start page of maven (however, in gradle plugin page they provide much more useful link). So I try to clarify it a little.

Brief how-to

1. Go to OSS, register, create issue for your future artifact
2. Go to bintray, register there and create repo (this is repo for artifacts, not your repo for code).
3. Add code to build.gradle - see comments below or just copy-paste examples from pu4spark or atr4s. Also see how to hide credentials at stackoverflow answer.
4. Run gradle bintrayUpload.
5. Go to bintray again, press Sync with JCenter. Wait for some time (usually, hours).
6. Sync with mavenCentral.

*Also take a look at this nice guide with a lot of screenshots; it is designed for Android, but the first part remains valid for any jars.

Brief comments to build.gradle

Note that bintray’s signing can be used for uploading to mavenCentral (if you are more lazy than paranoid).

MavenCentral doesn’t pass files without javadocs (see the almost-5-year-old issue) and they don’t care if you are using Scala. There are projects for generating javadocs, but it seems to be much simpler to just replace javadocs by scaladocs (see example).

Inclusion of mavenCentralSync section, which was in original guide, seems to be useless, because you will anyway need to manually sync package with jcenter first.

Also be careful with pom.xml generation: I didn’t research this bug, but if you move any of 3 append lines to pomConfig method, i.e. replace them by lines like the following:

description 'A library for Positive-Unlabeled Learning for Apache Spark MLlib (ml package)'

then gradle fails to add description tag to pom.xml for no reason! Even if we place something before description in pomConfig, it still won’t shown.

There are many guides and even sites to help in choosing a license for your code without understanding of these licenses. Obviuosly, these guide have to abstract from details in order to be comprehensible and one or two abstractions can’t be well-suited for everyone.

Here I’d like to provide yet another abstraction by focusing on practical, even pragmatic, concerns; in other words – if you want to extract maximum profit from the code without regard to particular types of freedom.

First of all, if you just started and have nothing working yet, then simply postpone this decision. Most probably, you won’t need to decide. Feel free to store the code at bitbucket (paranoids like me like private repos) or github – btw, not everybody knows that no explicit info in github repository means “exclusive copyright”.

If you have managed to create an end-user application like text editor or game and believe (and want) it to be sellable, then good luck with startup, keep the code proprietary.

If enough time passed you don’t belive that the app can be sold – e.g. because it is yet another text editor – then go with GPL. The reason is simple: users would have source code, which gives some pluses for reputation and possible bug fixes, but you are protected from Embrace, extend and extinguish.

From all kinds of GPL, prefer AGPL, because it prevents from Application Service Provider loophole, or SaaS loophole. which is especially relevant, if you are paranoid like me if your application can be used as a service.

If you are still hesitate if the app can be sold and think about dual licensing, then I have to disappoint you: it is extremely unlikely that dual-licensing will work for you. Read this good meta-analysis; in short: it can work only if your app is supposed to be used as a part of something and, much harder condition, if it can take a huge market share. In addition to working dual-licensed examples from the paper above, I’d add Stanford NLP, which is still the best Java NLP library.

If you created a library, then basically you have a choice from 2 alternatives: MIT/BSD vs Apache 2.0. Motivation is simple again: you want to have as many users as possible, because their number is directly proportional to gained reputation (ok, cover it by goodwill), and only permissive licenses can provide that.

LGPL and joke licenses like WTFPL (and other not OSI-approved licenses) scare away users, especially influential ones, almost as much as copyleft. Public domain seems to have similar effects.

Returning to the final choice, Apache vs MIT/BSD, I should warn that some guys report that paranoid lawyers in their companies prohibit usage of Apache libs (see this explanation).

Very roughly: if you belive that (a) your lib can be useful for big corporations, (b) these corporation would fear to be patent-trolled, and (c) you don’t fear to be patent-trolled (kind of opposite thoughts, from my point of view), then don’t use Apache and go with MIT/BSD. From these 2 choose the one with letters you like more; or the most popular one, which is MIT (stats is based on github repos with explicit license; however, 80% contain no license and I still don’t understand what ‘no license’ row of the table means).

One more argument for Apache 2.0 relates to easier combination of your code with existing Apache libs: Apache 2.0 is compatible with MIT/BSD, i.e. you can declare all your code to be Apache, even if you are using MIT libs, but not otherwise. Sure, you can license your files as MIT and keep Apache license for other files, since these licenses are permissive after all, but it would require a bit more efforts: prepend license to each file, choose what to write in github and maven, etc.

And if you do not use Apache libs in your code, then I’d suppose that it is either from the case described in First of all-paragraph, or it contains a lot of self-invented wheels. (BTW, why the invention is so simple in English idiom? For instance, Russians tend to reinvent the bicycle in similar cases.)

Disclaimer: of course, I am not a lawyer. (BTW, why the corresponding acronym is so popular, while it is so terrible after all?)

PU learning vs semi-supervised learning and outlier detection

Similarity stems from the fact that in all cases we try to build binary classifier from insufficient data: in outlier detection we don’t have enough positive instances, while in SSL setting we simply don’t have enough any labeled data.

Returning to distinctions, in SSL we use unlabeled data only for improving classifier – however, this is rarely used in practice; at least in my experience an addition of unlabeled data didn’t improve results, even for very small labeled data.

In anomaly detection (how handy it is to have perfect synonyms – the simplest way to cope with tautology!), outliers are usually not similar to each other, while non-outliers, i.e. negatives, are similar, e.g. they are assumed to be from the same distribution; in other words, we can’t say that 2 outliers are more similar to each other than an outlier – to a non-outlier.

Contrariwise, in PU we have positives similar to each other and we know (hope) that they are not similar to negatives. (And we can’t simply declare all unlabeled instances as negatives, because these unlabeleds may contain a lot of positives, not less than we have explicitly labeled as positives.)

Good example is a websites classification: if we want to separate restaurant sites from all others, we can easily crawl dozens of positives, but it’ll be hard to crawl representative negatives, since we don’t know a priori enough information about representativeness itself.

Note that it can be applied in other fields, e.g. image classification, in particular – land cover classification.

PU learning vs supervised learning

What is rarely mentioned is that ordinary supervised (binary) learning can sometimes be more correctly modeled by PU. In the example above, one who didn’t hear about PU would somehow sample negatives from all sites and apply good old SVM or random forest xgboost.

And maybe s/he’ll get lucky and – sufficiently good results. This strategy, which is called negative sampling, has definitely a right to exist and even to be included into the name of the most popular way of word2vec training, namely skip-gram model with negative sampling (SGNS).

The only thing I’d like to warn is that you should use such negative sampling consciously, otherwise you risk to have a kind of sample selection bias, i.e. obtain a good quality on cross-validation, but fail in production.

One more example when shift from supervised learning to PU can help is word sense disambiguation – choosing correct meaning for a term (word or collocation) depending on the context:

Assume that you have data labeled with non-perfect guide, e.g. without formulation of boundary cases or needed level of sense-granularity, and by non-perfect annotators, e.g. CS-students.

Let’s consider the example sentence: The army entered the city. Here term army may be disambiguated to top-level Army, or to specific Ground forces, or to known-from-context US army. But other concepts like films or newspapers (see wiki disambiguation page) are obviously wrong.

In this case, any of the 3 correct meanings can be chosen by an annotator, but only one can be, so non-chosen meanings would include both negative and positive instances, so these non-chosen meanings are actually unlabeled instances – hello, PU!

Of course, it is better to fix the source of the problem, i. e. 2 assumptions mentioned above, but it is not always feasible.

PU learning as unsupervised learning

Another useful pipeline that was tested a couple of times (including my PhD work): from all unlabeled examples somehow extract positives, then learn PU on that. Note that such positives should be precise, but heterogeneous/representative at the same time, so prefer simple heuristic.

Thus you’ll have fully unsupervised approach that may learn from data in some sense – of course, if your task lets invention of appropriate heuristic for seed positives extraction.

PU tools

Despite its usefulness in practice, PU learning seems underrated: Among possible implementations all I found is a 10-years old C-program (btw, this site lists many good papers on subject) plus several abandoned python libs, e.g. this and that.

I don’t fully understand the reasons. Maybe, people just apply negative sampling or reinvent simple PU algorithms not naming this as PU.

In last project where I needed PU, I used Scala and Apache Spark MLlib, therefore I implemented a pair of simplest algorithms: Traditional and GradualReduction aka PU-LEA.

In a nutshell, they wrap probabilistic classifiers like Logistic regression or Random forest by the following strategy: consider all unlabeled as negatives; learn probabilistic classifier on that; extract so-called reliable negatives, i.e. those instances classified with too low confidence; learn classifier on positives and those reliable negatives; repeat.

Spy-EM should be implemented as well, but in my case simplest algorithms worked good enough – and notably better than trivial 1-step approach (when you consider unlabeled instances as negatives and get SL classifier right away).

Ranking-based SVM and other sophisticated algorithms seem to be more sensitive to noise in data, so they should be used only for pure PU setting, not the unsupervised pipeline described above.

Anyway, pu4spark is now ready for use; it is also available at jcenter and mavenCentral (uploading it there turned out to be non-trivial and I’m going to describe this soon).

Word2vec becomes especially helpful, when we work with small text data and face sparseness problem in its worst. A popular way to cope with it is to train word2vec model on some huge data like Wikipedia dump and use these good vectors for words of texts we actually want to process.

In this setting, we don’t need to train model in our program or even ourselves; there are collections of pre-trained models ready to download). All we have to do is to load pre-trained model into some map (dictionary, in case of Python) and to compute cosines between vectors.

Ok, we can write such code in an hour! Ok, there should be a library doing exactly that. Let’s spend this hour choosing it!

Let’s assume we write our app in some JVM language like Java or Scala (in Python: simply use genism; in C/C++: find original word2vec saved by some guy in github after the shutdown of code.google).

As usual, implied requirements to the lib: permissive license, little dependencies, readable code.

So, what are our options?

Java libraries

The first one in google for “word2vec java” is deeplearning4j - it is huge, requires a lot of stuff, so we’d prefer to start with something more lightweight; we return to this by the end of the post.

The next one is Word2VecJava

What catches our eyes? They use double[] for vectors storage; despite the fact that binary format stores floats, so there would be no information loss. And all potential loss in accuracy caused by operations under vectors of floats instead of vectors of doubles would be negligible compared to the loss caused by twice lower dimension of the vector (on condition of the same memory consumption).

Thus we have 1 more requirement for a lib: float vectors. Moreover, as float arrays, not Lists, because, we all know, Float weights as double double (sorry for bad wordplay).

The third hit leads to Apache Spark.

Again, too heavy, plus they don’t have sim/cos; it is not that hard to implement, but what was the reason not to create this? What is worse, it still can’t load pre-trained models of gensim or original word2vec, see old issue unresolved yet.

There are also some abandoned non-English repos with strange code in Readme file.

Not so much. Let’s google for Scala.

Scala libraries

read_word_vectors.scala - ok, it is licensed under Apache 2.0, utterly simple one-file with no dependency.

But it is based on Scala’s List, which compiles to Java’s List, see corresponding SO answer. In addition – while it doesn’t already matter – this lib doesn’t contain sim/cos and hasn’t been updated since the very 2013.

word2vec-scala – Apache License, Array of float, one-file with no dependency; sim/cos implemented. The only minor note that it hasn’t been updated since the very 2013.

Luckily, Refefer updated it: split into 2 classes (Reader and Model) and rewrote Model so that it is based on spire and, hopefully, works faster.

I have chosen this library.

Small war story

I checked the model provided with the code – unsurprisingly, everything worked.

I took the model trained by modern genism; tried to load it and it failed with EOFException during model file reading.

Then - hour for choosing library was coming to end - I searched for exception message at stackoverflow, read the top answer diagonally (is there such an idiom in English?) – it advised to simply catch exception. I copy-pasted this voiceless catch – the main mistake – and everything seemed to work fine.

However, I started to get many misses for completely common words. So, I took a look at the source code a little more:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def readVector(reader: VecBinaryReader, vecSize:Int, normalize: Boolean): (String, Array[Float]) = {
     // Read the word
    val word = reader.read[String]

    val vector = new Array[Float](vecSize)
    for((f, i) <- reader.read[Stream[Float]].take(vecSize).zipWithIndex) {
      vector(i) = f
    }

    // Eat up the next delimiter character
    reader.read[Byte]

    // Store the normalized vector representation, keyed by the word
    word -> (if (normalize) normVector(vector) else vector)
}

This line looks susceptible:

// Eat up the next delimiter character
reader.read[Byte]

Let’s examine the excerpt from the model file:

ID/united_states 2dr?`C?>?{???>?`?>???>o>[?!???d?Y?>?,??q? ???"??%?????>?"=??l?=!	??#F=??$??8|??J(????>X(?p7>v???`????>??=??????>)%??u????>Hb?.?
??=?n????1???=??>yX?>?????1>???>???><?{?s???!-?>U????
>L??>~7??
???V?H*>?P?C ?? ????>?bD=*.?/?M?	???>]w????l?:?"?tÑ>?9?Wo???.K>M?'?T??>?>??>w]5?D6??Q??)???U???3>XW;??>e*,?????I???J???Y???>B?C????[
?|W^???>n?????>%Rn>???ID/world_war_ii ???>??mMP=c????>?>???&???????K=?lY?>???K????o?yx?_?l)?b?;?p???>??P??'???:???????=?????Wu???>?K?=???=?u??+??o]=??`>????????d???+Nh?l??m??=??>??k?2?<i?>?(??????J???@???
?'??4?>?WZ???OVG>?#>?{?1r?A?>s3???>?R?>?H??????*?
=?Qc?V??.

>x;[?@5?>?t?U????>1?$?7?(??=@??????c?>??H??>??h??7w>S^y??????-?????????CJ=??{??	????>D???|?>??zK3=?z??d=??ID/association_football ???>???j8Q?ao)?2???>)?|???

If we look at the distance between 2 words ID/united_states and ID/world_war_ii, we find 400 bytes, thus there is no delimiter in new model. Btw, it proves that each element weighs 4 bytes, i.e. it is float.

It is not a bug: obviously, an old model has a delimiter, but then the model format has changed; I don’t know, who introduced that – maybe, gensim.

Of course, we can determine model format automatically, because we know word vectors size (it is written in the model file prefix), but it is much simpler to set it up in parameter and, most probably, no one would use old models.

So this fix was implemented (and merged). Btw, dl4j uses the same workaround.

Compare deeplearning4j with word2vec-scala

Actual task I needed to accomplish with word2vec is to rank term candidates, i.e. to compute semantic similarity between some set of words/collocation and some set of keywords.

I performed this task with 2 libs (dl4j and word2vec-scala), keeping all other conditions the same, including word2vec model. I didn’t try to make precise evaluation, so there is no averaging of multiple launches, to say nothing about JVM warming-up.

I measured 2 times:

1. Time needed to load word2vec model: word2vec-scala requires about 60-70s, while dl4j – 150-180s.

2. Time needed to compute semantic similarities.

Table below contains one row per dataset; 2nd column shows count of similarity computation calls, when word2vec model contains both words; 3rd column shows count of calls for vectors, when word2vec model doesn’t contain one word.

As we can see, dl4j works significantly slower.

This may be caused by several reasons. Probably, there is some flaw in experiment setup, so – again – take these results with a grain of salt. Probably, dl4j works so slow because of missing native libraries on my machine; I tried to install different blas listed in ND4j guide, but it isn’t trivial and I didn’t invest much time into that.

However, note that deeplearning4j keeps evolving, for example their new guide looks much nicer; maybe, Intel MKL, which dl4j recommends to use and which I didn’t check, already speeds up this.

In sum, when you read this, dl4j may work faster, but be ready to spend time playing with setup. Or, if all you need is read model and compute a few dot-products – use word2vec-scala.