If we want to use text in Machine Learning algorithms, we’ll have to convert then to a numerical representation. It should be no surprise that computers are very well at handling numbers.

A machine learning model only understands numbers. The bag of words model is the simplest way to turn text into them: count how often each word appears and use those counts as the numbers.

The name says it all. You throw all the words of a document into a bag, so the order is gone and only the counts are left.

We convert text to a numerical representation called a feature vector. A feature vector can be as simple as a list of numbers.

The bag-of-words model is one of the feature extraction algorithms for text.

Feature extraction from text

The bag of words model ignores grammar and order of words.
We start with two documents (the corpus):

'All my cats in a row',
'When my cat sits down, she looks like a Furby toy!',

A list in then created based on the two strings above:

{'all': 0, 'cat': 1, 'cats': 2, 'down': 3, 'furby': 4, 'in': 5, 'like': 6, 'looks': 7, 'my': 8, 'row': 9, 'she': 10, 'sits': 11, 'toy': 12, 'when': 13 }

The list contains 14 unique words: the vocabulary. That’s why every document is represented by a feature vector of 14 elements. The number of elements is called the dimension.

Then we can express the texts as numeric vectors:

[[1 0 1 0 0 1 0 0 1 1 0 0 0 0]
 [0 1 0 1 1 0 1 1 1 0 1 1 1 1]]

Lets take a closer look:

'All my cats in a row' = [1 0 1 0 0 1 0 0 1 1 0 0 0 0]

If we follow the order of the vocabulary:

bag of words, feature extraction

we’ll get a vector, the bag of words representation.

Bag of words code

We’ll define a collection of strings called a corpus. Then we’ll use the CountVectorizer to create vectors from the corpus.

  # Feature extraction from text
# Method: bag of words 
# https://pythonprogramminglanguage.com

from sklearn.feature_extraction.text import CountVectorizer

corpus = [
'All my cats in a row',
'When my cat sits down, she looks like a Furby toy!',
'The cat from outer space',
'Sunshine loves to sit like this for some reason.'
]

vectorizer = CountVectorizer()
print( vectorizer.fit_transform(corpus).todense() )
print( vectorizer.vocabulary_ )

The steps

  • Split the text into lowercase words, usually with split().
  • Count them with a Counter or a dictionary.
  • Pick a fixed list of words, the vocabulary, and use the counts for those words as the feature vector.

Scikit-learn has it ready in CountVectorizer. It also removes stop words like the and of, and it can count pairs of words with the ngram_range argument.

Two limits worth knowing: the order of the words is gone, so bag of words cannot see that not good is different from good not, and a document with a word repeated many times gets a very large number while a long document with many different words looks small. TF-IDF is the usual next step, it weighs the words instead of counting them.

Want more practice on this? Practise this on PyChallenge, there are short browser exercises you can run right after reading.