NLP-2

• single document file (e.g. mail, pdf, doc, …)

• collection of documents

• minimal string of characters t

• with meaning to humans

• union of all words in all of the documents

• list of common, non-informative words

• e.g. me, no, the, are….

• string of characters

• but not necesarrily a word

• -> result of tokenization process…

• stop word removal

• map to lower case

• punctuation removal

• demarcating and partitinoing string

• into smaller units…

• reducing inflected words to their word stem

• => map semantically equivalent words to same tokens

• assign each word in the vocabulary a one hot encoded vector

• => eg sort alphabetically

• pro: simple (e.g. good as baseline for further methods)

• con: simple, sparse

• significance of a word for a document in the context of a corpus

• -> TF: significance of word for document

• -> IDF: specifity of word for the document w.r.t. whole corpus

=> larger is better…

• lowercase

• stemming

• tokenization

• calculate TF-IDF for each word (in vocabulkary -> also words not in document but corpus…)

• -> create vector from it…

• examine impact of copy paste from stack overflow

• -> collcet set of vuln. code snippets

• -> apply TF-IDF

• -> train classifier

• are binaries related / malicious?

• learn the semantics of a word

• -> use this knowledge to predict probability of other words being close to a specific word (based on learning on text)

• -> represent this as softmax vector (probability for each other word to be related to this word…)

• we have sequence of words

• -> initialize a dictionary of the words

• -> encode each word as 1-hot vector

• model two layer dense network and train:

• (X,y) = (input, neighbor)

• -> use several neighbors with distance c…

• apply softmax loss to it…

• maximize

• 1/l (number of words)

• Sum over all words

• Sum over neighbors

• p_theta(v_i+j|v_i)

=> maximize for each word

=> the confidence that actually neighboring words are neighbors

-> as p_theta(v_i+j|v_i) =

v_i+j * softmax(f(v_i))

=> thus if it has high confidence -> the respective entry in the softmax output should be close (optimally) 1….

• not maxximizatin

• but minimize negative log ….

• => same outcome but better optimized minimizers….

• numerically stable…

• TF-IDF fast

• Word2Vec slow

  
  

• TF-IDF fast

• word2vec -> fast (lookup after training…)

• TF iDF: summarization via word counts

• word2vec: semantics w.r.t. only neigbhoring word

• TF-IDF -> document embedding

• Word2vec -> word embedding

• TF: sparse, size |V|

• wodd2vec: dense, size h

• no word sense disambiguation (cell as in prison, biology, telephone, battery,…)

• cannot handle out of vocab words

• ignores order of words (up to neighboring relationship=

• based on word conuts or word co-occurence -> far from real semantic understanding…

• simple summation -> 1/n * Sum over embedding vectors (with n being number of words in document)=

• weighted summation via TF-iDF

• -> Sum over words i in document j

• TF-IDF(i,j) * embedding of word i