assignment 2 training your mind (pst)

Assignment 2: Parts-of-Speech Tagging (POS)

Assignment 2: parts-of-speech tagging (pos) #.

Welcome to the second assignment of Course 2 in the Natural Language Processing specialization. This assignment will develop skills in part-of-speech (POS) tagging, the process of assigning a part-of-speech tag (Noun, Verb, Adjective…) to each word in an input text. Tagging is difficult because some words can represent more than one part of speech at different times. They are Ambiguous . Let’s look at the following example:

The whole team played well . [adverb]

You are doing well for yourself. [adjective]

Well , this assignment took me forever to complete. [interjection]

The well is dry. [noun]

Tears were beginning to well in her eyes. [verb]

Distinguishing the parts-of-speech of a word in a sentence will help you better understand the meaning of a sentence. This would be critically important in search queries. Identifying the proper noun, the organization, the stock symbol, or anything similar would greatly improve everything ranging from speech recognition to search. By completing this assignment, you will:

Learn how parts-of-speech tagging works

Compute the transition matrix A in a Hidden Markov Model

Compute the emission matrix B in a Hidden Markov Model

Compute the Viterbi algorithm

Compute the accuracy of your own model

Important Note on Submission to the AutoGrader #

Before submitting your assignment to the AutoGrader, please make sure you are not doing the following:

You have not added any extra print statement(s) in the assignment.

You have not added any extra code cell(s) in the assignment.

You have not changed any of the function parameters.

You are not using any global variables inside your graded exercises. Unless specifically instructed to do so, please refrain from it and use the local variables instead.

You are not changing the assignment code where it is not required, like creating extra variables.

If you do any of the following, you will get something like, Grader not found (or similarly unexpected) error upon submitting your assignment. Before asking for help/debugging the errors in your assignment, check for these first. If this is the case, and you don’t remember the changes you have made, you can get a fresh copy of the assignment by following these instructions .

0 Data Sources

1 POS Tagging

1.1 Training

Exercise 01

1.2 Testing

Exercise 02

2 Hidden Markov Models

2.1 Generating Matrices

Exercise 03

Exercise 04

3 Viterbi Algorithm

3.1 Initialization

Exercise 05

3.2 Viterbi Forward

Exercise 06

3.3 Viterbi Backward

Exercise 07

4 Predicting on a data set

Exercise 08

Part 0: Data Sources #

This assignment will use two tagged data sets collected from the Wall Street Journal (WSJ) .

Here is an example ‘tag-set’ or Part of Speech designation describing the two or three letter tag and their meaning.

One data set ( WSJ-2_21.pos ) will be used for training .

The other ( WSJ-24.pos ) for testing .

The tagged training data has been preprocessed to form a vocabulary ( hmm_vocab.txt ).

The words in the vocabulary are words from the training set that were used two or more times.

The vocabulary is augmented with a set of ‘unknown word tokens’, described below.

The training set will be used to create the emission, transmission and tag counts.

The test set (WSJ-24.pos) is read in to create y .

This contains both the test text and the true tag.

The test set has also been preprocessed to remove the tags to form test_words.txt .

This is read in and further processed to identify the end of sentences and handle words not in the vocabulary using functions provided in utils_pos.py .

This forms the list prep , the preprocessed text used to test our POS taggers.

A POS tagger will necessarily encounter words that are not in its datasets.

To improve accuracy, these words are further analyzed during preprocessing to extract available hints as to their appropriate tag.

For example, the suffix ‘ize’ is a hint that the word is a verb, as in ‘final-ize’ or ‘character-ize’.

A set of unknown-tokens, such as ‘–unk-verb–’ or ‘–unk-noun–’ will replace the unknown words in both the training and test corpus and will appear in the emission, transmission and tag data structures.

Implementation note:

For python 3.6 and beyond, dictionaries retain the insertion order.

Furthermore, their hash-based lookup makes them suitable for rapid membership tests.

If di is a dictionary, key in di will return True if di has a key key , else False .

The dictionary vocab will utilize these features.

Part 1: Parts-of-speech tagging #

Part 1.1 - training #.

You will start with the simplest possible parts-of-speech tagger and we will build up to the state of the art.

In this section, you will find the words that are not ambiguous.

For example, the word is is a verb and it is not ambiguous.

In the WSJ corpus, \(86\) % of the token are unambiguous (meaning they have only one tag)

About \(14\%\) are ambiguous (meaning that they have more than one tag)

Before you start predicting the tags of each word, you will need to compute a few dictionaries that will help you to generate the tables.

Transition counts #

The first dictionary is the transition_counts dictionary which computes the number of times each tag happened next to another tag.

This dictionary will be used to compute: $ \(P(t_i |t_{i-1}) \tag{1}\) $

This is the probability of a tag at position \(i\) given the tag at position \(i-1\) .

In order for you to compute equation 1, you will create a transition_counts dictionary where

The keys are (prev_tag, tag)

The values are the number of times those two tags appeared in that order.

Emission counts #

The second dictionary you will compute is the emission_counts dictionary. This dictionary will be used to compute:

In other words, you will use it to compute the probability of a word given its tag.

In order for you to compute equation 2, you will create an emission_counts dictionary where

The keys are (tag, word)

The values are the number of times that pair showed up in your training set.

Tag counts #

The last dictionary you will compute is the tag_counts dictionary.

The key is the tag

The value is the number of times each tag appeared.

Exercise 01 #

Instructions: Write a program that takes in the training_corpus and returns the three dictionaries mentioned above transition_counts , emission_counts , and tag_counts .

emission_counts : maps (tag, word) to the number of times it happened.

transition_counts : maps (prev_tag, tag) to the number of times it has appeared.

tag_counts : maps (tag) to the number of times it has occured.

Implementation note: This routine utilises defaultdict , which is a subclass of dict .

A standard Python dictionary throws a KeyError if you try to access an item with a key that is not currently in the dictionary.

In contrast, the defaultdict will create an item of the type of the argument, in this case an integer with the default value of 0.

See defaultdict .

Expected Output #

The ‘states’ are the Parts-of-speech designations found in the training data. They will also be referred to as ‘tags’ or POS in this assignment.

“NN” is noun, singular,

‘NNS’ is noun, plural.

In addition, there are helpful tags like ‘–s–’ which indicate a start of a sentence.

You can get a more complete description at Penn Treebank II tag set .

Part 1.2 - Testing #

Now you will test the accuracy of your parts-of-speech tagger using your emission_counts dictionary.

Given your preprocessed test corpus prep , you will assign a parts-of-speech tag to every word in that corpus.

Using the original tagged test corpus y , you will then compute what percent of the tags you got correct.

Exercise 02 #

Instructions: Implement predict_pos that computes the accuracy of your model.

This is a warm up exercise.

To assign a part of speech to a word, assign the most frequent POS for that word in the training set.

Then evaluate how well this approach works. Each time you predict based on the most frequent POS for the given word, check whether the actual POS of that word is the same. If so, the prediction was correct!

Calculate the accuracy as the number of correct predictions divided by the total number of words for which you predicted the POS tag.

88.9% is really good for this warm up exercise. With hidden markov models, you should be able to get 95% accuracy.

Part 2: Hidden Markov Models for POS #

Now you will build something more context specific. Concretely, you will be implementing a Hidden Markov Model (HMM) with a Viterbi decoder

The HMM is one of the most commonly used algorithms in Natural Language Processing, and is a foundation to many deep learning techniques you will see in this specialization.

In addition to parts-of-speech tagging, HMM is used in speech recognition, speech synthesis, etc.

By completing this part of the assignment you will get a 95% accuracy on the same dataset you used in Part 1.

The Markov Model contains a number of states and the probability of transition between those states.

In this case, the states are the parts-of-speech.

A Markov Model utilizes a transition matrix, A .

A Hidden Markov Model adds an observation or emission matrix B which describes the probability of a visible observation when we are in a particular state.

In this case, the emissions are the words in the corpus

The state, which is hidden, is the POS tag of that word.

Part 2.1 Generating Matrices #

Creating the ‘a’ transition probabilities matrix #.

Now that you have your emission_counts , transition_counts , and tag_counts , you will start implementing the Hidden Markov Model.

This will allow you to quickly construct the

A transition probabilities matrix.

and the B emission probabilities matrix.

You will also use some smoothing when computing these matrices.

Here is an example of what the A transition matrix would look like (it is simplified to 5 tags for viewing. It is 46x46 in this assignment.):

| A |…| RBS | RP | SYM | TO | UH|… | — ||—:————-| ———— | ———— | ——– | ———- |—- | RBS |…|2.217069e-06 |2.217069e-06 |2.217069e-06 |0.008870 |2.217069e-06|… | RP |…|3.756509e-07 |7.516775e-04 |3.756509e-07 |0.051089 |3.756509e-07|… | SYM |…|1.722772e-05 |1.722772e-05 |1.722772e-05 |0.000017 |1.722772e-05|… | TO |…|4.477336e-05 |4.472863e-08 |4.472863e-08 |0.000090 |4.477336e-05|… | UH |…|1.030439e-05 |1.030439e-05 |1.030439e-05 |0.061837 |3.092348e-02|… | … |…| … | … | … | … | … | …

Note that the matrix above was computed with smoothing.

Each cell gives you the probability to go from one part of speech to another.

In other words, there is a 4.47e-8 chance of going from parts-of-speech TO to RP .

The sum of each row has to equal 1, because we assume that the next POS tag must be one of the available columns in the table.

The smoothing was done as follows:

\(N\) is the total number of tags

\(C(t_{i-1}, t_{i})\) is the count of the tuple (previous POS, current POS) in transition_counts dictionary.

\(C(t_{i-1})\) is the count of the previous POS in the tag_counts dictionary.

\(\alpha\) is a smoothing parameter.

Exercise 03 #

Instructions: Implement the create_transition_matrix below for all tags. Your task is to output a matrix that computes equation 3 for each cell in matrix A .

Create the ‘B’ emission probabilities matrix #

Now you will create the B transition matrix which computes the emission probability.

You will use smoothing as defined below:

\(C(t_i, word_i)\) is the number of times \(word_i\) was associated with \(tag_i\) in the training data (stored in emission_counts dictionary).

\(C(t_i)\) is the number of times \(tag_i\) was in the training data (stored in tag_counts dictionary).

\(N\) is the number of words in the vocabulary

The matrix B is of dimension (num_tags, N), where num_tags is the number of possible parts-of-speech tags.

Here is an example of the matrix, only a subset of tags and words are shown:

B Emissions Probability Matrix (subset)

Exercise 04 #

Instructions: Implement the create_emission_matrix below that computes the B emission probabilities matrix. Your function takes in \(\alpha\) , the smoothing parameter, tag_counts , which is a dictionary mapping each tag to its respective count, the emission_counts dictionary where the keys are (tag, word) and the values are the counts. Your task is to output a matrix that computes equation 4 for each cell in matrix B .

Part 3: Viterbi Algorithm and Dynamic Programming #

In this part of the assignment you will implement the Viterbi algorithm which makes use of dynamic programming. Specifically, you will use your two matrices, A and B to compute the Viterbi algorithm. We have decomposed this process into three main steps for you.

Initialization - In this part you initialize the best_paths and best_probabilities matrices that you will be populating in feed_forward .

Feed forward - At each step, you calculate the probability of each path happening and the best paths up to that point.

Feed backward : This allows you to find the best path with the highest probabilities.

Part 3.1: Initialization #

You will start by initializing two matrices of the same dimension.

best_probs: Each cell contains the probability of going from one POS tag to a word in the corpus.

best_paths: A matrix that helps you trace through the best possible path in the corpus.

Exercise 05 #

Instructions : Write a program below that initializes the best_probs and the best_paths matrix.

Both matrices will be initialized to zero except for column zero of best_probs .

Column zero of best_probs is initialized with the assumption that the first word of the corpus was preceded by a start token (“–s–”).

This allows you to reference the A matrix for the transition probability

Here is how to initialize column 0 of best_probs :

The probability of the best path going from the start index to a given POS tag indexed by integer \(i\) is denoted by \(\textrm{best_probs}[s_{idx}, i]\) .

This is estimated as the probability that the start tag transitions to the POS denoted by index \(i\) : \(\mathbf{A}[s_{idx}, i]\) AND that the POS tag denoted by \(i\) emits the first word of the given corpus, which is \(\mathbf{B}[i, vocab[corpus[0]]]\) .

Note that vocab[corpus[0]] refers to the first word of the corpus (the word at position 0 of the corpus).

vocab is a dictionary that returns the unique integer that refers to that particular word.

Conceptually, it looks like this: \(\textrm{best_probs}[s_{idx}, i] = \mathbf{A}[s_{idx}, i] \times \mathbf{B}[i, corpus[0] ]\)

In order to avoid multiplying and storing small values on the computer, we’ll take the log of the product, which becomes the sum of two logs:

\(best\_probs[i,0] = log(A[s_{idx}, i]) + log(B[i, vocab[corpus[0]]\)

Also, to avoid taking the log of 0 (which is defined as negative infinity), the code itself will just set \(best\_probs[i,0] = float('-inf')\) when \(A[s_{idx}, i] == 0\)

So the implementation to initialize \(best\_probs\) looks like this:

\( \textrm{if}\ A[s_{idx}, i] <> 0 : best\_probs[i,0] = log(A[s_{idx}, i]) + log(B[i, vocab[corpus[0]]])\)

\( \textrm{if}\ A[s_{idx}, i] == 0 : best\_probs[i,0] = float('-inf')\)

Please use math.log to compute the natural logarithm.

The example below shows the initialization assuming the corpus starts with the phrase “Loss tracks upward”.

Represent infinity and negative infinity like this:

Part 3.2 Viterbi Forward #

In this part of the assignment, you will implement the viterbi_forward segment. In other words, you will populate your best_probs and best_paths matrices.

Walk forward through the corpus.

For each word, compute a probability for each possible tag.

Unlike the previous algorithm predict_pos (the ‘warm-up’ exercise), this will include the path up to that (word,tag) combination.

Here is an example with a three-word corpus “Loss tracks upward”:

Note, in this example, only a subset of states (POS tags) are shown in the diagram below, for easier reading.

In the diagram below, the first word “Loss” is already initialized.

The algorithm will compute a probability for each of the potential tags in the second and future words.

Compute the probability that the tag of the second work (‘tracks’) is a verb, 3rd person singular present (VBZ).

In the best_probs matrix, go to the column of the second word (‘tracks’), and row 40 (VBZ), this cell is highlighted in light orange in the diagram below.

Examine each of the paths from the tags of the first word (‘Loss’) and choose the most likely path.

An example of the calculation for one of those paths is the path from (‘Loss’, NN) to (‘tracks’, VBZ).

The log of the probability of the path up to and including the first word ‘Loss’ having POS tag NN is \(-14.32\) . The best_probs matrix contains this value -14.32 in the column for ‘Loss’ and row for ‘NN’.

Find the probability that NN transitions to VBZ. To find this probability, go to the A transition matrix, and go to the row for ‘NN’ and the column for ‘VBZ’. The value is \(4.37e-02\) , which is circled in the diagram, so add \(-14.32 + log(4.37e-02)\) .

Find the log of the probability that the tag VBS would ‘emit’ the word ‘tracks’. To find this, look at the ‘B’ emission matrix in row ‘VBZ’ and the column for the word ‘tracks’. The value \(4.61e-04\) is circled in the diagram below. So add \(-14.32 + log(4.37e-02) + log(4.61e-04)\) .

The sum of \(-14.32 + log(4.37e-02) + log(4.61e-04)\) is \(-25.13\) . Store \(-25.13\) in the best_probs matrix at row ‘VBZ’ and column ‘tracks’ (as seen in the cell that is highlighted in light orange in the diagram).

All other paths in best_probs are calculated. Notice that \(-25.13\) is greater than all of the other values in column ‘tracks’ of matrix best_probs , and so the most likely path to ‘VBZ’ is from ‘NN’. ‘NN’ is in row 20 of the best_probs matrix, so \(20\) is the most likely path.

Store the most likely path \(20\) in the best_paths table. This is highlighted in light orange in the diagram below.

The formula to compute the probability and path for the \(i^{th}\) word in the \(corpus\) , the prior word \(i-1\) in the corpus, current POS tag \(j\) , and previous POS tag \(k\) is:

\(\mathrm{prob} = \mathbf{best\_prob}_{k, i-1} + \mathrm{log}(\mathbf{A}_{k, j}) + \mathrm{log}(\mathbf{B}_{j, vocab(corpus_{i})})\)

where \(corpus_{i}\) is the word in the corpus at index \(i\) , and \(vocab\) is the dictionary that gets the unique integer that represents a given word.

\(\mathrm{path} = k\)

where \(k\) is the integer representing the previous POS tag.

Exercise 06 #

Instructions: Implement the viterbi_forward algorithm and store the best_path and best_prob for every possible tag for each word in the matrices best_probs and best_tags using the pseudo code below.

• Remember that when accessing emission matrix B, the column index is the unique integer ID associated with the word. It can be accessed by using the 'vocab' dictionary, where the key is the word, and the value is the unique integer ID for that word.

Run the viterbi_forward function to fill in the best_probs and best_paths matrices.

Note that this will take a few minutes to run. There are about 30,000 words to process.

Part 3.3 Viterbi backward #

Now you will implement the Viterbi backward algorithm.

The Viterbi backward algorithm gets the predictions of the POS tags for each word in the corpus using the best_paths and the best_probs matrices.

The example below shows how to walk backwards through the best_paths matrix to get the POS tags of each word in the corpus. Recall that this example corpus has three words: “Loss tracks upward”.

POS tag for ‘upward’ is RB

Select the the most likely POS tag for the last word in the corpus, ‘upward’ in the best_prob table.

Look for the row in the column for ‘upward’ that has the largest probability.

Notice that in row 28 of best_probs , the estimated probability is -34.99, which is larger than the other values in the column. So the most likely POS tag for ‘upward’ is RB an adverb, at row 28 of best_prob .

The variable z is an array that stores the unique integer ID of the predicted POS tags for each word in the corpus. In array z, at position 2, store the value 28 to indicate that the word ‘upward’ (at index 2 in the corpus), most likely has the POS tag associated with unique ID 28 (which is RB ).

The variable pred contains the POS tags in string form. So pred at index 2 stores the string RB .

POS tag for ‘tracks’ is VBZ

The next step is to go backward one word in the corpus (‘tracks’). Since the most likely POS tag for ‘upward’ is RB , which is uniquely identified by integer ID 28, go to the best_paths matrix in column 2, row 28. The value stored in best_paths , column 2, row 28 indicates the unique ID of the POS tag of the previous word. In this case, the value stored here is 40, which is the unique ID for POS tag VBZ (verb, 3rd person singular present).

So the previous word at index 1 of the corpus (‘tracks’), most likely has the POS tag with unique ID 40, which is VBZ .

In array z , store the value 40 at position 1, and for array pred , store the string VBZ to indicate that the word ‘tracks’ most likely has POS tag VBZ .

POS tag for ‘Loss’ is NN

In best_paths at column 1, the unique ID stored at row 40 is 20. 20 is the unique ID for POS tag NN .

In array z at position 0, store 20. In array pred at position 0, store NN .

Exercise 07 #

Implement the viterbi_backward algorithm, which returns a list of predicted POS tags for each word in the corpus.

Note that the numbering of the index positions starts at 0 and not 1.

m is the number of words in the corpus.

So the indexing into the corpus goes from 0 to m - 1 .

Also, the columns in best_probs and best_paths are indexed from 0 to m - 1

In Step 1: Loop through all the rows (POS tags) in the last entry of best_probs and find the row (POS tag) with the maximum value. Convert the unique integer ID to a tag (a string representation) using the list states .

Referring to the three-word corpus described above:

z[2] = 28 : For the word ‘upward’ at position 2 in the corpus, the POS tag ID is 28. Store 28 in z at position 2.

states[28] is ‘RB’: The POS tag ID 28 refers to the POS tag ‘RB’.

pred[2] = 'RB' : In array pred , store the POS tag for the word ‘upward’.

Starting at the last column of best_paths, use best_probs to find the most likely POS tag for the last word in the corpus.

Then use best_paths to find the most likely POS tag for the previous word.

Update the POS tag for each word in z and in preds .

Referring to the three-word example from above, read best_paths at column 2 and fill in z at position 1. z[1] = best_paths[z[2],2]

The small test following the routine prints the last few words of the corpus and their states to aid in debug.

Expected Output:

Now you just have to compare the predicted labels to the true labels to evaluate your model on the accuracy metric!

Part 4: Predicting on a data set #

Compute the accuracy of your prediction by comparing it with the true y labels.

pred is a list of predicted POS tags corresponding to the words of the test_corpus .

Exercise 08 #

Implement a function to compute the accuracy of the viterbi algorithm’s POS tag predictions.

To split y into the word and its tag you can use y.split() .

Congratulations you were able to classify the parts-of-speech with 95% accuracy.

Key Points and overview #

In this assignment you learned about parts-of-speech tagging.

In this assignment, you predicted POS tags by walking forward through a corpus and knowing the previous word.

There are other implementations that use bidirectional POS tagging.

Bidirectional POS tagging requires knowing the previous word and the next word in the corpus when predicting the current word’s POS tag.

Bidirectional POS tagging would tell you more about the POS instead of just knowing the previous word.

Since you have learned to implement the unidirectional approach, you have the foundation to implement other POS taggers used in industry.

References #

“Speech and Language Processing”, Dan Jurafsky and James H. Martin

We would like to thank Melanie Tosik for her help and inspiration

Provide details on what you need help with along with a budget and time limit. Questions are posted anonymously and can be made 100% private.

Studypool matches you to the best tutor to help you with your question. Our tutors are highly qualified and vetted.

Your matched tutor provides personalized help according to your question details. Payment is made only after you have completed your 1-on-1 session and are satisfied with your session.

Implementing PST Developing Your Own PST Program

User Generated

Description

Directions:

Choose two PST skills that meet the needs of your team or specific situation.

• Communication skills
• Imagery skills
• Relaxation skills
• Stress management skills
• Managing psychic energy
• Attentional skills
• Self-Confidence and goal-setting skills

Develop a four week PST program that includes:

• acquisition

Identify your specific needs and the time you are willing to commit to PST.

The following outline can help you organize your thoughts. Your project may include this information and should be written specific to your team’s needs or situation. Remember, this is your project, personalize it, design it to meet your needs. Write it as you would like to implement your training program.

Introduction: the setting for your PST program.

Team information.

• Sport, size of team, skill level, age, number of assistants, etc.
• Team and athletes needs assessment.

What two psychological skills will you introduce and why?

• PST program objectives and goals.
• Any obstacles you must overcome?
• How will you evaluate your PST program?
• Mandatory or volunteer?
• Orientation

How will you present an overview of their PST program?

• When, where, how long?
• Activities, if any are planned.
• Approach you will take to encourage their presentation.
• Education Phase of PST.
• Importance of skills and their effect on performance.
• Key concepts for each skill.

How skills will be taught and practiced.

• Identify self-evaluation tools for each skill.
• Goals for each skill; how athletes will set personal goals.
• Acquisition Phase of PST.
• Structure of training program.
• Individual or group sessions? Why?
• Supervised practice or independent practice?
• Why/Where and how much time for practice?
• Activities for each skill.
• Teaching progression for each skill.

How will athletes monitor their progress?

• Practice Phase of PST.

How skills will be integrated into practice; possibly competition?

• Outline and develop your four-week practice plan.
• Evaluation of PST program: how will you determine the success of your program?

HERE IS EXAMPLE FROM INSTRUCTOR THE WAY IT SHOULD LOOK. http://www.bcconline.info/psyc13mk/pst.html

Explanation & Answer

24/7 Study Help

Stuck on a study question? Our verified tutors can answer all questions, from basic  math  to advanced rocket science !

Similar Content

Related tags.

methodology Argosy University development development psychology assignment lucifer effect therapies allopathic medicines environment anxiety education academic

Underground A Human History of the Worlds Beneath our Feet

by Will Hunt

Twelve Years A Slave

by Solomon Northrup

The Fault in Our Stars

by John Green

The Book Thief

by Markus Zusak

Unf*ck Yourself

by Gary John Bishop

Rules Of Civility

by Amor Towles

Z for Zachariah

by Robert C. O’Brien

The President is Missing

by James Patterson, Bill Clinton

by Niccolò Machiavelli

working on a study question?

Studypool is powered by Microtutoring TM

Copyright © 2024. Studypool Inc.

Studypool is not sponsored or endorsed by any college or university.

Ongoing Conversations

Access over 35 million study documents through the notebank

Get on-demand Q&A study help from verified tutors

Read 1000s of rich book guides covering popular titles

Sign up with Google

Sign up with Facebook

Already have an account? Login

Login with Google

Login with Facebook

Don't have an account? Sign Up