PMI Examination Methodology.

I'm posting excerpts from an interesting thread in PMZilla (for the complete version visit PMZilla). The thread was originally posted by dbissonn and his question was: 
"How are PMP exam questions weighted?

After passing numerous prep exams, including Rita's, I failed my first real PMP exam.  I called PMI, and while they would not use the term, apparently some questions wieighted, are given more credit than others.  It used to be a straight 61%, but no more.  Here is a quote from an email they sent me:

"Scaled Scoring is not a percentage-based scoring system, that is, the assessment passing point is assigned to a point of difficulty instead of a number or percent of correct responses."

The problem is no one will say anything about what is more or less weighted.  I suspect that, for example, not being able to do EV, or Crit Path, or float, will cost you more than not knowing what a fixed fee contract implies.

 Does anyone know how PMI actually determines how one has passed or failed? "

He continues...
"The PMP exam report does not give % at all.  There are three ratings for each of 5 process groups plus professional and social resp.  Results are not broken down for KNOWLEDGE AREA).  Proficient is highest, Moderately Proficient, and Below Perficient.  I got the lowest in three. I have no way of knowing what I missed or their weighted value, in fact I am sure PMI does not want that made public.  It is possible, and this is a rough guess, to score 74% right for the whole test, but miss several high weighted questions, or score below 61% in one area, and still fail.

 On the last 10 or so practice exams I was never below 70%, and consistently above 75.  I think the real issue here is not how PMI does things, they are trying to judge if someone is truly knowledgeable, and they are using sophisticated testing methods to do so.  I have no problem with that.  But I think we need realize that preparation exams like Rita Mulcahy's FASTrack (US$299!!!) use a much less sophisticated method...they only consider a straight percentage, which is NOT the way the real exam works.



 I suggest no one call himself prepared with less than 80% right on practice exams."


Here's a consolidation of my responses:
"We all would like to hear how the exams are evaluated, wouldn't we? But I guess from the way the exam and the grading is setup, PMI is trying to ensure that there is no way to dis-assemble or reverse-engineer the exam process. Why?

First a look at what we know: 
The exam would result in a "pass" or "fail" and does not annouce the percentage or grade scored. We know that of the 200 questions 175 are actually used for evaluating the candidate and one must have answered atleast 106 out of 175 correctly in order to pass. (A combination of "Modified Angoff Technique" and psychometric method adopted by PMI since Sept 2005.) That's 61% of the questions.

Now the exam transcript, after you finish the exam, actually breaks the exam into Process Groups and your profeciency in it. And according to PMI the spread of questions is thus:

Initiating - 23 Questions

Planning - 46

Executing - 53

M&C - 42

Closing - 18

Professional & Social Responsibility - 18

The transcript does not merely say, "Pass/Fail" for each of the knowledge areas. It rates it as "Proficient", "Moderately proficient" and "Below Proficient". And you need to have Moderately Proficient and above in 4 of the 6 knowledge areas to pass. (correct me here, if i'm wrong) (moderately proficient in 4 of 6 KAs will result in a MAX score of 100, which is 6 less than the requisite 106. Which means the scores have to be a either Moderately Profient in all the areas or a combination of Moderately proficient and Proficient in 4 of 6 Knowledge Areas.) 
One could try to reverse engineer this and say that a 61% pass on each of the knowledge area (14 in Initiating, 28 in Planning, 32 in Executing, 26 in M&C, 11 in Closing, 11 in Professional Conduct. Moderately Proficient on all, sums up to 122) could fetch us passing results, but it needs to be confirmed if it is just that or more. In reverse-engineering, we need to ensure that the number of correct answers sums up to 106 or more and then we could look at the areas that the correct answers come from and rate it on proficiency.

 Add to this the fact that the questions are rated using the Modified Angoff method, where volunteers for PMI adjudge the difficulty level of the questions. The tougher a question, lower are the points gained from getting the answer right, and easier the question, higher the points.One would therefore notice that the way the exam goes, we'll see a whole bunch of easy questions appearing together followed by a series of really tough ones.

There is no sure fire way of determining how the results would appear. But to be on the safer side, ensure that you are proficient on all the individual Knowledge Areas while taking mock exams. And by Proficient I'm presuming 80% and above. Then go for the complete exams.

There is another point to the way the exam is structured and evaluated. Well, like all the big kahunas of PMP training and preperation have been saying, no one person can get all of the answers right in the exam. That's because of the nature of the exam and each person's take on the subject. But end of the day, the reason some of us are PMPs is because of a common understanding of the practices of Project Management.

On a side note, I really don't understand how RMC's products got so much of a hype. They are good, but really not that great! Especially when you consider what they charge for it. There are free products out there that should be making a killing for the kind of quality they deliver (case in point Oliver Lehmann's 75 and 125 questions)."


So... Your thoughts or comments?...

Sample PMP application spreadsheet - v3

Finally,

Here's the 3rd cut of the PMP application spreadsheet. Thanks to everybody who took a moment to share feedback.

Sample PMP Application Version 3

pmp-application-spreadsheet version 2.xls

***For people having difficulty downloading from Rapidshare, I have hidden the .xls file in the following image file (cause blogger does not support uploading of any other file types).

Download this image file and open it with WinRar (download an evaluation version of winrar from here: www.rarlabs.com)***

Updates and fixes:

More columns for people who have a lot of project experience to fill in. ;)
Computes project experience just like the online PMI form: unique, non-overlapping months only.

Has similar interface but with a little more validation, etc.
Inserted the excel sheet into both the image files. The excel spreadsheet can now be extracted from images above. Just download to your system and open with winrar.

I do appreciate any comments or feedback. This spreadsheet is free to use and distribute, but please do provide credits where due. I'd like to thank the gurus Colin Legg and Barry Houdini for helping me with some tricky formulas.

Cheers!

Kano Method - I

Introduction to the Kano Method

As part of a Dissertation work, I used the Kano Method to evaluate Customer Satisfaction. Below are excerpts from the dissertation. (Marketability of Solar Home Systems in the Rural Areas of Thumkur and Doddaballapur - Ranjit Arvindakshan & Deepak Sriram)
When one talks about Product features and Customer satisfaction, it is generally assumed that customer satisfaction is somehow directly proportional to how functional the product is – that is, the less functional the product, the less satisfied the customer is likely to be, and the more functional the product, the more satisfied the customer. This was a very one-dimensional way of looking at Customer Satisfaction.
The Kano model for understanding product features and their influence of Customer Satisfaction was created by Noriaki Kano, Professor at Tokyo Rika University in 1984. In his model, Kano, suggests a 3-dimensional model to relate product features with Customer Satisfaction. Since then the Kano method has been used by many a Marketing Strategist while considering product features and specifications. The power of the Kano model in Marketing lies in its requirement to collect the Voice of the Customer for product specifications. It classifies product features into Delighters, Linear Satisfiers, and Must Haves.

Fig: Kano diagram

The curved blue line running across the top two quadrants show Attractive requirements. Notice that as the number of Attractive requirements increase, the more satisfied the customer is likely to get. The curved red line running across the bottom two quadrants indicate the Must-have requirements. These features are a given and the absence of the features is likely to get the customer increasingly dissatisfied. The diagonal green line is the One-dimensional requirement, indicating satisfaction versus functionality.

It is to be noted that as the customers get used to the Delighters (Attractive Requirements, or differentiators), over time they become Must-have requirements as the customer begins to expect those features in all the models.

The Kano model requires that we ask specific questions that assess Customer Requirements against functional and dysfunctional features. The questionnaire is designed such that the questions have two parts: 1) How do you feel if a feature is present in the product? 2) How do you feel if that feature is not present in the product? The questions are rated on a 5-point scale:

1. I like it that way

2. It must be that way

3. I am neutral

4. I can live with it that way

5. I dislike it that way

The responses are then classified into one of six categories:

A: Attractive

M: Must-be

I: Indifferent

O: One-dimensional

Q: Questionable

R: Reverse

Category Q stands for questionable result. Normally, the answers do not fall into this category. Questionable scores signify that the question was phrased in-correctly, or that the person interviewed misunderstood the question or crossed out a wrong answer by mistake.

Category R, indicates that this product feature is not only not wanted by the customer but he even expects the reverse. For instance, when offering holiday tours it might well be that a specific customer segment wants pre-planned events every day, while another would dislike it.

Fig: Kano Evaluation Table

My PMP Lessons Learnt

That's right. I got PMP certified!

I realised, from my efforts to prepare for the certification, that the network of Project Managers is all about collaboration and sharing. There are countless forums that share information on preparing for the PMP, or resources to help in the studies. In the spirit of that, I thought I'll share some of my lessons learnt from the exams before the news of it gets stale. :)

I underwent training in July (4 days, all weekends at Quahance) and registered for the exam during my second day of the training. I realised from my last adventure that taking the time to register for the exam puts you off track and therefore not ready for the exam.

Some of the highlights from the exam:

• There were no negative-type or trick questions (questions with "not" in them eg. Which of these is NOT an example of Risk mitigation?)
• I thought the Activity on Arrow was done away with the introduction of the 4th edition, but i got an AOA question that required me to compute the critical path. Not so tough, but would be good to be prepared.
• Lots of questions on different types of EAC and earned value. The problems were not the straight forward "Calculate CV, given EV and AC" type, but a little more trickier. Good thing i was prepared.
• Lots of questions on contract types, Risk.
• It was easy in parts and tough in parts.
• I thought there will be lots of wordy questions but I only got a couple like that and it did not bother me much.
• Most questions were on Earned Value, Risk, Procurement, communication and conduct (not in any order)
• Finished the exam in 2hrs and 10mins. This is with me taking the time to read and also go over a few questions that i had marked for review.
• I did make changes to the original selection for the questions that i had marked for review
• When the questions got tough, it got tougher and tougher before easing up again. Some of the choices on the tough questions seemed like all choices could be right (i'm guessing these were the sampling questions). The moderate and easy ones could easily be spotted with just one right choice.
• I finished the exam with about 30 mins to spare. Could have spent some more time going through some of the answers i guess, but i had a full bladder by then.

Exam prep LL

• I'm glad that i registered for the exam on the second day of the PMP prep training. The more you delay your application and registration process, the less likely you are to succeed in the exam itself.
• I tried my best to get some of the guys from my training batch to get together for a joint study, but that never worked and i put in my efforts on my own. - Don't bank all your efforts on a group study. If the motivation levels are not the same across your group, be ready to hike it by yourself.
• Always be in study mode. If you have a small study material that you got from the training institute, go over it daily irrespective of whether you have put in hours to study or not. This will ensure retention of key topics.
• The podcast from Cornelius Fishner is really good. I heard a few samples from his site and wish i had invested some money in it. But i did not have the time for that.
• My resources: PMBOK, Andy Crowe and Oliver Lehmann (both the 75 questions and the 125 questions: www.oliverlehmann.com), and no i did not read Rita. I used Rita's fastrack questions for one sample exam but gave it up after using Oliver's questions. Though the questions he has are a little tougher, what i really liked is instead of explaining the correct answer he would point to the source. A lot of the source happened to be books in the PMI library, so i must have read quite a bit about a lot of things that were in addition to what was in Andy Crowe or the PMBOK. Highly recommended!!
• There were a few questions that still stumped me and I contacted Sameer, my trainer from Quahance, for the answers and he was kind enough to respond to me over the weekend despite that being his busiest time! Thanks, Sameer! Also helpful was his mnemonic of the processes which laid the foundation to retaining a lot of the knowledge in the PMBOK.
• Also, if you can get your hands on the "Q&As for the PMBOK guide, Fourth Edition" from PMI; It will be very helpful. I saw a couple of questions today that were straight from the Q&As from the 3rd edition that was available to me in the PMI library. (Curiously the Q&As fourth edition is not available in the PMI library though it is out on print and also on amazon etc). Bottom line, PMI membership = value for money.
• I did not memorize the ITTOs but i knew the process well enough, but i guess it might help to remember what Tools and Techniques are used in which process. There were a lot of questions on these - especially in Quality, Communication, Risk and Procurement.
• In retrospect, i wish i had spent some more time studying. Would have breezed through the exam. But i suffered from the two classic symptoms of most projects: Parkinson's law and the student syndrome. Even in the last week before the exam, i lost 3 whole days due to some issues at home.
  But what the heck, nothing beats seeing "Congratulations" on the screen soon after the end of test survey is done - I did not read what was on the screen beyond that. :)

On with the job hunt now. Good luck to you all!

Disclaimer: PMP, PMI, PMBOK etc are registered trademarks of Project management Institute. Other trademarks and copyrights as applicable.