1	Capability Maps Demonstrating our expertise using Behaviorally Anchored Self-Assessments
2	Problem as Posed by Martin Fowler et al "How do we make it accepted that able people who are expensive end up being better value than cheap, less able developers?"
3	Performance Signatures
4	What are the Drivers? Liability Institutional Consistency Recognition of Specialties Wage compression Underemployment Pay for value
5	Persistent Questions “How could pay for performance truly work?” “Can people really be paid on merit?” “Market value is real value, isn’t it?” We claim competency-based compensation using BASA is practical.
6	Some Current Attempts to Evaluate Merit Sun Certification Microsoft Certification Brainbench TopCoder “Exams can’t measure key capabilities…”
7	Computerworld April 14, 2003
8	Untangling Words Learning, understanding Knowledge, behavior Inferred data, snapshot data Test scores, authentic performance
9	Certification et al: Evaluation alternatives Certification Resumes Portfolios Personal recommendation/project reputation Professional licensure Authorship, recognized trade leadership Professional assessment services Educational credentials, transcripts
10	Practical Problems Credibility is uncertain, untraceable Knowledge vs. Behavior confusion Don’t show authentic performance Predictability vague, retained knowledge nil Scalability unlikely Having to rewrite often is tedious Often expensive and delayed
11	History of BASA: Behaviorally Anchored Self-Assessment Originated as way to measure effectiveness of educational system Used in competence-based performance reviews Used in K-12 education Used in college elearning HR theory cites it as best practice Often described as time consuming
12	What is a BASA? Usually a one page document Organized by Developmental Stages of Expertise Lists behavioral anchors (strongly inferential statements depicting a capability) ~Quarterly the operator/trainee indicates T/F of each anchor item An expert corroborates each item for accuracy
13	The Working Principles Dev. stages as emergent view of expertise Self-awareness of capability yields accurate data ‘Anchors’ are compelling evidence Strong inference from anchors Expert corroboration Simplicity in context and format Compactness via interpolation between anchors Highly reusable & chartable over time
14	Developmental Stages of Expertise Novice Advanced Beginner Competent Proficient Expert Master
15	Anchoring Each Stage (Inferentially)
16	Developmental Stages of Expertise Popular View
17	Logging Expertise in Real Time
18	BASA used extensively, by other names Post-graduate oral exam Medical profession High Risk equipment operator licensure Military: Classical and BASA embedded in simulations Driver’s license K-12 College elearning Some business examples in mfg., hotel, others.
19	Semantic Hierarchy Capability Maps based on Assessments based on Rubrics based on Behavioral anchors based on Understanding/automatic skill based on Self-awareness
20	Example with Mathematics Capability map for a student’s current skills in math: algebra, geometry, stat, calc Assessment for 8^th grade mathematics Specific skill matrices (called ‘rubrics’) {Add, Subt, Mult, Div} for whole numbers Fractions Etc. Example behavioral anchor item would be ‘two place integer multiplication tables’
21	QAL Success Story Case Study Quality Assured Label, Inc. Overview Self-Assessment Using Developmental Stages of Expertise* Performance Review Format Observations of 7 Years BASA Experience Salary Structures/Compensation Plan Capacity Planning Benefits to Employee and Supervisor of BASA Benefits to the Company *Based on Developmental Staging in the Learning Process, Gary L. Jedynak, copyright, 1991.
22	Self-Assessment using Developmental Stages of Expertise Positions used—Press Operator, Finishing Operator/Inspector, General Production Worker Assessment is done in 6 month or 1 year intervals Performance expectations & predictions are based on level of expertise Self-assessment provides data to supervisor and resulting feedback to employee
23	Performance Review Format Review Format is based on Developmental Stages of Expertise Self-Assessment & Supervisor Assessment A structured review format was developed for production jobs Review form can be completed on computer Performance is expected to match level of expertise. If not, why not? Assessment too high or too low? Employee not working to capability? Employee not following procedures? Employee needs to learn new skills due to equipment or process changes?
24	Observations of 7 Years BASA Experience Supervisor and employee agree on ratings or discuss why they don’t agree. Behaviorally anchored descriptions result in “show me” statements. If the employees can’t do what they claim, it is obvious. Assessment is based on major skill/knowledge categories—employees learn in some areas more easily than in other areas.
25	Salary Structures/Compensation Plan Salary levels link directly to Developmental Stages of Expertise Brings objectivity to the system Takes away the “arguing” about money Can justify larger & faster increases for higher performers and/or learners Can start “trainees” at a lower rate if company puts them on a faster review cycle
26	Capacity Planning The number of employees at each level in each position creates a mathematical “score” of production capacity Can help make or justify decisions regarding whether to improve equipment, train employees or both Capacity changes over time with additional learning
27	Benefits to the Employee & Supervisor Based on the employee’s current level of expertise, together they create future development plans. BASA can be used to address performance problems. If employees claim they are capable of performing at a certain level, they can be held accountable.
28	Benefits to the Company Developmental Stages can help a company understand total production capability and capacity based on number of employees at each stage. An individual’s compensation can be tied directly to level of expertise and performance so company is not over-paying or under-paying. BASA and Developmental Stages take the subjectivity out of compensation so the “argument” goes away.
29	Press Operator Dev Stage Cklst
30	Press Operator cont’d
31	Questions on the way to BASA Won’t people cheat? Don’t one page assessments leave a lot out? How I have an advantage in using BASA? If it’s so easy then how can it be believable? (Empirical ways do work: Navy dolphins and police dogs.) Your questions?
32	‘The Learning Model’ (Dots represent tested items/’points’)
33	The Understanding-Consolidation-Expertise Model (Dots represent anchored behaviors)
34	Creating Assessments First draft must be done by Expert in field Dev Stages of Expertise as template/guide Have BASA advisor present initially First draft can be done in a hour or two Should be fun creating this ‘seed’ document Give to other experts for their review and additions Discuss and resolve different views Document quickly converges and becomes robust
35	Experience of doing a BASA It has self-apparent structure It uses terms familiar to the field People know themselves--should be easy Effective even in rapidly changing technology, because BASA focuses on stable aspects
36	Creating an Assessment: Object Oriented Design/Programming First draft done by Expert in the field: Expert knows/decides: What are the key capabilities and understandings? For OOD/OOP: Improve development of highly complex systems Divide-and-conquer through small-grained class, object and method solutions Highly maintainable and reusable code High cohesion and low coupling
37	Creating an Assessment: Rubric Areas for OOD/OOP Expert selects subject areas for Rubrics: For OOD/OOP: OO understanding Coding practices and standards Language and IDE familiarity Refactoring expertise and code reuse Debugging / inspecting skills Design Patterns Unified Modeling Language
38	Creating an Assessment: Use Dev. Stages of Expertise as Guide Novice Not familiar with terminology or procedures Advanced Beginner Can sometimes do simple tasks but unpredictably Competent Effectively uses basic techniques and procedures Proficient Productive, does many techniques and procedures with automaticity Expert Highly productive due to experience and automaticity Master High degree of expertise and recognition, extends the discipline
39	Creating an Assessment: Rubric 1: OO understanding Novice Limited familiarity with OO terminology and concepts Advanced Beginner Basic understanding of classes, inheritance, instances and methods Competent Conversant in OO terminology Proficient Thorough understanding of OO terminology and all elements of their OO language Expert Able to evaluate pros & cons of different OO coding techniques (e.g. instance composition and delegation vs. class inheritance for behavior specialization)
40	Creating an Assessment: Rubric 2: Coding practices and standards Advanced Beginner Can edit class and method definitions, with unpredictable results Competent Can successfully but unevenly edit class and method definitions Codes with a focus on algorithms and logic Proficient Uses language idioms, best practices, and local standards Aware of impedance mismatches between OO and non-OO I/O sources Familiar with their local frameworks and architectural layering Codes with a focus on efficiency and “cleverness”
41	Creating an Assessment: Rubric 2: Coding practices and standards cont’d Expert Codes with a focus on maintainability; creates literate, readable code Designs frameworks and architectures, solves impedance mismatches Knows the Law of Demeter – don't reach through private implementations
42	Creating an Assessment: Rubric 3: Language and IDE familiarity Novice Limited familiarity with OO terminology and concepts Advanced Beginner Knows syntax of one OO language Comfortable with browsing classes and methods in their language’s IDE Competent Comfortable with the most frequently used classes in their OO language Proficient Comfortable with main path of their software configuration management (SCM) tools
43	Creating an Assessment: Rubric 3: Language and IDE familiarity cont’d Expert Full awareness of SCM life-cycle, alternatives, policies and procedures Master: Extends language, modifies IDE
44	Creating an Assessment: Rubric 4: Refactoring expertise, code reuse Advanced Beginner Uses cut and paste technique to reuse code Competent Reuses existing code if it is visible and convenient to do so Proficient Looks for existing code to reuse, refactors it if convenient Expert Refactors regularly as a standard coding practice Looks for application improvement through refactoring Always looks for existing code to reuse, refactors it if necessary, and eliminates other duplicate code Master Discovers and documents new refactorings, automates existing ones
45	Creating an Assessment: Rubric 5: Debugging / inspecting skills Advanced Beginner Can use debuggers and inspectors to observe snapshots of execution Competent Can observe execution behavior and values while stepping through Proficient Can use debugger breakpoints and inspector value editing to control and change runtime behavior Expert Regularly uses debuggers and inspectors to find and eliminate bugs by controlling and changing runtime behavior Master Extends and modifies debuggers / inspectors
46	Creating an Assessment: Rubric 6: Design Patterns Advanced Beginner Has basic understanding of the idea of patterns Competent Knows the commonly used patterns in their language/local environment Proficient Conversant in most common patterns, including those not currently used in their local language and environment Basic awareness of industry trends in patterns, methodologies, etc. Expert Uses patterns exactly when appropriate, able to evaluate pros & cons of one pattern over another in a given context for “goodness of fit” Master Discovers and documents new patterns
47	Creating an Assessment: Rubric 7: Unified Modeling Language Advanced Beginner Can follow explanations of some UML diagrams Competent Understands the most common UML diagrams Can create some UML diagrams without assistance Proficient Can create the most common UML diagrams, understands all UML diagrams Expert Effectively understands and creates all UML diagrams Master Authors UML or methodology practices
48	Why not wider usage of BASA? HR performance reviews are not a popular brainstorming area (totem poling legacy) Originated in academia, not in business BASA started with assessing learning, but assessing understanding is easier Requires long-term commitment for best results Leads to no more old boys’ clubs Intriguingly usage is growing!
49	What is the future of BASA? Emergent, distributed, online community growth Increased competition will require it Increasing need for predictability Online community can promote expert corroboration Online document mgt. will simplify assessment archival and authentication Real-time, embedded self-assessment likely in tutorials
50	Summary QAL provides a long duration example Practices becoming more common Approach has promise in software Long standing wage-expertise problem getting more attention
51	Capability Map a