Using Story Points for estimation

• In agile software development, story points are units of relative size used in estimating requirements as an alternative to units of time
• You are trying to answer the question of size/complexity not duration first
• The second important property of points is that the units are relative, not absolute. They form a consistent measurement - the relative size/complexity of one requirement to the other does not shift as much as the effort/duration necessary to complete it (or, put differently, the size/complexity is a variable that changes less often and can be applied to different teams without the need to re-estimate)
• The discussion about the estimates is no longer on "why is this going to take you so long" but rather "why is this so complex"?

Visit: http://en.wikipedia.org/wiki/Story_points for more details.

Root Cause Analysis of Defects

Here is a list of potential (common) defect Causes which can be used to divide the root cause of a defect list:

• Unintended Side Effect – The bug is due to an unanticipated side effect from a change somewhere else in the code.  This category can be used when the programmer would say to himself/herself “I had no idea a change in that area would affect something over in this area”. 
  
• Logic / Simple Err – The bug is due to a simple programming error.  This could include bad logic in an if statement, a typo, an uninitialized variable, a missing break statement, not checking for a nill pointer, etc. 
  
• Overlooked Use Case – The bug is due to the software being used in some way that the programmer didn’t expect. 
  
• Code Path Not Done – The bug is due to a code path not being completely implemented.  This could be the result of incomplete work being checked in and tested.  It could be the result of forgetting to come back to fill in some code.  This cause should always be selected if an “Unimplemented” assert was found in the code. 
  
• Misunderstood Rqmt – The bug is due to a misunderstanding of the feature requirements. 
  
• Requirements Changed – The bug is due to a change in the feature requirements at some point while the software was being written.  One example could be a changed UI. 
  
• External Dependency Chg – The bug is due to a change in an external subsystem or application.  This could include API changes or semantic changes in a library. 
  
• Design Flaw – The bug is due to a design flaw in the original implementation. 
  
• Code Rot or Hack – The bug is due to code that has become convoluted in logic or implementation because it has had so many changes done to it.  Another example is a bug due to a partial fix for some other problem that was put in due to lack of time. 
  
• Code Misunderstanding – The bug is due to the programmer not fully understanding the true functionality of the code.  This could be due to the code being overly complex.  Another possibility is that the programmer didn’t fully understand a new architecture that they were helping to implement.  Or perhaps they “cut and pasted” several lines of code without understanding what it did.
  

Collection of Best Practices of Software Development

Best practice is an idea that asserts that there is a technique, method, process, activity, incentive or reward that is more effective at delivering a particular outcome than any other technique, method, process, etc. The idea is that with proper processes, checks, and testing, a desired outcome can be delivered with fewer problems and unforeseen complications. Best practices can also be defined as the most efficient (least amount of effort) and effective (best results) way of accomplishing a task, based on repeatable procedures that have proven themselves over time for large numbers of people. - Wikipedia

I am writing this post to document the learnings & best practices that have either worked for me in my past or I strongly believe that these would work in practical circumstances and help replicate demonstrated success.

Design and Development:

1. Good, detailed requirements (Functional and Non Functional) help writing better deisgn, code and better test cases.

2. Keep the Design Simple.

3. Perform peer reviews.

• Provide training to ensure that your team members share a common understanding of the review process. Adopt some written procedures, checklists and forms for documentation of formal reviews. 
• Keep time for code reviews in the project plan
• Only people who would contribute to the review should participate. Big review teams are often not as productive as a small team. Keep it 3 -7
• Give ample time to the reviwers to review individually before coming for the meeting
• Reviewers Critique the Product, not the Producer
  
• Focus on the substance not the style of the code being reviewed
  
• Review meetings should not drift into problem solving

4. Keeping a threshold for open bug assigned to all individual (developers) help curb the hassles of the End game of a product release.

5. Test the code before Check-in.

Configuration Management:

1. Continuous commit and Integration by all developers to the mainline with Self Testing code i.e. Unit Test cases. I have not worked on Test Driven Development(TDD) but having a strong unit test case framework is the least that is required for continuous integration. A continuous integration server can also be used which acts a monitor to the repository. Every time a commit against the repository finishes the server automatically checks out the sources onto the integration machine, initiates a build, and notifies the committer of the result of the build. Many organizations do regular builds on a timed schedule, such as every night. This is not the same thing as a continuous build and isn't enough for continuous integration. The whole point of continuous integration is to find problems as soon as you can. Nightly builds mean that bugs lie undetected for a whole day before anyone discovers them. Once they are in the system that long, it takes a long time to find and remove them.

TDD is a design technique that drives the development process through testing. In essence you follow three simple steps repeatedly: Write a test for the next bit of functionality you want to add; Write the functional code until the test passes; Refactor both new and old code to make it well structure.

2.  Maintain a single source repository for all developers (CVS, VSS, Subversion, Perforce etc.) and keep the branches to a minimum.

3. Automate the build process. Keeping it manual can be a a breeding ground for mistakes. A common mistake is not to include everything in the automated build. You can use a smart tool to check the dates of the source and object files and only compile the module if the source date is later.

4. Daily Build and Smoke Test helps keeping the build stable.

5. A strong BVT (Build Verification Test) before build release to QE, can help save a lot of effort later. Usage of Virtualized machines can reduce the cost of your test labs drastically.
If you have build problems quite common, put a calendar on the wall that shows a full year with a small square for each day. Every day the QA group would put a green sticker on the day if they had received one stable build that passed the commit tests, otherwise a red square.

6. Keep a standard place/server location to keep the build for usage. It makes it easy to find and backup.

Testing:

1. Use functional requirements to write test cases and test design at the same time that the developer is performing the low level design and coding. This will help in refinement of the requirements and hence the implementation before the build is provided for testing.

2. Have formal Entry and Exit Criteria for SDLC phases and milestones.

3. Understand and implement in your test design how to write variations (different input, expected output) of a test to gain code coverage.

4. Create a sound plan for multiple platform testing by periodic rotation of the platforms being tested.

5. Plan Internal Beta testing leveraging on existing resources of your company. Dependent on the product, you may use the product within your organization at a production level.

6. Plan External Beta or Prerelease programs.

7. Develop User Scenarios where exhaustive testing is not possible. It also helps in reducing the complexities of the test cases.

8. Pay attention on usalibity, it was be the first factor for a user to define Quality. See what is the latest happening in Human Computer Interface.

9.   Teaming testers with Developers. Form the right team where the skills of various team members compliment each other.

10. Use Code coverage to see the efficiency of the test cases and minimizing regression tests.

11.  Automated Environment Generator: Go in for automated deployment and rollback of the build to the QA machines or servers.

12. Instrument the code for MTTF: You can implement a reporting back of failures and failure data which can help in diagnosing the problem.

13. Get the test plans reviewed by peers before getting it reviewd by the Product Management and Development teams.

14. Do not automate without optimizing the test cases eg. for functional and code coverage. Automating Junk helps creating junk faster.

15. It is important to analyze the results (predominately the Failures) of an automation run as soon as the results are provided.

16. Replication of user environment would help in replication of non-reproducible defects in house.

17. Usage of Virtualized machines can reduce the cost of your test labs drastically.

 

Project Management:

1. Do not create reports that are not required.

2. Perform post mortems and document the learnings while they are fresh.

3. A Plan is only as good as its implementation.

4. If you fail to plan, you plan to fail.

Further Reading:

http://www.ibm.com/developerworks/websphere/library/techarticles/0306_perks/perks2.html

http://www.construx.com/

http://www.sdbestpractices.com/

http://www.martinfowler.com/articles/continuousIntegration.html (ThoughtWorks)

http://www.processimpact.com/articles/revu_sins.html

http://www.belizenorth.com/articles/TIMEBOX.htm

http://www.spmn.com/lessons.html

http://www.ddj.com/architect/184415753

http://www.perforce.com/perforce/bestpractices.html

http://testing.com/

http://www.chillarege.com/authwork/TestingBestPractice.pdf

2 min Refresher for SCRUM - An agile development process

Please see the link http://www.controlchaos.com/about for a more detailed insight.

Scrum is a wrapper for existing engineering practices:

• Product Backlog > Series of Sprints > Sprint Planning meeting > Sprint Goals > Sprint Backlog > Daily Scrum meeting > Sprint Review meeting
  
• Roles: Product Owner, Scrum Master, Scrum Team
  
• Product Owner: Prioritize Product Backlog, provides and clarifies requirements
• Scrum Master: Responsible for the success of the scrum, establishes Scrum rules, shields the team from obstacles
• Scrum Team (PIGS - committed): Is cross functional (7±2). Can add to the product backlog, expands and self organize sprint goal and backlog to sprint tasks, can add or remove tasks, update estimates
• Observers (Chickens): People who are involved but not dedicated to the project 
• Sprint (normally 30 days) produces a potentially shippable set of functionality. No changes in sprint goal allowed in a sprint, unless the sprint goal is not making any sense. Stabalization sprints can be used for brining the quality to the required level or nearing the release. 
• Daily Scrum 15 minutes of scychronization not for problem solving - What have you done since the last Scrum? What will you do between now and the next Scrum? What got in you way of doing work? - Gives a clear status to all, creates a peer pressure for delivering on time.
• Sprint Review: The team (each team member) presents to management, customers, users and the Product Owner, the product increment built during the Sprint. Powerpoint presentations are forbidden!
• Scrums of Scrums is called Meta-Scrum

Difference between Iterative and Incremental Development:

Remember, you build a house iteratively and a colony incrementaly.

Alistair Cockburn Link

Sticky Minds Link

Qualities that you should look in a Software Tester

• They are explorers: Software testers aren't afraid to venture into unknown situations. They love to get a new piece of software, install it on their PC, and see what happens.
  
• They are troubleshooters: Software testers are good at figuring out why something doesn't work. They love puzzles.
  
• They are relentless: Software testers keep trying. They may see a bug that quickly vanishes or is difficult to re-create. Rather than dismiss it as a fluke, they will try every way possible to find it.
  
• They are creative: Testing the obvious isn't sufficient for software testers. Their job is to think up creative and even off-the-wall approaches to find bugs.
  
• They are (mellowed) perfectionists: They strive for perfection, but they know when it becomes unattainable and they're okay with getting as close as they can.
  
• They exercise good judgment: Software testers need to make decisions about what they will test, how long it will take, and if the problem they're looking at is really a bug.
  
• They are tactful and diplomatic: Software testers are always the bearers of bad news. They have to tell the programmers that their baby is ugly. Good software testers know how to do so tactfully and professionally and know how to work with programmers who aren't always tactful and diplomatic.
  
• They are persuasive: Bugs that testers find won't always be viewed as severe enough to be fixed. Testers need to be good at making their points clear, demonstrating why the bug does indeed need to be fixed, and following through on making it happen.
  

Tools to explore for your project

Tools to explore for your project

S. No.	Name	Type	Company	URL	Platforms	More...
1	Quick Test Professional	Automation	Mercury/HP	http://www.hp.com/	Windows only	Suitable for Web applications
2	Quality Center/Test Director	Test Case Management	Mercury/HP	http://www.hp.com/	Windows(TD), Linux and Solaris	Also for Defect Management
3	Rational Clear Quest	Software Change Management	Rational/IBM	http://www-01.ibm.com/software/awdtools/clearquest/index.html	-	-
4	Rational TestManager	Test Case Management	Rational/IBM	http://www-01.ibm.com/software/awdtools/test/manager/	-	-
5	Rational Purify	Memory Leaks and corruption	Rational/IBM	http://www-01.ibm.com/software/rational/offerings/quality/	Win, Linux, Unix	-
6	Rational Functional Tester	Automation	Rational/IBM	http://www-01.ibm.com/software/awdtools/tester/functional/index.html	-	-
7	-	-	-		-	-
8	Silk Central	Test Case Management	Seque	-	-	-
9	Open Source Tools – Bugzila, Jira etc.	Test Case Management	-	-	-	-
10	Bounds Checker	Memory Leaks	-	-	-	-
11	Consume Memory	Memory Blocker	-	-	-	-
12	BullsEye	Code Coverage	-	-	-	-
13	Eat Disk	Hard Disk Consumption	-	-	-	-
14	J Meter	Load Testing	-	-	-	-
15	Web Application Stress	Load Testing	Microsoft	http://www.microsoft.com/	-	-
16	Fortify 360	Security		http://www.fortify.com	-	-
17	iMacros	Automation - Firefox only	iOpus	http://www.iopus.com/imacros/firefox/	Win, Mac, Linux	-

Software Certifications for Quality/Testing

ISTQB:

The ISTQB was officially founded as an International Software Testing Qualifications Board in Edinburgh in November 2002. The ISTQB is responsible for the international qualification scheme called "ISTQB Certified Tester". http://www.istqb.org/

---------------------------------------------------------------------------------------

Software Dimensions Consulting and Training, Inc.: (http://www.softdim.com/)

is an international company founded in 1996 with the mission to promote disciplined software engineering practices. Software Dimensions achieves this goal through two educational institutions that operate under its charter. These are The International Institute for Software Testing (IIST) and The International Institute for Software Process (IISP).

The International Institute for Software Testing - IIST (http://www.testinginstitute.com/ctm.php):

is an educational and professional development organization that has been founded to advance the software test and quality assurance profession by promoting and recognizing professionalism through education, consulting, publications, conferences, and certification.

CSTP: The International Institute for Software Testing (IIST) has been offering the Certified Software Test Professional (CSTP) certification since 1999.

CTM: The International Institute for Software Testing (IIST) has been offering the Certified Software Test Professional (CSTP) certification since 1999. Currently there are thousands of people at different stages in the CSTP program. Although CSTP has been serving the purpose of establishing a foundation of software testing and providing test professionals with the skill and knowledge necessary to perform different test activities, a gap still exists in the management skills required by test managers and test leads to effectively manage the test process, the test project and the test organization. The Certified Test Manager (CTM) certification has been created to fill this gap. CTM is based on the Test Management Body of Knowledge (TMBOK) developed by IIST through its Advisory Board.

---------------------------------------------------------------------------------------

ITIL (http://www.itilcertification.org/; http://www.itilcertification.net/index.html, http://www.itil-officialsite.com/):

---------------------------------------------------------------------------------------

QAI:

There are 9 certifications given by QAI in Quality/Testing field. Please see the latest information and details on http://www.softwarecertifications.org/

1. Certified Associate in Software Quality (CASQ)
2. Certified Software Quality Analyst (CSQA)
3. Certified Manager of Software Quality (CMSQ) - Should have an active CSQA Certification.

4. Certified Associate in Software Testing (CAST)
5. Certified Software Tester (CSTE)
6. Certified Manager of Software Testing (CMST) - Should have an active CSTE Certification

7. Certified Software Process Engineer (CSPE)
8. Certified Quantitative Software Process Engineer (CQSPE)
9. Certified Software Project Manager (CSPM)

Template for a Software Master Test Plan

Following can be included into a Master Test plan:
0. TOC
1. Stakeholder/Important Project Contacts
2. Reference Documents
3. Acronyms & Terminology
4. Project Brief including major components
5. Project Quality Goals and deliverable (Includes Non Goals)
6. Testing Strategy (Project and Component level)
7. Quality Metrics and status reporting
8. Resources (People, Hardware, Software, Vendors)
9. Responsibility Division (Till sub team/Team member level)
10. Schedule (Project, Milestone and Testing)
11. Quality Entry and Exit Criteria for milestones
12. Testing and reporting frameworks
13. Dependencies, Risks and Mitigation
SIGN OFF
Appendixes

Risks Associated with Software Testing

Based on the project and time the Risk quoted low here can become Medium to High viceversa:
Some High to Medium Risks:
1. Changing Requirements ~ Can translate into changed test cases / environment / scripts / automation.
2. Schedule Slippage ~ Testing time shrinks if development time increases.
3. Breaks in Testing Framework ~ Reliance on a low quality framework can have drastic impact on the testing schedules.
4. Relying on a New Tool in the process

Some Medium to Low Risks:
1. Attrition: Where project is not process driven, people leave taking away the knowledge as well.
2. 3rd Party component usage in the development may make tester miss useful insight of implementation i.e. capability of Gray Box testing.
3. Transition of roles and responsibilities across teams/people
4. Delivery from Vendors

Software Metrics and what do they say

Software metrics deals with the measurement of the software product and the process by which it is developed. Metrics can be used to improve software productivity and quality.

Software metrics may be broadly classified as either product metrics (measures of the software product) or process metrics(measures of the software development process).

Metrics can also be classified as Objective (should always result in identical values for a given metric, as measured by two observers) and Subjective metrics.

Metrics can also be classified as Primitive (directly observed), or computed (computed from other metrics)

Product Metrics:

1. Size Metrics:

• LOC (Lines of code) - Can be interpreted differently based on blank lines and comment lines, non-executable statements, multiple statements per line, and multiple lines per statement. The most common definition of LOC seems to count any line that is not a blank or comment line, regardless of the number of statements per line.
  
• Function Points (Albrecht) - The value of function points is the total of the weighted values: inputs: 4, outputs: 5, inquiries: 4, and master files: 10.

2. Complexity Metrics:

• Cyclomatic Complexity - v(G) = e − n + 2, where e is the number of edges, and n is the number of nodes in the control flow graph wherein each node corresponds to a block of sequential code and each arc corresponds to a branch or decision point in the program.
• Knots - Drawing transfer-of-control lines from statement to statement in a program listing.

3. Quality Metrics:

One can generate long lists of quality characteristics for software—correctness, efficiency, portability, maintainability, reliability, etc. Unfortunately, the characteristics may overlap and conflict with one another; for example, increased portability (desirable) may result in lowered efficiency(Undesired).

• Defect Metrics - number of design changes; number of errors detected by code inspections; number of errors detected in program test; number of code changes required
• Reliability Metrics - mean time to failure (MTTF).

4. Effort Measurement Metrics:

• Testing: Effort spent in a. Requirement Analysis, b. Testware creation - Test Strategy, Test specifications, Checklists etc., c. Training and knowledge sharing, d. Test Automation, e. Test environment setup, f. Userstory testing, g. System Testing, h. Bug Validation
• Development: a. Requirement Analysis, b. Architecture & Design (HLD&LLD), c. Coding, d. Unit Testing, e. Integration, f. Bug fixing

Here are some practical standard Defect Metrics that I use:
BUG TRENDS
1. Find vs Fix Ratio: If plotted with a differnt curve for each priority can help let you know the progress towards release. High priority defects should reduce and lower priority defects should increase when the application is getting stabler. You may do it for individual components or modules to access the stablity for components.
2. Average Age of Defects: If plotted with a differnt curve for each priority can help let you know the right selection of defects by the development teams. You may do it for individual components or modules.
3. Open Defects by Severity/Priority/Developer: Tells about the current status of the project and can be exptrapolated (when project is plainly into bug fixing i.e. no further new implementation) to see the projection for meeting the schedule or schedule variance. Also helps you fix deferal targets.
4. Closed defects by Severity/Priority/Area: In a process driven company, it tells you about the quality of a certain area/project delivery. Detailed analysis can help in accessing areas of improvement upstream.

OTHERS
5. Build Breaks: Is indicative of the quality of the build and BVT process.
6. Test Code Coverage: Is indicative of the quality of the test cases. Can be helpful in acertaining the final quality of coverage using Automation.
7. Percentage Automation & percentage defects found by Automation: Maturity of Automation within the project/module

8. Defects by injection phase: Indicator of most problematic phase in software development - Requirement analysis, design, coding, documentation, system test, Acceptance Testing

9. Defects by detection phase: Indicator of most efficient phase in software development, in terms of finding and removing bugs - Requirement analysis, test planning, feature testing, System Testing, Acceptance testing.