3 Constraints & Limitations: The complete & correct information about the system and software required by STQC is not provided in time by the Implementation Agency (IA). The system and software is not thoroughly verified by client &/or developer for readiness before offering it to STQC. Project in general is already behind the schedule, at the time when STQC gets involved. Limited exposure of business domain & technologies used to STQC team. In order to resolve these issues an alternate approach is being suggested here so that STQC activities can be taken up on fast track mode without compromising the overall quality of work. The assessment take up by STQC using this approach will be based on combination of: a) Verification of testing process and procedures (including test planning, test case designing, test execution and test reports) of IA for compliance with testing standards. b) Demonstration of testing carried out by IA for compliance (coverage and depth of testing) with project requirements. c) Sample testing by STQC assessment team. It will provide the same level of confidence as give by independent third party testing. 1.4 Basis of Approach Suggested: The approach proposed will help reducing the time taken by STQC in familiarization and understanding the IT system and software. The primary assessment technique suggested is system and software audit (ISO/ IEC , clause Software Audit) although it will be combined with limited review (IEEE Std , IEEE Standard for Software Reviews) and testing (ISO/IEC (new) Software Engineering: Software Product Quality Requirements and Evaluation (SQuaRE) - Requirements for quality of Commercial Off-The- Shelf (COTS) software product and instructions for testing) techniques as well. The basis for adopting the approach is based on the Software Audit Process defined in the International standard ISO/IEC (IEEE Std ) - Systems and software engineering - Software life cycle processes. The purpose of the Software Audit Process is to independently determine compliance of selected products and processes with the requirements, plans and agreement, as appropriate. As a result of successful implementation of the Software Audit Process: a) An audit strategy is developed and implemented; b) Compliance of selected software work products and/or services or processes with requirements, plans and agreement is determined according to the audit strategy; c) Audits are conducted by an appropriate independent party; and d) Problems detected during an audit are identified and communicated to those responsible for corrective action, and resolution. Software audit is conducted to ensure that: a) As coded, software products (such as a software item) reflect the design documentation. b) The acceptance review and testing requirements prescribed by the documentation are adequate for the acceptance of the software products. c) Test data comply with the specification. d) Software products were successfully tested and meet their specifications. e) Test reports are correct and discrepancies between actual and expected results have been resolved. f) User documentation complies with standards as specified. g) Activities have been conducted according to applicable requirements, plans, and contract. In addition it is recommended that User Acceptance Testing (UAT) of the system/ software is also carried out by the end-users. 2

4 2.0 Purpose & Objectives: The primary purpose of carrying out Assessment & Evaluation (mainly through audit technique) of the system and software is to assure quality from user s viewpoint and provide confidence to users. The key objectives of the assessment by STQC are to verify: Compliance to Project Objectives & RFP/ Contract Requirements Fulfillment of quality characteristics Adherence to applicable regulations, standards and specifications set out in the RFP Timely identification & fixing of anomalies (defects/ nonconformities/ issues) in the system and software 3.0 Scope: The assessment will primarily cover system & software of the IT project. However, same approach may be used for assessment of other project components such as project documentation, processes, Software, Hardware, Network, SLAs, etc. 4.0 Responsibilities: The assessment team will be comprised of following members from STQC: Lead Assessor: Overall management the team Assist in team selection Preparation of audit programme/ checklist Control over the team s work Interfacing with the auditee management Preparation/ submission of audit report Audit conduction Assessor: Communicate audit requirements Be effective & efficient Document observations Report Results Verify corrective action effectiveness Remain within scope Support other team members Implementation Agency (IA): IA provides development, operation and maintenance services to project and will be responsible for followings: Making the required inputs available to STQC Ensure readiness of the system to be verified and timely provide the documentation and the required information to STQC. Provide access of the system to STQC Initiate timely action to fix the anomaly (defects/ nonconformities) reported Demonstrate the satisfactory closures of the reported anomaly (defects/ nonconformities) IA may himself be a client in some cases. Auditee (Client): The client of the system will support the STQC team in the following activities: Initiates the assessment process through a request. Act as interface between STQC & IA Provide domain expertise Provide support & assistance during assessment 3

5 5.0 Assessment Approach: The approach to be followed for assessment is essentially based on combination of review, testing & audit techniques, though primary focus on audit methodology. Assessment will Assessment will be performed to: Verify the documentation for adequacy of description & details as per RFP/ Compliance Criteria Verify Completeness, Correctness, Clarity & Consistency of system and software documents such as requirements and user documentation. Verify Traceability of documents to previous and later phases of the project. Check that the system/ software are in the state of readiness for the assessment by STQC (i.e., the system/ software is fully developed, internally verified and deployed). Also check that the system/ software is deployed on the required platform (hardware and software configurations). Verify that all the requirements are implemented as per specified requirements in the RFP/ Compliance Criteria Verify the internal QA activities of IA (various review/ test/ audit records of IA) Demonstration of execution of test cases by IA. Independent testing by STQC on business critical functions on sample basis. Verify Compliance with applicable Standards & Regulations Identify anomalies & ensure that they are addressed before system and software is deployed. Verify adequacy, implementation and effectiveness of defined practices/ processes as per applicable requirements & standards The assessment approach is described in detail below: Assessment - Steps: Assessment cycle will consist of following steps: A. Study & Preparation B. Planning C. Execution D. Reporting Corrective action will be taken by IA on the reported defects/ nonconformities E. Verification & closure of reported anomalies (defects/ Non-conformities) F. Final Report Inputs Required for Assessment: Assessment will required following minimum inputs: RFP/ Compliance Criteria Documents Requirements Document Traceability between RFP & Requirements Document Test Documentation including Test Plan, Test Cases and Test Reports Assessment - Activities & Tasks: Assessment consisting of following activities and tasks will be undertaken: A) Study & Preparation: Study & understand the project requirements from RFP Study requirements document & understand the system & software requirements. Study & understand system and software documentation (UM and related information), documentation structure and relationships. 4

6 B) Planning: Work out & finalize assessment methodology. Decide about assessment locations & environment in consultation with client/ IA. Appoint assessment team members Prepare assessment checklist & criteria/ audit objectives Prepare & communicate the assessment schedule C) Execution: Review the system and software documentation. Carryout traceability check (mapping) between requirements as given in the RFP and system and software document. Verify correctness, completeness, clarity, consistency, traceability & Document control of document. Record document review observations/ logs. Get the demonstration of the system and software from IA. Walk through and check whether the system and software is fully functional and is in state of readiness. Identify the critical business areas/ functions (high risk) in consultation with users and carryout detailed verification (through test cases of STQC). Generally these critical business functions should not be more that 15% of the overall software functionalities. Verify that system and software functions are in compliance with the specified functional requirements and identify defects/ issues (if any). Verify system and software documents, logs, reports & records Also verify compliance to applicable regulations, standards and specifications set out in the RFP. Collect information and analyze Record the assessment observations/ logs. D) Reporting: Analyze the assessment observations/ logs. Identify anomalies/ issues & assign severity/ risk. Prepare Anomaly (defect/ non-conformance) Report & submit to client/ IA. IA to initiate corrective action on the reported anomaly and submit Corrective Action Report to STQC. E) Verification & Closure: Verify corrective action (Resolution/ fixing) of the reported anomaly for closure from the Corrective Action Report. Carryout re-assessment of the system and software. Record the assessment observations/ logs. Prepare Final Report. Anomalies identified will be classified in terms of severity as follows: Anomaly Description Urgent The failure causes a system crash or unrecoverable data loss or jeopardizes personnel. Absence of the document, i.e., document not available (missing). Absence or total breakdown of process requirement. High The failure causes impairment of critical system functions and no workaround solution exists. Critical information/ contents of the document completely or partially missing/ wrong/ misleading. The defect/ deviation is given high attention to resolve on priority. Critical requirement is completely or partially missing/ wrong/ misleading. The defect/ deviation is given high attention to resolve on priority Medium The failure causes impairment of critical system functions, though a workaround solution does exist. 5

8 Annexure I Definitions and Explanations Software Audit: is software review in which one or more auditors who are not the members of the software development organization conduct An independent examination of software product, software process(s), or a combination of these to assess compliance with specifications, standards, contractual agreement or other criteria. Software Product: include technical document (IEEE std 1028 refers a list of 32 examples of software product subject to audit) like plans, contracts, specifications, designs, procedures, standards and reports, but also nondocumentary products such as data, test data, deliverable media etc. Establish: Mean to define, document, and implement. Requirements and Specifications: Requirements: Generally Quality System states that design input requirements must be documented, and that specified requirements must be verified. The distinction between the terms requirement and specification is not further clarified. Specification: A specification is defined as a document that states requirements. It may refer to or include drawings, patterns, or other relevant documents and usually indicates the means and the criteria whereby conformity with the requirement can be checked. There are many different kinds of written specification, e.g. system requirements specification, software requirement specification, software design specification, software test specification, software integration specification etc. All of these documents establish Specified requirements and are design outputs for which various forms of verification are necessary. Verification and Validation: Verification: provides objective evidence that the design outputs of a particular phase of the software development life cycle meet all of the specified requirements for that phase. It looks for consistency, completeness and correctness of the software and supporting documentation, as it is being developed and provides support for a subsequent conclusion that software is validated. Validation: it is a part of the design validation for a finished device confirmation by examination and provision of objective evidence that software specifications conform to user needs and intended uses, and that the particular requirements implemented through software can be consistently fulfilled. In practice software validation activities may occur both during as well as the end of the software development life cycle to ensure that all requirements have been fulfilled. 7

9 Annexure II Overview of ISO/IEC ISO/IEC25051: Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) - Requirements for quality of Commercial Off-The-Shelf (COTS) software product and instructions for testing. Scope(who can use this standard): o Certification bodies that may wish to establish a third-party certification scheme (international, regional or national) (ISO/IEC Guide 28); o Requirements for conformity assessment are: Clause 5.1 Product Description Clause 5.2 Requirements for user documentation Clause 5.3 Quality requirements for software Clause 6.0 Requirements for test documentation Clause 6. Requirements for test documentation: The test documentation shall contain: a) The test plan; b) The test description; c) Test procedures d) The tests results e) Anomaly Report Clause Software conformity evaluation: Conformity evaluation: Product description conformity evaluation User documentation conformity evaluation o For and no specific techniques or tools are recommended. o The Standard suggests review of the test documentation including the descriptive part for the anomalies found Clause 7.5 Conformity evaluation report o The COTS software product identification; o The date of evaluation completion and, if any, testing completion; o The computer systems used for testing (hardware, software, and their configuration o The documents used, with their identification; o The summary of conformity evaluation activities and, if any, testing activities; o The summary of conformity evaluation results and, if any, testing results; o The detailed results of conformity evaluation and, if any, testing; o The list of non-conformities to requirements if any. Clause 7 Software conformity evaluation: Clause 7.0 discusses the conformity assessment processes and gives two alternatives: 1. If the supplier provides only the COTS software product, without the test documentation, the thirdparty shall: a. Carry out a conformity evaluation of the product description, the user documentation, and the software according to subclause 7.3; b. Record the results in a conformity evaluation report, according to subclause If the supplier provides the COTS software product and the test documentation, the third-party shall: a. Carry out a conformity evaluation of the product description and the user documentation according to subclauses and 7.3.2; b. Carry out a conformity evaluation to determine the conformity of the test documentation to the requirements in Clause 6; c. Record the results in a conformity evaluation report, according to subclause

10 Software Nomenclature: (Name, Version) Procedure for Assessment of System and Software Template for Test Cases/ Scenarios and Test Log Annexure III Test Scenario & Test Cases: Test Scenarios: Test Scenario1: <Test Scenario Name> Test Scenario Description: Pre-conditions: Sr. Test Test Step Description Expected Output Actual Output Result No. Step Note: The test data for test steps of the test scenarios should be prepared in the following table. Comments: Test Cases & Test Data: Test Step: Sr. No Test Data (Valid/ Invalid) Field 1 Field 2 Test data - - Field n Actio n Expected Output Actual Output Result Comments: Note: Prepare the test scenario (as per table 3.1 above) describing the test steps for the test scenario using specified workflows, navigations & other relevant information for the software. For the test steps, wherever required, define the test cases along with test data (as per table 3.2 above) using the specified business logic/ rules, validation rules & other relevant information for the software. For test execution, start with test steps given in the test scenario, and use test cases with appropriate test data from the corresponding to the test case table. Test results as observed for each of the test step as well as test cases should be recorded the Actual Output field of the tables 3.1 & 3.2. In case of defect, the details of observations with complete details (to allow traceability & repeatability) should be recorded in the Actual Output field in the tables. Also screen shot may be captured and stored. The Actual Output should be recorded with adequate details so as to help to create test observations. From the test observations, defects/ problems/ issues should be identified and recorded in the test log below. 9

Frequently Asked Questions (FAQ) Guidelines for quality compliance of eprocurement System 1. What is eprocurement? Electronic Procurement (eprocurement) is the use of Information and Communication Technology

SOFTWARE DEVELOPMENT STANDARD FOR SPACECRAFT Mar 31, 2014 Japan Aerospace Exploration Agency This is an English translation of JERG-2-610. Whenever there is anything ambiguous in this document, the original

The document in this file is adapted from the IEEE standards for Software Project Management Plans, 1058-1998, which conforms to the requirements of ISO standard 12207 Software Life Cycle Processes. Tailor

(STP) Template Items that are intended to stay in as part of your document are in bold; explanatory comments are in italic text. Plain text is used where you might insert wording about your project. This

1. PURPOSE The purpose of this document is to detail the gaps between the superseded standard ISO 9001: 2008 and the current standard ISO 9001: 2015, to assist existing and potential clients assess their

Levels of Software Testing There are different levels during the process of Testing. In this chapter a brief description is provided about these levels. Levels of testing include the different methodologies

Aerospace Guidance Document Introduction AS9100, AS9110 and AS9120 all include ISO 9001:2008 registration and specify additional requirements for a quality management system for the aerospace industry.

Software Testing Rajat Kumar Bal Introduction In India itself, Software industry growth has been phenomenal. IT field has enormously grown in the past 50 years. IT industry in India is expected to touch

TOTAL QUALITY MANAGEMENT II Chapter 13: QUALITY AUDIT Dr. Shyamal Gomes Introduction: The term audit was defined in the 16th Century as the official examination of the accounts with verification by reference

Registrars Independent third parties who assess an organization s Quality Management System Not controlled by ISO US registrars have no government sanction Private companies performing their service for

ISO 9001 Registration Guidance Document Introduction This document is written to help you understand your organization s role and responsibilities in the registration/certification process and to understand

ISO/IEC 20000-1 Registration Guidance Document Introduction This document is written to help you understand your organization s role and responsibilities in the registration/certification process and to

Development, Acquisition, Implementation, and Maintenance of Application Systems Part of a series of notes to help Centers review their own Center internal management processes from the point of view of

INTRODUCTION What auditors should look for: the items listed in these headings that the ISO requirement is met that the requirement is met in the manner described in the organization's documentation Page

Implementing an Energy Management System Using ISO 50001 This article will address issues related to sustainability efforts, through energy management as it relates to ISO 50001, Energy Management System

Page 1 of 31 Audit Date Audit Description Lead Auditor Audit Team Members ISO 14001:2004 Auditable Clauses: (Tick those to be evaluated during this audit) 1. The checklist should be used by auditors to

Prevention is better than cure Quality Assurance - Karthik This maxim perfectly explains the difference between quality assurance and quality control. Quality Assurance is a set of processes that needs

TL 9000 and TS16949 Comparison www.questforum.org Copyright QuEST Forum 2007 1 Purpose This summary is intended to give those familiar with TS16949 requirements a general sense of the additional requirements

Application Functional Safety IEC 61511 Introduction Functional safety must be an integral part of the project execution if we shall succeed to make safe application program We can t test and audit safety

Central Page 284 of 296 Application of software product quality international standards through software development life cycle Mladen Hosni, Valentina Kirinić Faculty of Organization and Informatics University

GAP-ANALYSIS CROSS REFERENCE BETWEEN ISO/IEC 17020:2012 AND ISO/IEC 17020:1998 The principle standard used by International Accreditation Service (IAS) for the accreditation of Inspection Bodies is ISO/IEC

Client Overview Our client is the leading provider of health insurance related solutions for providing online and easy access to health insurance. Our client offers these services to a range of consumers

TM ComplianceSP TM on SharePoint Complete Document & Process Management for Life Sciences on SharePoint 2010 & 2013 Overview With increasing pressure on costs and margins across Life Sciences, the industry

Annex 7 Software Project Audit Process 1. Introduction 1.1 Purpose Purpose of this document is to describe the Software Project Audit Process which capable of capturing different different activities take

Acquire or develop application systems software Controls provide reasonable assurance that application and system software is acquired or developed that effectively supports financial reporting requirements.

4 Testing General and Automated Controls Learning Objectives To understand the reasons for testing; To have an idea about Audit Planning and Testing; To discuss testing critical control points; To learn

QA Roles and Responsibilities There are various roles on projects, some people may play more than one role. You should always check with your organizations testing methodology on what your role(s) are.

Generic CMMS Quality Assurance Plan Scope In accordance with the Quality Policy, the Quality System of CMMS is based upon the requirements and structure of ISO (the International Organization for Standardization)

FAT MEDIA QUALITY ASSURANCE STATEMENT NOTE 1: This is a CONTROLLED Document as are all quality system files on this server. Any documents appearing in paper form are not controlled and should be checked

Client information note Assessment process Management systems service outline Overview The accreditation requirements define that there are four elements to the assessment process: assessment of the system