Audit of the management and delivery of procurement: chapter 6

Annex 1 methodology, audit criteria and detailed testing results

Methodology

The audit was conducted in accordance with the Internal Auditing Standards of the Government of Canada, as supported by the results of the AEB's Quality Assurance and Improvement Program.

The audit was based on a combination of interviews, documentation review, data analysis and file examination for the testing of efficiency and compliance against TB Contracting Policy and ECCC's departmental guidance documents.

Audit methodologies included the following:

• Interviews: Key individuals from both the functional and program areas were interviewed to obtain a better understanding of the procurement and contracting processes and related contract files. Employees from Montreal, Ontario, National Capital and Edmonton regions were interviewed.
• Documentation review: Government of Canada procurement and contracting policies, statutes and regulations, such as the TB Contracting Policy and the Financial Administration Act, and ECCC's guidance documents, processes and other documents relevant to procurement and contracting were reviewed.
• Data analysis: Data analysis was used to identify potential anomalies, such as contract splitting, retroactive contracting and repetitive non-competitive contracts covering three fiscal years from 2011-2012 to 2013-2014. Data was extracted from ECCC's financial system.
• Sample selection: Procurement transactions in the period from April 1, 2013, to March 31, 2014, were selected based on a risk assessment and data analysis. For the purpose of this audit, a procurement transaction is defined as a procurement financial operation, including a purchase order, a contract, a call-up against a standing offer agreement or a supply arrangement, and any amendments. The sample included goods, services and construction contracts awarded using competitive sourcing methods, such as traditional competitive sourcing, standing offers, supply arrangements and advance contract award notices, and non-competitive sourcing. AEB selected the sample from the three branches having the highest volume of transactions (MSC, STB and CSB) and from the Communication Branch, because of the type of procurement transactions. The four branches represent a broad range of ECCC's activities, covering both program and support functions.
• Efficiency testing: Using the same procurement transactions selected for the compliance testing; the files were reviewed to determine the time it takes to process a requisition from start to finish based on key control dates. The intent was to compare the results against both the service standards in place at the time of the audit and current service standards.
• Compliance testing: The selected contract files were examined to determine compliance against key contracting policy requirements, including procurement initiation, contract bidding and award process, administration, approvals and close-out. For the verification of the approvals of the different procurement phases, a verification of the signature against the Specimen Signature Record was done to validate that the individual had the proper approval authority at the time of the transaction. For FAA sections 34 and 33, additional verification was done to ensure the payment was made to the right supplier, and that the amount paid matched the invoice amount and the amount recorded on the batch payment. If more than one payment was made, the testing included all those invoices issued within the audit period. For more details on the key requirements that were tested and the equivalent number of tests conducted, see Annex 1.
• Guidelines for Assessments against Criteria:
  The assessment of transaction audit results against audit criteria are partially a matter of professional judgement, informed by the number, significance and nature of the errors or issues identified (e.g., exceptional nature or conditions). With respect to the number of errors identified, the following is used as guidance for large sample sizes (minimum 50 items):
  • Met: errors equal/less than 4%
  • Partially met: errors between 4-8%
  • Not met: errors greater than 8%
• It is important to note that our conclusion on whether a criterion is met, partially met or not met is based on and applies to the sample examined.

Project key dates

Opening conference (launch memo)

April 2014

Audit plan approved

November 2015

Fieldwork completion

November 2015

External Audit Advisory Committee recommendation

June 2016

Deputy Minister Approval

November 2016

Note: The audit was initially launched in December 2013 and was subsequently postponed at FB's request to accommodate year-end priorities. In addition, AEB's unusual capacity shortfalls in 2014-15 contributed to the delays (including long-term medical leave and staff turnover).

Audit criteria

Objective 1 - Determine if key elements of a management control framework support efficient procurement activities aligned with departmental operational needs.

Objective 2 - Provide assurance that ECCC's procurement transactions were in compliance with GoC requirements and departmental and central agency policies and guidelines.

Detailed testing results

Note: The number of tests under the Contract Administration section exceeds the number of samples as there was more than one amendment made for some of the contracts sampled and all amendment made within the audit period were examined.