Vendor Performance Program: Q & A

The Vendor Performance Program is IO's program to monitor, track and inform vendors about their recent performance on IO projects. Vendors will be given a performance score whereby their performance will be assessed against specific contract requirements.

This performance score, together with a report of performance infractions, will be updated on a monthly basis and disclosed to each vendor. This information will be commercially confidential and only provided to vendors.

Vendor Performance Program scores are based on disclosed, clear and measurable contract criteria. Vendors should not be surprised by their rating.

The score will inform each vendor of the evaluation deductions, if any, that will be applied, if such vendor participates in an upcoming procurement.

The Program's main objectives are to ensure value for money and protection of the public interest. IO expects vendors working on projects for our clients to perform at their best.

Vendors who perform well should continue to strive for the highest standards. Vendors who perform below their contractual requirements will want to consider how their past performance could be a factor in future procurements.

The program will track the performance of vendors falling within three main service categories:

• Construction contractors on P3 projects;
• General contractors providing general contracting services to either IO or its Project Service Management Providers (PMSP), operating on behalf of IO;
• Architects, interior designers, engineers, and security consultants procured by IO's PMSPs, operating on behalf of IO; and
• Service providers on P3 projects, provided:
  • (a) The project in question relates to a social or vertical asset, and
  • (b) Both IO and, in cases where IO is not the Contracting Authority, the Contracting Authority on the project elect to apply the Program to the project.

As part of its due diligence in procurement and project management, IO collects performance information on vendors for all ongoing projects within a specified time period - twenty-four (24) months in the case of construction contractors and general contractors procured directly by IO as well as all vendors procured by IO's PMSPs, operating on behalf of IO, and thirty-six (36) months in the case of facilities maintenance service providers procured directly by IO.

This information is based on clear, precise, and legitimate sets of criteria that are monitored as part of IO's regular role in contract management.

Each vendor's score will be automatically calculated based on the number of times a performance infraction (as described by the pre-established criteria) has occurred and been recorded within the aforementioned time periods.

The larger the number of performance infractions, the greater the deduction that will be applied against that vendor's next submission.

The following lists are a summary of the criteria that IO will use to monitor vendor performance. Each criterion is linked to a clear provision (or set of provisions) within the contract or, in the case of P3 construction contractors, project agreement. Detailed descriptions of these criteria, together with applicable interpretation notes, are disclosed within the program appendices as well as in the sample monthly reports.

Construction Contractors

1. The Contracting Authority has exercised remedial rights due to a Vendor breach of Contract.
2. The Contracting Authority has sought indemnification under the Contract due to a Vendor breach of Contract.
3. The Contracting Authority has made a direct claim for, or set off of, proceeds from payments due as a result of a Vendor breach of Contract.
4. The Contracting Authority has issued a notification to the Vendor stating that the Work has been performed in a manner that will result in the Vendor being unable to meet completion requirements.
5. The Contacting Authority has exercised its right to uncover work and said work has been found to be defective.
6. The Contracting Authority has remedied defective work attributable to the Vendor.
7. Pursuant to its legislative or regulatory power, a Governmental Authority has issued an order to the Vendor which has resulted in a follow-on notice of violation, fine, or other disciplinary action against the Vendor.
8. The Contracting Authority has drawn upon deposited or withheld amounts in respect of the rectification of Minor Deficiencies.
9. The Vendor's performance security has been enforced against and/or drawn upon.
10. The Contracting Authority has requested a rectification plan and the Vendor has failed to deliver said plan in a timely manner.
11. The Contracting Authority has requested a rectification plan and the plan delivered by the Vendor fails to meet the prescribed requirements of such plan.
12. The Vendor has failed to deliver the Project in a timely manner.
13. The Project was delivered later than that date set out by the first Substantial Completion Countdown Notice.
14. Vendor personnel were not made available to perform their agreed upon role.
15. Vendor personnel were substituted without obtaining prior written consent of the Contracting Authority.
16. Recommendations to rectify health and safety infractions have not been implemented or addressed in a timely manner.

General Contractors

1. The Contracting Authority has made a direct claim for, or set off of, proceeds from payments due as a result of a Vendor breach of Contract.
2. The Contracting Authority has issued a notification to the Vendor stating that the Work has been performed in a manner that will result in the Vendor being unable to meet completion requirements.
3. The Contacting Authority has exercised its right to uncover work and said work has been found to be defective.
4. Pursuant to its legislative or regulatory power, a Governmental Authority has issued an order to the Vendor which has resulted in a follow-on notice of violation, fine, or other disciplinary action against the Vendor.
5. The Contracting Authority has drawn upon deposited or withheld amounts in respect of the rectification of Minor Deficiencies.
6. The Vendor's performance security has been enforced against and/or drawn upon.
7. The Contracting Authority has requested a rectification plan and the Vendor has failed to deliver said plan in a timely manner.
8. The Contracting Authority has requested a rectification plan and the plan delivered by the Vendor fails to meet the prescribed requirements of such plan.
9. The Vendor has failed to deliver the Project in a timely manner.
10. Vendor personnel were not made available to perform their agreed upon role.
11. Vendor personnel were substituted without obtaining prior written consent of the Contracting Authority.
12. Recommendations to rectify health and safety infractions have not been implemented or addressed in a timely manner.
13. The Vendor has failed to notify the Contracting Authority of critical discoveries in a timely manner.
14. The Vendor has been notified of necessary work within the Warranty period and failed to complete such work.
15. The Vendor has failed to notify the Contracting Authority of property damage or bodily injury within a timely manner.
16. The Vendor has failed to deliver closeout documents in a timely manner.
17. The Vendor has failed to adhere to the security protocols necessary to perform the work.
18. The Vendor failed to deliver a baseline schedule within a timely manner.

Architects, Engineers, Interior Designers, and Security Consultants

1. The Vendor has failed to report matters of a materially significant nature.
2. Vendor personnel were not made available to perform their agreed upon role.
3. Vendor personnel were substituted without obtaining prior written consent of the Contracting Authority
4. The Vendor has failed to provide Record Drawings in a timely manner.
5. The Contracting Authority has made a direct claim for, or set off of, proceeds from payments due as a result of a Vendor breach of Contract
6. The Vendor has failed to adhere to the security protocols necessary to perform the work.
7. The Vendor has failed to provide Contract deliverables in a timely manner.
8. The Contracting Authority has rejected Contract deliverables on qualitative grounds.
9. The Vendor has spent cash allowances prior to the Contracting Authority's approval.
10. The Vendor has undertaken Additional Services and/or revised the required services prior to the Contracting Authority's approval.

Service Providers

1. Contracting Authority has exercised a Remedial Right due to a Vendor breach of Contract.
2. Contracting Authority has sought indemnification under the Contract due to in excess of three (3) Vendor breaches of Contract within the previous twelve (12) months or has sought indemnification in excess of $250,000 for any one such breach.
3. The Vendor has accumulated the number of Failure Points needed to warrant the issuance of a Warning Notice in any one-month period.
4. The Vendor has accumulated the number of Failure Points needed to warrant the issuance of a Monitoring Notice in any three-month period.
5. Vendor personnel were substituted without obtaining prior written consent of Contracting Authority.
6. Contracting Authority has conducted a Performance Audit pursuant to the Contract and has determined the Vendor has not performed its obligations in any material respect.
7. The Vendor has failed to perform rectifications or Maintenance Work identified by a Performance Audit.
8. The Vendor has performed rectifications or Maintenance Work identified by a Performance Audit contrary to its obligations associated with the Performance Audit.
9. The Vendor has been issued a direction pursuant to the Contract and its response to such direction does not comply with Contracting Authority's requirements, or the Vendor has been issued a direction by a Governmental Authority related to labour, employment and/or human resources matters, and fails to comply with the Governmental Authority's requirements.
10. The Vendor has failed to comply with a material reporting requirement.

Recorded instances of infractions are calibrated to a specified percentage point deduction from the vendor's technical submission. Deductions are calibrated based on the objective of addressing performance in ways appropriate to the procurement method as outlined below;

IO Request for Qualifications

Deductions calibrated for use in IO RFQs will address performance in two ways:

1. Determining the threshold number of infractions that a vendor has accumulated, at any given time, before that vendor is statistically more likely of moving from the last qualifying position to the first non-qualifying one; and either
2. Determining the threshold number of infractions that a Construction Contractor has accumulated, at any given time, before that vendor is statistically more likely of failing to achieve the 60% minimum score threshold required of the Construction Team Member Capability and Experience section, or its equivalent, or
3. 3. Determining the threshold number of infractions that a Service Provider has accumulated, at any given time, before that vendor is statistically more likely of failing to achieve the 60% minimum score threshold required of the Facilities Management Team Member Capability and Experience section, or its equivalent.

PMSP Procurements

Deductions calibrated for use in PMSP procurements will address performance by determining the threshold number of infractions that a vendor has accumulated, at any given time, before that vendor is statistically more likely of failing to achieve the 60% minimum score threshold required of the technical submission, or its equivalent.

The Vendor Performance Program sends a signal to the marketplace that performance on projects matters. The Program is designed so that it would require a vendor to have a series of significant infractions over a sustained period of time in order to seriously impact their standing in future procurements.

For most vendors and most procurements, deductions will not have a significant impact on the outcome. IO has done statistical analysis of past projects and can provide the following insight on the consequence of vendor performance that needs improvement.

IO Request for Qualifications

PMSP Procurements

In order for a vendor's infraction record to cause it to outright fail to reach the minimum scoring threshold of a PMSP procurement, a vendor must accumulate at least eight infractions in the 24-month period before the procurement's submission deadline.

The infraction thresholds were selected by IO taking into consideration the materiality of the infraction criteria and the observed frequency of such infractions in the history of past projects.

It would require repetitive material non-performance by any given vendor in order for their success to be in jeopardy.

The program infraction criteria and thresholds for infraction accumulation as well as the criteria themselves will be reviewed on a biannual basis, with any resulting changes being incorporated into the program in January or July.

The program infraction criteria have been developed to address a variety of performance issues that have been observed on past projects. While some events may have different commercial, reputational, or programmatic risks, performance in all instances is important to IO. It is important to avoid the unintended consequences of construction contractors not taking seriously certain infractions because of a lower deduction value.

For most vendors, deductions will not have a significant impact on success at procurements.

The point deduction calibrations, which have been developed using a data-driven statistical model, if applied to past procurements, would not have had a significant impact in most instances.

For example, as of November 2016, out of the 43 P3 RFQs that closed and were evaluated since 2008 (9 BF, 13 DBF, 15 DBFM Social and 6 DBFM Civil):

52 applicant teams (21.5% of total applicants) would have had a change in RFQ ranking, but only in 3 out of those 52 instances would the change in ranking have caused a change in the prequalified party shortlist; and

in only two instances (of 242 total applicants) would a team have failed to meet the minimum scoring threshold of 60% required for the Construction Team Member Capability and Experience section. Even in these two instances, the team that would have failed was already not in a prequalifying position.

All aspects of the program, including infraction criteria, thresholds for infraction accumulation, and scoring calibrations will be reviewed twice a year. Updates flowing from these reviews will be implemented in January and July.

IO's Procurement Department will administer the program.

Administration of reporting during construction and maintenance phases:

The occurrence of any infractions will be reported and reviewed as part of Infrastructure Ontario's regular internal monthly reporting of performance against the defined infraction criteria. Senior management at Infrastructure Ontario, including representatives of the project delivery, legal and procurement departments, will conduct due diligence to ensure that reporting is accurate and sufficient documentary detail exists to substantiate all performance infractions.

Infraction totals will then be updated by IO's Procurement department and communicated to vendors via Monthly Reports. The reports are commercially confidential. Vendors, however, may determine for themselves whether to share their reports and with whom.

Administration of applicable deductions during the procurement process:

A deduction to be applied against a vendor in any given procurement will be the deduction set out for that vendor in the month in which the procurement submission is received.

During the procurement evaluation process, vendor performance point deductions will be applied to a bidding team's score only after the scoring evaluators have reached consensus on their evaluation of submissions. This is a safeguard to ensure that there are proper checks and balances during evaluation and that the evaluation process continues to be fair.

Infrastructure Ontario will track and record the performance criteria stated above for those projects in which Infrastructure Ontario is defined as a co-Sponsor or Owner under the project procurement and those for which Infrastructure Ontario has an official governance role.

Projects in which Infrastructure Ontario does not have a lead procurement or contract management role will not apply to the dataset except for cases in which IO and Contracting Authority have agreed to track the performance of Service Providers within the maintenance phase of a P3 project.

No infraction criteria will be tracked for the purposes of the Vendor Performance Program on any other projects.

Vendors who have not been involved on an IO project will not have a score until they accrue an infraction through participation in an IO project. IO will not record infractions on projects more than 24 months after their completion in the case of construction contractors and general contractors procured by IO as well as vendors procured on IO’s behalf by its Project Management Service Providers. In the case of facilities maintenance service providers directly procured by IO, IO will not record infractions on assets more than 36 months after the end of their maintenance phase.

IO recognizes that joint venturing is a successful and important team approach to undertaking some projects, particularly P3 projects. From a performance perspective, joint ventures ought to share all risk and responsibilities among their constituent entities. This means that in monitoring performance, any instances of poor performance infractions will be attributed in full to all members of a joint venture.

During the bidding phase, when firms are forming joint ventures, the number of infractions attributable to a bidding joint venture will be derived from the arithmetic average of the Vendor Performance Program scores that each constituent member of the joint venture has at the time of the RFQ submission, based on the proportion of the joint venture interest.

For example, if as of January 1, 2017, Contractor ABC's point deduction for Design-Build-Finance RFQs is minus 1.75% (based on having accumulated two infractions in the last 24 months with a further 50% reduction during the 2017 phase in period) and Contractor XYZ does not have any record of infractions (and thus a 0% deduction), and Contractor ABC and XYZ are in a 50/50 joint venture bidding to an RFQ, then for any prequalification submission received up until February 28, 2017 in which Contractor ABC and Contractor XYZ are acting as 50/50 joint venture partners, the total deduction applied to the score of the joint venture will be minus 0.875% of the Construction Team Member Capability and Experience available points, which will be automatically subtracted from that section following the completion of individual and consensus evaluations.

While vendors may form joint ventures many months in advance of a procurement being issued, the Vender Performance Program score of any given vendor is only taken into account for bidding purposes on the date of the submission. IO and its partners are not privy to the timing of formation of joint ventures, and the most fair and transparent means of applying a vendor performance score to a joint venture is at the time of submission.

The respective members of a joint venture will continue to receive their own monthly reports, and can use this information to determine their approach to a submission accordingly.

Once a submission has been submitted, then it will not matter, for evaluation or scoring purposes in the procurement, whether one or more of those vendors' scores have changed since the date of submission. Any subsequent change in a vendor's score will not have a further impact for that particular joint venture in that specific procurement.

The Vendor Performance Program is intended to give a clear signal to the market that IO is serious about vendors performing well and that there are consequences for persistent and material poor performance. Vendors should not be surprised by any of the information they receive. A vendor's future success is an incentive for sub-par performers to improve.

All bidders are on a level playing field. The types of infractions described by the performance criteria above are very serious and will only occur in a contract when a vendor has engaged in persistent and continuous sub-standard performance or behaviour. There will not be any ability to dispute or appeal whether an infraction described by any performance criterion has occurred.

If there is an administrative error in the tracking of infractions we will correct our information. IO will ensure quality control of the data and processes that result in the scores.

Consideration of past performance on projects in a procurement is very different from the determination of where legal liability is apportioned under a contract. The Vendor Performance Program does not take away any of a contractor's rights to dispute a specific event under a contract or project agreement. Rather, the program is designed to capture the series of behaviours that lead to any given infraction being registered.

For example, if the Contracting Authority has drawn upon the contractor's performance security (e.g. a letter of credit), a performance infraction would be recorded against such contractor. That contractor may choose to dispute whether the Contracting Authority was legally entitled to draw upon the performance security in the first place - and may in fact win such legal challenge. Even in this situation, the P3 Vendor Performance Program would continue to register an infraction on the account that significant performance failures would have necessarily led up to the Contracting Authority needing to seek recourse on a project issue by drawing on performance security in the first instance. It is these types of performance failures which lead up to the occurrence of an infraction event which IO is seeking to discourage and eliminate through the implementation of the program.

No, the Vendor Performance Program infraction criteria capture not only the infraction events themselves, but the series of behaviours that lead to any given infraction being triggered. This is done without taking away any of a vendor's rights to dispute matters under a contract or project agreement.

Contracts and project agreements address specific incidents based on their provisions. The Vendor Performance Program infraction criteria measure not only whether an event occurred but also capture the series of behaviours over a period of time that led up to such event.

More importantly, the Vendor Performance Program scoring adjustments relate to a vendor's qualifications, experience and capability to perform work as assessed in procurement. Consideration of these occurrences of significant poor performance events in a procurement is very different from the determination of where legal liability is apportioned under a contract.

The infraction criteria that define sub-standard performance are material in nature and will only be triggered when significant performance issues present themselves on a project. Nevertheless, infractions recorded for vendors will expire after set number of months from the date of the infraction being logged. This ensures that where infractions have occurred, they are not unfairly held against a vendor indefinitely. By continuing to perform in accordance with the terms of the contract or project agreement, a vendor is assured that in time, all infractions will automatically fall away.

All projects are important to Infrastructure Ontario and it is our expectation that each vendor perform to its best in each project that it has been awarded. While all projects have their share of risks that must be managed by the vendor, all vendors have the opportunity to understand those risks during the procurement process. That being said, it is not necessarily true that more complex projects will result in a higher probability of infractions being recorded, since many of the criteria are directed to behaviours of poor performing vendors, not the risk of a project.  

More importantly, there is no objective way of distinguishing what is a 'complex' project from other projects and there is no fair way of determining what threshold number of projects would even entitle any vendor to special accommodations. In short, trying to adjust infraction weightings or calibrations to take into account project risk or volume would be even more unfair to vendors bidding to projects and would arbitrarily result in IO favouring certain bidders over others.

Good performance is performance in accordance with the contract or project agreement. The incentives to perform well are already clear within those arrangements.

Performance infractions will only be tracked against a vendor for limited period of time. After this time the infractions will expire.

Vendors are expected to perform to their best in each project, regardless of the delivery model or service category. Transposing infractions allows procurements to consider past poor performance across IO's entire infrastructure program, leveraging similarities in contract criteria to do so.

Not all infractions are transposable and expired infractions will not be transposed. Vendors will receive an explicit outline of all infractions, including those transposed, in their monthly program correspondence.

The Vendor Performance Program does not include any criteria relating to ethical bidding behaviour, conflict of interest disclosure, or other practices engaged during the bidding process.

Procurement documents and the IO procurement process already have significant safeguards to address ethical bidding practices, disclosure of perceived, potential, and actual conflicts of interest, and engagement by bidders or contracting parties in illegal acts.

Selection of subcontracts and their ultimate oversight under the project agreements and contracts is, and always has been, the responsibility of the subcontracting entity.

We simply want to improve the performance of vendors and reduce the instances of poor performance.

We want all vendors to meet the high standards we set as part of Ontario's infrastructure program. The Vendor Performance Program will demonstrate its value when we reach the state when no vendors have any points deducted from their evaluation.

No. All vendors must meet our stringent procurement requirements. This includes scoring well in all categories. Vendors who have experience in IO's infrastructure program will be familiar with our requirements and the key ingredients to achieve success. Vendors who have had success in our program and performed well in the past should expect to continue to be successful in the future.

Vendors who consistently fail to perform as specified will be forced to decide if they want to be a part of IO's infrastructure program as their future participation could be jeopardized.

Any construction contractor, general contractor, or Service Provider procured directly by IO with at least one (1) ongoing project will receive a Monthly Report. Vendors procured by a PMSP on behalf of IO will receive a Monthly Report in the event that they have at least one (1) unexpired infraction on their performance record.

Samples of Monthly Reports can be found in Appendix F of the program, available here.