Public sector software procurement: why it’s still broken
The other week I went to a supplier briefing on the New Government Rules for Sourcing that will “make it easier for you to do business with Government”. I’d been involved in a workshop organised by NZRise (a group representing NZ owned IT businesses) at the end of last year plus with Novopay and the proposed new IRD Tax system fresh in everybody's minds I went in with high hopes that there were going to be some fundamental changes to the procurement rules/guidelines that many of us in the software vendor community are crying out for.
I came away disappointed. Although there were some positive changes, I felt that it wasn’t going to fundamentally change things: we’re still going to have too many project failures, lost opportunities to save taxpayers money and lost opportunities to grow the NZ export knowledge economy.
Here’s my take on why.
First of all everybody knows that most big software projects fail. It’s simple: the complexity is too big in relation to the rate of change (therefore unpredictability). The other problem with big projects is that smaller vendors (i.e. most of the ones in NZ) tend to have very little chance of getting involved (in Europe they are waking up to this problem). The answer is obvious to most people in the industry (hint: break up into small projects, as smart folks like Rod Drury and Ben Gracewood have explained recently).
I don’t think the new rules help to address the project size problem however that’s probably a rant for another day. So let’s focus on procurement for smaller software projects that should have a reasonable chance of succeeding (I’m thinking around $NZ40,000 - $NZ400,000).
In my experience this is how the majority of software procurement processes work:
- There is an internal process to define business requirements.
- Sometimes a Registration of Interest (ROI) is issued
- Either an “open” or “closed” Request for Proposal is issued (RFP). An open RFP is publicised and anybody can respond, a closed RFP is issued to a small number of vendors. Most RFP’s ask for fixed cost and fixed scope.
- Typically the vendor is not allowed to meet the purchaser and if they submit questions the questions and answers are sent out to all other vendors.
- The purchaser evaluates between 4 (if closed) and 30 (if open) responses and picks one or more vendors to give final presentations and start contractual negotiations.
- The purchaser picks a winner and contracts are signed
This process works generally pretty well when the business is able to clearly define core requirements that roughly map to a number of popular, typically reasonably slow moving, commercial off-the-shelf products (COTS). Examples are DMS, CRM, CMS, LMS, Accounting, (dare I say it Payroll but for lots of reasons clearly Novapay wasn’t COTS) systems etc.
This generally implies that the purchaser wants a product that lots of other people are already using and is willing to adapt business requirements to how the product functions (both now and in the future).
I don’t have a problem with this. My problem is when this procurement process is applied to software projects very different from the above; that is,. projects where it’s very difficult to define business requirement (e.g. stakeholders and users don’t know what is now possible, don’t know what they don’t know, and don’t know what they want until they see it) and there are limited or no options for COTS solutions, i.e. either a fully or partially bespoke solution is required.
So we are talking about the sorts of software project which involves smart technology specialists (generally the vendor) and smart domain experts (generally the purchaser) putting their heads together to create a software solution that achieves some business outcomes.
These projects require innovation / creativity, a “one team” approach and would typically follow an Agile / Lean Startup methodology that focuses on getting working software out to real users quickly to validate assumptions.
Given the increasing rate of change in the web / mobile world, the now relatively short short lifetime of any solution and the rise of the “meta solution” that is made up of connected cloud based components (e.g. Google maps, Xero, Mailchimp etc) I believe that more and more software projects are falling into this category.
So what happens now when a traditional procurement process described above is applied to this category of project? Some of the things I have experienced:
The best vendors simply don’t respond. The questions asked in the ROI/RFP documents are impossible to answer and the risk/time/effort vs reward doesn’t stack-up.
Smart vendor sales teams respond but write what the client wants to hear. The most convincing liars (including making up costing figures) get through to the next stage.
Lots of vendors (i.e. often 20+ if it goes straight to RFP) put a considerable amount of time/effort into responding (often they assume the budget is bigger than it actually is). Often the majority of the vendors are local and therefore there can be a significant economic loss to the region (for example it turns out the budget for the project is 150K, 20 companies invest 15K’s worth of time to try to win therefore the process costs the region/country 300K - it’s crazy)
Vendors provide low up front costs. They know they can recoup losses later on when the client is locked in and inevitable scope changes when “actual” requirements become clear.
RFP documents generally don’t do a good job at providing useful requirement/feature prioritisation. The tendency is for lots of people in the early stages to throw in every requirement they can think of which results in vastly over complex and expensive (and therefore high risk) solutions.
The RFP response process becomes a competition where vendors invest considerable time trying to guess the solution (and perhaps building key parts of it) to try to win. Given that they have very little or no exposure to stakeholders/users at that point then this can be a frustrating and futile process. The exception to this is when a vendor is an incumbent and/or has insider knowledge of the business - they then stand a much better chance of winning.
The purchaser gets a bunch of RFP responses that are impossible to compare and cost estimates are worthless (e.g. they can range from 50k to 1m)
By the time the requirements analysis and procurement process has taken place (often 6+ months) the world has changed and everything is invalid.
Vendors who have lots of time on their hands respond (perhaps because they are not very good or they are new).
Once the vendor has finally been selected the project is now setup to fail. Purchaser and vendor straight away move into combat mode where each starts arguing over the detail of what was signed in the contract. Often it’s the difficult to define non-functional requirements (e.g. usability, security, scalability etc) and the inevitable missed but vital functional requirements that cause the biggest conflicts.
So what’s the alternative?
The current trend seems to be creating supplier panels which can help in some respects but also has a whole lot of problems which I won’t go into now. I believe purchasers need to think differently: they’re not trying to buy something, they are trying to select a vendor partner who they can work with to achieve some outcomes (and build a long-term relationship). In the procurement rules it clearly states that the purchaser must select a capable vendor that gives the “best value for money” or “lowest price”.
This reinforces the message that the purchaser must get a fixed cost for a fixed scope of work (otherwise how could they assess value for money or lowest price from RFP responses). At the heart of the problem is that the vendor cannot (and should not) give a fixed price and fixed scope at the RFP stage. History shows us that if vendors are forced to do this then they make stuff up with often disastrous consequences.
So if we accept that fixing scope and cost is a really bad idea at RFP stage then how can we achieve a procurement process that ensures that the purchaser fairly selects the best vendor and gets the best value for money over the lifetime of the relationship with the vendor.
To do this I believe the procurement process needs to focus on three things:
1) Vendor credibility.
There is obviously the housekeeping stuff like size of company, stability, skillsets offered etc but the most important part is speaking to actual current and previous clients. For smaller vendors (under 30 people) will they let you talk to all their clients over the last couple of years (including the projects that haven’t gone so well). For larger vendors it becomes more important to identify and speak to the current / previous clients of the team / department who the purchaser will be working with.
2) Cultural alignment
How well do the culture and values of the purchaser and vendor align? Check how well the “official” culture and values maps to to the reality (having some informal conversations with staff at all levels can give valuable insight). If the “values” of two organisations align well then they are far more likely to get through the inevitable difficult situations later on when things don’t go smoothly.
3) Ensuring value for money
The reality is that no purchaser (especially a government agency) wants to give a vendor an open cheque book. So if the vendor hasn’t committed to a fixed scope/cost before contracts are signed then how does the purchaser ensure that they are picking a vendor who will always give great value for money? Well actually they can’t, however what they can to is to put in place strong incentives to do so (and a way out if they don’t). At the heart of these incentives are contestability and risk-sharing.
Contestability essentially means that the purchaser can replace the vendor at any time in the project.
Nobody wants this to happen but it is vital that it can happen. In order for this to happen there must be no IP or other contractual barriers (i.e. no lock-in by using proprietary or obscure technology) and there must be other vendors able to take over if necessary. Modern Agile methodologies have a “sprint” cycle of between two to four weeks where working/tested software is created every cycle.
With agreed conditions the purchaser should have the ability to switch vendors at the end of any sprint. Having an independent expert to periodically ensure that the vendor isn’t gradually maneuvering things into a lock-in situation can be a good idea.
Risk-sharing involves numerous well documented techniques for ensuring that both the purchaser and vendor are motivated to achieve the business outcomes for the lowest cost/time.
The RFP can specify these or vendors can be asked to propose risk sharing strategies that they are comfortable with.
Some practical recommendations for purchasers:
When considering business requirements and business case, also consider the cost of doing nothing (and therefore the financial or other impact of a lengthy procurement process). Set a date when a vendor will be selected and stick to it.
Issue an ROI to open market. In this document i) ideally the budget range should be stated (i.e. the upper limit is where the business case starts to not stack up or the figure that has been formally allocated), ii) it should be made clear that IP developed by the vendor can be commercialised by the vendor later on and iii) contestability requirements clearly stated. The ROI can be used to identify if there are any COTS solutions (and therefore how the rest of the procurement process should run)
Make it easy to respond (web forms are a great way to capture structured responses). Please don’t ask for 10 printed copies.
Do your homework and select a small number (i.e. under 5) of credible vendors who you ask to respond to RFP.
Create a lightweight RFP document that asks the vendors the right questions (see above). The RFP should state or request risk sharing options.
Encourage face to face meetings with all short-listed vendors. Ensure those meetings i) include the people who will actually be doing the project, ii) are private (i.e. what happens in those meetings isn’t shared with other vendors)
Short-listed vendors (between two and three) could be paid a fixed amount to undertake the first part of their process that focuses on scoping, ideation and sometimes prototyping (undertaken by the proposed team members). This is the perfect opportunity to asses how the purchaser/vendor project team will work together, plus gives the team an opportunity to explore high level options and create realistic scope/cost estimates. The purchaser will gain a huge amount of knowledge along the way (in some cases this knowledge will cause them to abort or radically change the project at that point - potentially saving a lot of money)
I would argue that public sector organisations owe it to the taxpayer to do better in this space due to the potential cost savings and improvements in online public sector services. Government towns like Wellington can reap huge benefits if better partnerships between government agencies and software vendors can be built.
If we can get this right and software vendors can be encouraged/helped to commercialise and export the intellectual property gained then it could have a major positive effect on the local and New Zealand economy.
Mark Pascall is the Director of 3months, a Wellington based web and mobile software development company. For over 20 years he has been involved in public and private sector software procurement processes as both a purchaser and vendor.