TAMPA BAY, Fla. - Prior to the state's failed SunPass upgrade in June, troubled FDOT contractor Conduent accidently left a test site - with customers' account information - on an unsecured public internet page - a risk that experts say would have made it easy for hackers to access and exploit 6.5 million customers' accounts.
But the state never told customers of the breach risk, even though FDOT's own data security experts expressed concern in May about the mistake.
"The instant you put anything on the internet publicly, there are attempts to hack it," said Bryan Graf, Vice President of Development at Abacode CyberSecurity Experts. “Exposing a test site to the Internet, especially without encrypting it's traffic, could potentially increase the chance that maliciously – or accidentally – your information could be released and used by a malicious actor"
Graf said hackers from foreign countries, including Russia and China, are regularly scanning the websites of American corporations and institutions for vulnerabilities. And even though the SunPass test site was password-protected, he says it wouldn't be hard for any skilled hacker to access personal account information on an unsecured site.
It's not clear how long the SunPass test site was left vulnerable, but a May 31 email from an FDOT security risk and compliance expert, obtained by 10Investigates through a public records request, warns the agency, "this is concerning and could potentially put our customer’s (sic) data at risk."
“We need to discuss this ASAP!,” the consultant wrote in an ensuing email to Florida Turnpike Enterprise Director of Toll Systems Buzz Holland.
The test site was removed immediately, but FDOT could not tell 10Investigates if had tried to determine if any customer data had been compromised.
An FDOT spokesperson said in a statement, "the Department requested and was provided broader access to the Conduent Network Architecture for the Department’s own security team to exercise additional oversight."
But the agency never warned its 6.5 million active accounts with 9.1 million active transponders that they may want to change their passwords and review their records to protect their personal data.
The audit that might have prevented the SunPass Saga
FDOT had concerns about its contractor's data security protections as far back as November 2017, according to emails reviewed by 10Investigates. The agency hired outside auditing firm RSM to prepare Conduent for a security review known as a "SOC 1" or "SSAE16" audit.
However, FDOT employees and consultants suggested in emails that Conduent wasn't prepared to move forward with the review at the time. Those in charge of overseeing the project then spent months emailed each other seeking information about Conduent's internal controls, which detail how the company would protect the integrity of its financial transactions and customer data.
To this date, FDOT and Conduent still have not performed the audit, even though it is required by Conduent's contract with the state.
Emails between FDOT employees and consultants also document concern over Conduent's failing system tests in April, as the contractor was nearing another deadline to launch the upgrade.
"Our overall concern is that Conduent has not demonstrated sustained performance," FDOT CPA Adriene Pierce wrote Walter Kristlibas, an FDOT consultant working for engineering firm Atkins. "Of the 49 transactions tested, 5 initially failed. Conduent was able to make the corrections and ultimately execute all transactions; however, at this point we expected a flawless run."
Kristlibas responded to Pierce asking if there was "an underlying issue that is a ticking time bomb" that threatened the already-delayed May 14 launch data.
FDOT insiders tell 10Investigates pressure was mounting at that point to get the new SunPass system live, even if it wasn't perfect yet. The launch date was ultimately delayed again, to the first week of June.
That's when that ticking time bomb went off.
What was supposed to be a six-day upgrade turned into a months-long debacle, with massive account errors, crashed websites, and widespread customer service failures.
Computer errors also led to what appeared to be a small breach of personal data in June; FDOT maintains only 15 users were affected, but the agency failed to follow legal guidelines for notifying customers after a breach.
At no point during 10Investigates' reporting of the previous breach did FDOT ever notify the press or any customers of its May mistake, which may have compromised millions of customers' accounts.
In addition to the issues related to a possible widespread data breach, Conduent is facing questions about its ability to maintain basic security standards regarding its financial processing and cardholder data.
Journalists in Texas recently reported the company lost its compliance with the Payment Card Industry (PCI) Data Security Standards. A spokesperson for FDOT says the company's compliance in Texas has no effect on its compliance in Florida, but provided few other specifics.
For the first time since June, a Conduent spokesperson responded to 10Investigates' questions Monday, but has not yet provided comment.
State downplays problems, possible breach
Since the start of the SunPass meltdown in June, FDOT officials have been coy about their role in overseeing the failed project, refusing to field reporter phone calls, and refusing to answer some questions.
At times, the agency communicated almost exclusively through optimistic press releases that downplayed the significance of their problems and the risk to SunPass customers. Recently, the agency's general counsel has been providing documents to 10Investigates' records requests in a weekly Friday evening "document dump," with no other communication the rest of the week.
When confronted about the lack of transparency last month, FDOT Secretary Mike Dew promised 10Investigates his agency would do better.
SATIRE: "Is it too late now to say sorry?"
But when it comes to the May error that left customer account information on a forward-facing, unsecured internet site, FDOT has yet to alert customers that they may have been exposed.
"They have to know they're a target," Graf said.
FDOT has blamed Conduent for the SunPass failures, but it doesn't appear the corporation - or any of its employees or paid consultants - were held responsible for mistakenly exposing the system and customers' personal data.
Advice for SunPass customers
Graf says, without knowing if SunPass data was breached, all users should change their passwords as a precaution and keep an eye on accounts that may have been exposed.
Another expert, Sri Sridharan, with the Florida Center for Cybersecurity at USF, suggests changing passwords regularly is good regular cyber-hygiene for any online account. That applies for security challenge questions as well.
10Investigates has revealed how SunPass account errors are the responsibility of the customer to identify and report before refunds are issued.
Send your SunPass issues and other story tips confidentially to 10Investigates' Noah Pransky on Facebook or email him at email@example.com.
►Want to view an interactive timeline of the SunPass maintenance woes? Click or tap here
►Click here for a running report card of other SunPass problems 10Investigates has tracked.
►Make it easy to keep up-to-date with more stories like this. Download the 10News app now.