Dev/ops helping 2 agencies take on complex IT modernization

Don’t mention the word agile or the phrase dev/ops to the U.S. Citizenship and Immigration Service in the Homeland Security Department.

USCIS made the transition to this methodology about six years ago, so now it’s just how it does business.

USCIS is among the handful of agencies where agile is no longer a buzzword.

Sarah Fahden, the chief of the Verification Program Portfolio at USCIS, said the journey to being a dev/ops or agile shop started quickly after DHS made the project a high priority modernization program almost six years ago.

Advertisement

Fahden, who spoke on a panel at the Advanced Technology Academic Research Center’s Dev/Ops conference, said USCIS started moving pieces of the verification program to Amazon Web Services within three months.

“We’ve been using the strangler approach, so we are breaking off pieces of the app and rebuilding it in the cloud,” Fahden said on Ask the CIO. “Then we moved the rest of the application, the legacy pieces, to Amazon as well.”

The move to dev/ops, or what is also known as iterative development, received another boost a few months ago when USCIS achieved the ability to do continuous deployment of new software.

Fahden said USCIS now can deploy numerous times a day to production.

“Anytime there is a bug that we find out about, our users are very close knit into our operations, so they have instant reach back, and we also have an environment called ‘preview’ where user can opt-in to see new features before they go live to everyone else and they can choose to be live testers of that in production. They will give us the feedback and we will fix the bug immediately — sometimes in minutes or days depending on how much development work it takes and the backlog,” she said. “Everything is automated. Our entire infrastructure is dev/ops so no one touches production ever.”

USCIS is going even deeper into the dev/ops world by using microservices for each specific part of the verification modernization effort.

Over at the Federal Communications Commission, the dev/ops effort is newer, but having a similar impact.

Dustin Laun, a senior adviser for technology and innovation at the FCC, said three years ago the commission had 207 legacy systems, built from scratch using custom code.

Laun said he started to help the FCC move into the dev/ops world by taking a legacy application used by 400,000-to-500,000 end users and every carrier in the U.S.

“We rebuilt that system on a software-as-a-service (Saas) platform. We didn’t write any code. We picked a product and did this project in 60 days from procurement to launch,” he said. “We handed the keys over to the business and IT doesn’t even get involved in the project, very rarely, anymore. That is the vision of how we’d like to operate. IT doesn’t need to be the bottleneck.”

Laun said the other part of this vision is for the commission to stop writing code because there are plenty of commercial platforms that can get organizations 95 percent there.

“We went all in on a platform. We do our 80-15-5 model — 80 percent on that platform, 15 percent on SaaS and 5 percent write from scratch,” he said. “We built the Net Neutrality commenting system, that John Oliver loves to have people go to, from scratch and it’s a nightmare to maintain.”

Laun said he also doesn’t want requirements to modernize a legacy app. Instead, he wants to work with the business owner to rebuild the app in near real time, and then make changes as needed.

“Currently, we just did a rationalization of applications that made sense and we are picking them off,” he said. “Every day there should be something pushed to production. We invested in ServiceNow and use their custom application model and they have a process of moving through their environment. Half of what you can do in production you can do with no risk. Part of dev/ops to me means understanding where your real risk is and stop being a bottleneck to that, and only put governance around things that can have high impact.”

Fahden said USCIS took a similar approach with the verification program by rationalizing its apps, which she described as spaghetti code where all packages connecting to different tables in the database.

“We take small piece of it, but a full feature or functionality and rebuild it and modernize it in the cloud. It still connects back to the legacy application and refactor on the legacy side,” she said. “Then one piece at a time, we keep ripping out different pieces until eventually there isn’t going to be much left of that legacy application over time. We’ve created six microservices of different pieces of the app.”

Fahden said her next major initiative is to build a back-end person-centric service.

“We took all of the data, person records from the agency that came from nine different systems, about 835 million records, and then we resolved the entities, we are at 94 percent entity resolution,” she said. “We will now build a person-centric service with microservices attached to it where these systems will now redo how they use their person data. It’s pretty fascinating because never before has this been done at USCIS. All of our systems are really primarily transaction based, and then within each system there is a lot of keying in, manual processes, where people are typing in people’s names and things so the same person could be represented in numerous different ways across the different systems. It takes a lot of manual processes in order to figure out who that person actually is and which records are associated with them for the benefit they are trying to process. This will be quite a groundbreaking thing for USCIS.”