Essential big data tools for government
Big data. It’s massive. It comes in all types of formats. It’s dynamic, changing.
Government managers are looking to derive value from the mountains of data collected by their agencies to tackle a host of issues, including cybersecurity, fraud detection, crime prevention, medical research, weather modeling, intellectual property protection, operation efficiency and situational awareness.
A growing challenge is choosing the right technology to aid in collecting, processing, analyzing and storing massive amounts of data, especially since the pool of big data tools keeps expanding.
Unfortunately, there are no “must have” tool sets, since any initial big data deployment will be driven by an individual agency’s business requirements, according to the TechAmerica Foundation’s report, “Demystifying Big Data.”
Tools that ingest and extract data, index it, translate it and then clean it up for analysis and presentation are part of an expanding big data ecosystem, said Barbara Toohill, vice president and director of Mitre’s Homeland Security Systems Engineering and Development Institute.
To put these technologies to best use, agency managers need to understand the problem they are trying to solve and then determine what data is needed to solve that problem before investing heavily in tools, Toohill advised an audience of government and industry representatives at a recent FCW Executive Briefing on big data in Washington, D.C. “One of the challenges is that tools sound great in PowerPoint presentations, but are much more challenging when people start using them,” she said.
Still, there are core technologies that come in both open-source and proprietary solutions to support any agency’s big data portfolio. But where to start?
Here are some examples of program areas in federal agencies and non-federal organizations that are actively developing and applying big data tools and big data strategies.
1. Big Data Assist in Fraud Detection and Financial Market Analysis
- The Social Security Administration (SSA) is using a big data strategy to analyze massive amounts of unstructured data in the form of disability claims. The SSA is now able to process medical taxonomies and expected diagnoses more rapidly and efficiently to recreate the decision making process, better identifying suspected fraudulent claims.
- The Federal Housing Authority (FHA) has over 23 years of experience in leveraging analytics to manage a positive cash-flow fund. The FHA is the only sub-prime mortgage insurance fund that did not need a bail out during the housing bubble burst. They apply big data analytics to help forecast default rates, repayment rates, claim rates. Additionally, they use big data technologies in building cash flow models for likely scenarios to determine what premiums would need to be in order to maintain positive cash flow.
- The Securities Exchange Commission is applying big data strategies to monitor financial market activity. They are using natural language processors and network analytics to help identify nefarious trading activity.
2. Big Data in Health-Related Research
Big Data analytics can be used for identifying in advance and responding to hazardous natural disastersand or health issues such as the recent Covid-19.
The Food and Drug Administration (FDA) is deploying big data technologies across many labs involved in testing to study patterns of foodborne illness. The database, part of the agency’s Technology Transfer program, allows the FDA to respond more quickly to contaminated products that enter the food supply and contribute to the 325,000 hospitalizations and 3000 deaths related to foodborne illness every year.
3. Big Data Tools Assist in Government Oversight and Education
The Notice and Comment project provides instant access to over 4 million government documents, from federal regulations published by the Federal Register to local public notices. The project is using advanced analytics and natural language processing to ingest government documents and track changes in policies, laws or regulations. Users can comment or vote on pending federal regulations or local public notices with ease. The site’s data updates daily for real-time monitoring of proposed actions and emerging trends.
4. Big Data Tools Assist in Fighting Crime
The U. S. Department of Homeland Security (DHS) is a great example of the need for big data strategies in the public sector. The organization of this agency highlights the need for interoperability and integration of data across numerous government agencies. DHS has many examples of how to integrate well, yet provides many lessons learned for how interoperability could have been developed more successfully.
Big data has been one of the core benefits of Big Data. Big Data can help state government departments in uncovering crimes and other illegal activities that pose a security threat in society. Such data includes information of all sections of the public. Careful analysis of this can aid in identifyingsuspicious activities patterns that indicate fraud and or crimes.
5. Big Data tools helping Environmental Protection and Energy Exploration
National Aeronautics and Space Administration (NASA) and the U. S. Forest Service have been working on a big data strategy for several years to improve interoperability and integrated research efforts which enable them to better predict weather, ground conditions, and forest fire risks. This effort took a lot of coordination in data requirements and data governance ahead of time. In addition to the advanced technology, the challenge required many humans in the loop to better understand the problem and coordinate the use of available data to develop an integrated strategy.
6. Improved emergency response:
Big Data analytics can be used for identifying in advance and responding to hazardous natural disastersand or health issues such as the recent Covid-19.Data from various sources including healthcare organizations, medical professionals and municipal records can be used to identify the spread of the diseases. Moreover, aid can be provided immediately as the needy section of people can be easily recognized.
7. Coordination between departments:
The data collected or analyzed by different departments of state/federal governments can be stores on a single platform. Development of big data platforms and implementation of big data analytics will enable all government agencies to access data from a single source and work in the same direction.