Recently, the Office of the Director of National Intelligence (ODNI) unveiled a new Open Source Intelligence (OSINT) strategy and called OSINT “first-instance INT.” Public and private sector organizations understand the value this discipline can provide, but are also finding that the exponential growth of digital data in recent years has outpaced many traditional OSINT methods. Fortunately, artificial intelligence (AI) and machine learning (ML) are beginning to have a transformative impact on the future of information gathering and analysis.
What is Open Source Intelligence (OSINT)?
Open-Source Intelligence refers to the collection and analysis of information from publicly available sources. These sources may include traditional media, social media platforms, academic publications, government reports, and any other publicly available data. A key characteristic of OSINT is that it does not involve covert or covert intelligence gathering techniques such as human intelligence or social engineering. If I could get data while working for the US government, but can no longer as a civilian, that is not OSINT.
Historically, OSINT has been a time-consuming process involving several key steps:
- Identify sources: Analysts determine which public sources may contain relevant information.
- Data collection: Information is gathered from these sources, often using manual searches or web scraping tools.
- Data processing: Collected information is organized and structured for analysis.
- Analysis: Experienced analysts examine data to discover patterns, trends, and insights.
- Reporting: The results are compiled into reports for decision makers to make more informed decisions.
Although effective, this approach faces limitations due to the sheer volume of information available. It’s difficult for human analysts to process everything by hand, and valuable information can be hidden in complex patterns that are difficult for humans to detect. This is where AI/ML can be a huge benefit in collecting, processing and analyzing information, freeing up human analysts to focus on things they are uniquely qualified to do, such as providing context. As a side benefit, this shift often improves morale as people spend less time on mundane processing tasks and more time analyzing and reviewing information.
Tasks where AI/ML can be of immediate benefit include:
- Big data processing: Artificial intelligence systems can process and analyze vast amounts of data at speeds far beyond human capabilities. This allows OSINT practitioners to cast a much wider net than previously possible and still deal with the results.
- Real-time analysis: The volume of information flows in today’s digital world is staggering. AI-powered OSINT tools can monitor and analyze data streams in real-time, providing operational intelligence and enabling rapid response to emerging situations.
- Multilingual and multimodal analysis: AI can break down language barriers by translating and analyzing content in multiple languages simultaneously. Moreover, it can comprehensively process different types of data – text, images, audio and video, providing a more complete intelligent picture. Many of these capabilities, such as Whisper OpenAI, can be used offline, eliminating operational security (OPSEC) concerns.
- Predictive analytics: By analyzing historical data and current trends, AI can help predict future events or behavior, adding an active dimension to OSINT.
- Automation of routine tasks: AI can help automate many labor-intensive aspects of OSINT, such as data collection and initial filtering, freeing up human analysts to focus on higher-level analysis and decision-making. Things that used to be very difficult, if not impossible, to implement, such as accurate sentiment analysis, are now trivial.
U SANS Network Security in SEC497 Practical OSINT course and Art SEC587 Advanced OSINT The course will give students hands-on experience in using these AI capabilities to not only increase productivity, but also discover new opportunities.
While no technology is perfect, and we must consider the potential effects of hallucination before implementing AI, key elements of technology currently in use for OSINT include:
- Natural Language Processing (NLP): NLP allows machines to understand, interpret and create human language. In OSINT, NLP is critical to:
- Sentiment analysis of social media posts
- Object recognition to identify people, organizations, and places in text
- Topic modeling for the classification of large volumes of textual data
- Machine translation for multilingual intelligence gathering
- Computer vision: This technology allows machines to interpret and analyze visual information. In OSINT, computer vision is used to:
- Face recognition in images and videos
- Comparing faces to determine if the same person is in multiple images
- Detection of objects in images
- Optical character recognition (OCR) for extracting text from images
- Understanding the scene in the video material
- Machine learning and data mining: How many times have you heard “those who don’t know history are doomed to repeat it”? Machine learning is the epitome of this concept, as it allows systems to learn from data and improve their performance over time. In OSINT, they are used to:
- Predictive analytics to predict trends or events
- Anomaly detection to detect unusual patterns or behavior
- Clustering and classification of data to facilitate analysis
- Network analysis to understand relationships between entities
I’ve been involved in OSINT for nearly two decades and this is by far the most dynamic and exciting time I’ve seen, with new developments in the space happening literally on a daily basis. If you are going to be in Network security in Las Vegas this September, I look forward to discussing how this opportunity can improve our effectiveness today, as well as what we can expect in the future.
Not yet registered with SANS Network Security? Check out this page to see everything in store!
note: This article is written by an expert Matt EdmondsonChief SANS Instructor and Director of Argelius Labs, with ten years of professional OSINT experience.
