With 1500 news articles, 7000 blog posts and half a million tweets created every minute of every day, there is more open-source information available to analyse and utilise than ever before. There are many different definitions for it, but OSINT, or Open Source Intelligence, is essentially the intelligence generated from data collected from publicly available sources.
The Ukraine conflict is a prime example of the power of OSINT. In the days leading up to the Russian invasion, OSINT researchers made use of Google Maps to track a suspiciously large traffic jam on a road leading to the Ukrainian border. Dr Jeffrey Lewis, an expert in OSINT, tweeted, “Someone’s on the move” and soon after, Russian troops invaded Ukraine. Since then, OSINT has played a significant role in providing valuable insights and information throughout the conflict.
Today’s threat intelligence teams understand just how important OSINT is for increasing situational awareness and reducing risk. But there are also key challenges involved with producing and using OSINT effectively.
1. Challenge of data overload
The amount of digital data created worldwide reached 64.2 zettabytes in 2020 and is forecast to grow to more than 180 zettabytes (1015) by 2025 according to per Statista.
Not only is there a high volume of data, that data it also comes from a vast array of sources, including traditional newspapers, magazines, books and TV, as well as online news articles, blogs, social media, reports and forums. Add to that, open-source data is found in in numerous formats and unlike spreadsheets or formatted data feeds, it is often unstructured. With so many sources, in so many different formats, it can be difficult to uncover relevant data, especially at scale.
As a result, threat intelligence teams can get mired in mundane and labour-intensive tasks, spending too much time manually collecting, validating, de-duping and standardising data across Google, OSM feeds, Twitter, premium providers, reports and more. You’re employing clever people on your intelligence analysis teams – you want them spending their time doing clever things. But these non-value-added activities decrease their efficiency, and can lead to analyst burn out.
Solution: Automate data collection and processing
Given that around 90% of data that your analysts need to deal with is unstructured data, automation is key. Threat intelligence solutions like Silobreaker make sense of out of unstructured data, and can and should be used at all stages of the intelligence production cycle. Automation also means more efficient production processing of volumes of data and information handling, as well as the rapid, reliable repetition of tasks, which minimises human error and allows your analysts to focus on the interpretation and reporting of OSINT, so they can provide confident, actionable and timely answers to stakeholders.
2. Need for source diversification
One of the main issues with OSINT is assessing the accuracy and completeness of the information. Data may not be current or it may be inherently biased. Disinformation and misinformation can also make it challenging to separate fact from fiction. To counteract this, intelligence teams need to employ source diversification – collecting information that is as comprehensive and representative as possible.
Source diversification ensures that information from multiple sources, languages, and viewpoints is evaluated to gain a more complete and accurate understanding of any given topic, event or situation. One of the best ways to ensure source diversity is to collect from sources in different languages, but the challenge is being able to translate these sources in multiple languages quickly and effectively.
The time and resources required to manually collect the range of intelligence sources required for source diversification is huge, but not doing so can lead to source bias.
Solution: Streamline data assessment and aggregation
An intelligence tool like the Silobreaker Intelligence Hub can scale intelligence data collection instantly across millions of sources, mitigating bias source biases by extending collection across all topics, threats, actors, geographies and use cases. Silobreaker’s Relevance Engine enables multilingual entity detection, which is the identification and extraction of named entities – like names of people, organisations or locations – from text data in multiple languages.
This allows your intelligence team to analyse and understand information from a variety of sources in different languages. Similarly, aliasing – where the same entity or topic is referred to by different names – can be a challenge, but Silobreaker can to identify and link different aliases to the same entity or topic. This helps ensure that relevant information isn’t missed.
3. Difficulties of source analysis
When it comes to OSINT, source analysis is crucial in determining the strengths and weaknesses of the information and its credibility. Are certain sources controlled or influenced by specific actors or governments? Or does the source have their own interest and agenda? Answering these questions can help you spot possible biases that may affect the reliability of the information.
It’s also vital to validate the credibility and reliability of the source. This is based on factors like past track record of accuracy, level of expertise and whether the information is consistent with other sources and known facts. Context is another key element of source analysis. Understanding of the environment in which an event or situation is taking place, i.e., studying historical, cultural, political, and economic factors that may be influencing the situation, is key to generating actionable intelligence.
Solution: Get visibility across multiple data feeds and providers
Instead of toggling between various intelligence subscriptions, utilise a central platform that is provider-neutral. This will allow you to compare sources and measure reliability and credibility faster and more effectively. A tool like Silobreaker’s Relevance Engine will even allow you to analyse and visualise information to identify relationships between people, organisations and places. This can help your intelligence teams spot key players and hidden connections that can be used to support decision-making and reduce risk.
Maximising OSINT with the right tools
Despite the many benefits of using OSINT, there are several challenges that come with it. The sheer volume of information, combined with the need to vet its accuracy, completeness and bias, can make it difficult to use.
But these challenges shouldn’t discourage teams from utilising OSINT to gather intelligence. By shifting from a manual approach to an automated data collection and aggregation process, and using the right tools to make faster, more relevant connections between different types of OSINT information, your organisation can gain the strategic advantage when responding to threats.
Click here to learn how.