Know what kind of Data you are using?

By arvind |Email | Oct 17, 2018 | 7131 Views

There are plenty of fine articles that explain the difference between structured and unstructured data in a smooth way. One which I read explained structured data; as the data that can be organised easily, which makes them easy to analyze. While on the other hand unstructured data can be explained as; the information that either does not have a pre defined data model or is not organised in a predefined manner, making it uneasy to analyze. One of the main goals of a data scientist is to extract the structure part out of the unstructured part.  Natural language processing is a process of extracting something useful (e.g., sentiment, topics) from something that is essentially useless. 
An article suggested that structured data are generated solely from internal/enterprise systems while unstructured data are generated solely from social media sources. I think it would be useful to separate the format (structure vs. unstructured) of the data from source (internal vs. external) of data.
Sources of Data: Internal and External
Generally speaking, business data can come from either internal sources or from external sources. Internal sources of data reflect those data that are under the control of the business. These data are housed in financial reporting system, operational systems, HR systems and CRM systems, to name a few. Business leaders have a large say in the quality of internal data; they are essentially a byproduct of the processes and systems the leaders use to run the business and generate/store the data.
External sources of data, on the other hand, are any data generated outside the walls of the business. These data sources include social media, online communities, open data sources and more. Due to the nature of source of data, external sources of data are under less control by the business than are internal sources of data. These data are collected by other companies, each using their unique systems and processes.
Data Definition Framework

This 2*2 data framework is a way to think about your business data (See Figure 1). This model distinguishes the format of data from the source of data. The 2 columns represent the format of the data, either structured or unstructured. The 2 rows represent the source of the data, either internal or external. Data can fall into one of the four quadrants.
Using this framework, we see that unstructured data can come from both internal sources (e.g., open-ended survey questions, call centre transcripts) and external sources (e.g., Twitter comments, Pinterest images). Unstructured data is primarily human-generated. Human-generated data are those that are input by people.
Structured data also can come from both inside (e.g., survey ratings, Web logs, process control measures) and outside (e.g., GPS for tweets, Yelp ratings) the business. Structured data includes both human-generated and machine-generated data. Machine-generated data are those that are calculated/collected automatically and without human intervention (e.g., metadata).
The quality of any analysis is dependent on the quality of the data. You are more likely to uncover something useful in your analysis if your data are reliable and valid. When measuring customers' attitudes, we can use customer ratings or customer comments as our data source. Customer satisfaction ratings, due to the nature of the data (structured / internal), might be more reliable and valid than customer sentiment metrics from social media content (unstructured / external); as a result, the use of structured data might lead to a better understanding of your data.
Data format is not the same as data source. I offer this data framework as a way for businesses to organize and understand their data assets. Identify strengths and gaps in your own data collection efforts. Organize your data to help you assess your Big Data analytic needs. Understanding the data you have is a good first step in knowing what you can do with it.

Source: HOB