Big Data


 Data/ Big Data

   Big Data - Quotes
"Big data is a buzzword, or catch-phrase, used to describe a massive volume of both structured and unstructured data that is so large it is difficult to process using traditional database and software techniques." ( Webopedia ) 

"It is also an evolving term that describes any voluminous amount of structured, semi-structured and unstructured data that has the potential to be mined for information." (Tech Target)

"It originates from many sources including Internet search, finance and business informatics. Scientists encounter limitations in e-Science work, including meteorology, genomics, connectomics, complex physics simulations, and biological and environmental research.  Data sets grow in size in part because they are increasingly being gathered by cheap and numerous information-sensing mobile devices, aerial (remote sensing), software logs, cameras, microphones, radio-frequency identification (RFID) readers, and wireless sensor networks."  ( from Wikipedia )

  Big Data - In Simple Terms - What is it?
"Big data is so much data that it is too big for a spreadsheet or database. We are talking millions/billions/trillions of records. You also want the capabilities to mix, mash, cross-reference and analyze the data to look for associations, causes and similarities and for other future uses.
 
 We are going to start with learning how to work with large amounts of data by finding out how popular your name was when you were born
    • Is your first name one of the top 10 boys/girls names in popularity?
    • ... the top 100?    top 500?    top 1,000?   top 10,000? top 100,000?
    • How many people in the U.S. have the same name as you?
    • How about variations of it? How many different spellings are there for your name?
    • Names, like many things, occur in cycles. Is your name on the rise or on the way down?
    • Can you tell the rough age (in decades) that a person is by their name?
We will look at these questions using Big Data (all the names in the U.S. since 1897)



1. BigData - Background and Resources
    1.1  Resources and Links - Sources of Data to work with
    1.2  Basics - Working with Data in many shapes and forms
    1.3  Data Sharing and Different Forms of Data Files - We will be using CSV (Comma Separated Variables)


     Sample Projects Done by Students 
            - See the screenshots





2a. BigData - Working with Lists(Text), Tables and Arrays - LiveCode
    Example 1 - How popular was your name when you were born?
    2.0  The Internet - Finding Your Name on other websites, how popular was your name when you were born?
    Working with 30,000+ records (names) for the millions of people born that year
    2.1  The Data - loading the data and investigating it
    2.2  Finding a Name (Message Box, Find, Find Word) - learning how to use the Message Box and doing finds
    2.3  Sorting the Names (Sort) - learning how to sort the list
    2.4  Displaying the Statistics (itemDelimiter, foundLine() ) - show the popularity and total # of names in the list

    2.9  Wrap-Up - comparative times working with text/lists/arrays

    Example 2 - Counties in Maryland - Median Ages, Household Incomes, etc
    Working with lists and various types of lookups

      Desktop Computer App - Using Lists, Tables to present the data
         2.1  Loading Data (Answer File, Repeat, after vs into) - loading data, customizing it
         2.2  Sorting Data (Sort, char) - sorting the data in different ways, getting rid of the $ sign

      Mobile-Style App - Using Fields to present the data
         2.3  Finding Counties (Find) - doing lookups, making it more appealing
         2.4  DropDown Menus (DropDown menus) - populating the menus with your own data

        Doing Both in one App
         2.5  Multiple Views (set, visible) - showing/hiding the table view of all the data
 
        Graphing the Data
         2.6  Making Graphs (Graph widget, Chartdata) - displaying your data in a graph

       Problems? CleanUp 
         2.6  Problems-1 - solving hangs, crashes and slow running programs


    Example 3 - Finding a grocery store near you.
    Managing CSV files with mixed data, converting data to a more useable format
     2.1  First Start - building the user interface, loading data, discovering the problem
     2.2  Solution - removing the quotes and converting the data to tab delimited


 
2b. BigData - Working with Tables and Arrays - Python/Java
    2.0  Your Name - How popular was it when you were born?
    2.1  Table 1 - Data - storing data in tables, accessing the data
    2.2  Table 2 - starts with - more access control ("startsWith", "endsWith" functions)
    2.3  Table 2 - Boolean Logic - more complex queries, using Boolean logic
    2.4  Table 4 - Count 1 - keeping count using variables
    2.5  Table 5 - Count 2 - multiple counters, using if statements






3a. BigData - Advanced Techniques

     - LiveCode
     - Python
     - Java




4. BigData - Your Own Project
    4.0  Your Project - Design your own project


notes: