Author Archives:

Big Data, how big is big

Size

When it comes to Big Data, there is no way to get away from the discussion on the Size of data. It is one of the three V’s (Volume)

“Big data” is high-volume, -velocity and -variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. ~Gartner

 

The interesting part about the Volume of data is not just the sheer about of it that we generate on a daily basis but the fact that due to the lowering cost of storage, we are able to actually keep a good portion of it around.  This in turn gets us talking about size.  This is where people try to dazzle you with their intellect.  For example, you may have heard of an Exabyte.  There may be a possibility that you know that and Exobye is

1 EB = 1018 bytes (10 with 18 zeros behind it)

1000000000000000000B

But seriously even if you are a mathematician, the complexity derives from the fact that most people cannot visualize the size of data in that format.

Is that important?   You bet it is.  One of the key attributes of a Data Scientist is to be able to adequately convey the complex to non-technical people.  To bridge the gap between those with the tools and those with the power (or budget) in an organization.  This may be a small example but it is a good one, breaking down how big data is in terms people can understand is important.

grandmaSo all of that being said, I like to describe it in a way that even my Grandma can understand.  I just relate everything to a GB.  From GB harddrives, to GB in your phone, to GB in your camera, most people can wrap their head around the size because they are used to doing mental calculations based on how many songs, how many files, how many pictures they can hold on x number of GBs.

OK… So lets start talking about big numbers now.

Terabyte = 1024 GB.

With the low cost of storage now, many of you may have Terabyte drives at home or in your computer.  I have a 2TB drive that I carry around with me with EVERYTHING I need on it.  So this one is not much of a comprehension challenge.

Petabyte = Over 1 Million Gigabytes

Ok, so one step up. Over One Million Gigabytes.  What does that get us.  Well, the size of the 10 Billion photos on Facebook is 1.5 Petabytes.  Or the amount of Data processed by Google…. Per day is 20 Petabytes.

Exabyte = Over 1 Billion Gigabytes

When we get into Exabytes, we start talking about how much storage entire Data Center Building hold.  Like Utah Data Center for the Comprehensive National Cyber security initiative (CNCI).  Or cold storage for those Facebook pictures we mentioned.

Zettabyte = over a Trillion Gigabytes

It is said that over 2 ZB of data is created every day.  We don’t store this much (we don’t have the ability) but this much is created. The “Zetta” was recognized by the 19th International Committee for Weights and Measueres in 1991 (along with “Yetta”) .  So where is all of this data coming from?  See the infographic below.

DatainOneMinute

infographic by Domo.com

infographic by Domo.com

 

 

Yottabyte = over 5 thousand trillion Gigabytes

Of course if you are the NSA, you need more storage.  Like the NSA’s secret (ha ha) Data Center in Utah. http://www.foxnews.com/tech/2013/06/13/what-we3-know-utah-nsa-mega-data-warehouse/

Imagine being able to comb through that data.   A Yottabyte is so big that you have to start talking about small things to put in in comparison.  For example there are 1 Yottabytes of Atoms in 7000 human bodies.  Or A Yottabyte of 1TB hard drives would require a data center covering 1 million city blocks

 

 



Big Data — What is HBASE

hbaseThe great thing about Big Data technology is that there are so many tools in the Data Scientists Tool Belt. The bad thing about Big Data Technology is that there are so many tools in the Data Scientists Tool belt.

When we talk about the tools that we use when working with Big Data, an overwhelming majority will discuss Hadoop, the Apache foundations implementation of Map Reduce and Distributed File Systems (HDFS in this instance. Which was created by Doug Cutting after Reading papers on the subject produced by Google Engineers while he was at Yahoo. (He is now at Cloudera). But big data tools rarely if ever work alone. It is a collection of tools and databases that help Data Scientists be more effective in their analysis (or just help to speed things up).

One of these technologies is HBase. HBase is a non-relational (NoSQL) database that is a Java implementation of Google Big Table. It is what is referred to as a Columnar Database. As oppose to Relational Database which stores its Data in Rows, it stores its data in Columns.

So that’s easy to say, but what exactly does that mean. Lets start with the definition Google lays out in its document on Big Table. http://static.googleusercontent.com/media/research.google.com/en/us/archive/bigtable-osdi06.pdf

” A Bigtable is a sparse, distributed, persistent multidimensional sorted map.”

Lets break that down to see what that means.

Sparse

A database is said to be sparse because of lack of data but not in the traditional sense of the term which would usually mean that there are very few items in the database.  In relation to HBASE, it is called sparse because of its ability to have sparse data in its entities.  What this means is as opposed to a relational table that would  require you to fill out all (or most) of the fields (Think of a customer data table in a relational data base), a columnar database can be empty or NULL without it adversely affecting the database functionality.  In addition, this also gives you the added benefit of being able to add other pieces of data you would like to capture on the fly.  In a Relational Database, you create a schema (FirstName, LastName, SS#, TelephoneNumber) and hope that you have thought of all the data you need to capture at the time of creation.  The NoSql schema-less databases allow you to add fields when needed or discovered without interrupting the normal flow of the operations.

Distributed and Persistent

HBase utilizes HDFS (the Hadoop Distributed File System) to distribute data across several commodity servers.  This is how Hadoop and and HBase are able to work with vast amounts of data. It is based on another article from Google on the Google File System which Doug Cutting used as a basis for HDFS http://static.googleusercontent.com/media/research.google.com/en/us/archive/gfs-sosp2003.pdf .  We will discuss HDFS and DFS in more detail in another post.

Multidimensional sorted Map

A Map (sometimes called and associative array) is a collection where the index of what is being stored does not have to be an integer but can also be arbitrary string.  It is a collection of Key/Value pairs where the key is unique.  The Keys are sorted in lexicographical order. (Not alphabetical, nor Alphanumeric, but sorting on the Unicode value of the string)

What you gain/give Up

Using HBase allows you to store your data both preprocessing and post processing in HBase and gives you some greater flexability and host billions of rows of data with rapid access. The downside is that when you use HBase instead of HDFS, tools like Hive (SQL like retrieval of data) are 4-5 times slower than plain HDFS. In addition, the maximum amount of data you can hold is about 1 petabyte as opposed to 30pb in HDFS.

 In the next post will dive deeper into the specifics of HBase, like set up, usage, and analysis.



New WP & W8 Apps from the West

AppsAs a Developer Evangelist from Microsoft I help developers develop apps for Windows Phone or Windows 8. I cover the Western United States. Because of that, I wanted to do a post to showcase some of the apps that are coming out of the west. Of course, because Silicon Valley is part of the West Coast, I am sure you will recognize a few of these apps, but they are not all from big brand names.

Help me support the developers working hard on the west coast to build apps. Download the apps (All are free or have a trial) and then review them too… As a developer myself, I love to get ratings and feedback for my games and apps (I included my game on here too :) )

 

Hotel Tonight – Windows 8 – Get a great deal on a hotel TODAY.  I love this app, I am always last minute so it not only helps me get a room last minute but find really cool hotels to stay at. http://apps.microsoft.com/windows/en-us/app/hotel-tonight/2DAEDD08-2405-47F9-8069-4FEEC7BCCD8D

Scribed – Windows 8http://apps.microsoft.com/windows/en-us/app/scribd-read-unlimited-books/b6b0df3c-febf-4b02-b548-7acb4c91b1b1 – If you like to read, try Scribed to read thousands of books for $8.99 a month, I am checking it out using the free trial.

Interactive Touch Books for Kids – Windows 8http://apps.microsoft.com/windows/en-us/app/interactive-touch-books-for-kids/26145e23-b981-4795-a383-66277caa3086 – My daughter loves this, she is in the drag drop stage and wants to play this again and again.

Madagascar Math Ops – Windows 8  http://apps.microsoft.com/windows/en-us/app/madagascar-math-ops/66724460-7712-4241-be7e-eb172db21c66 – I might be starting my daughter a bit early on this but she loves the Madagascar characters.  Try it out and let me know what you think.

Dave In the Cave – Windows Phonehttp://www.windowsphone.com/en-us/store/app/dave-in-the-cave/e26cc5fa-f96d-4dfb-9ac6-d57b9cecac89 – (Shameless Self Promotion) Help Dave Escape the Caves, 20 levels of fun.  The first 10 may be easy but then it starts to ramp up and really makes you think about your next move.  Check it out and let me know what you think.  It is the first game I created.

Baby Connect (Active Logger)http://www.windowsphone.com/en-us/store/app/baby-connect/bd32d247-f3db-4b17-bfdb-7cb5b40639b6 –  If you have a baby you are always keeping track of things, pee, poo, food, sleep, repeat.. This is a great way to not only keep track, but to share with others that help to take care of your little one.

That’s it for now.  Download, rate, and let me know what you think too..  Im proud to show off apps coming out of the west.. keep them coming.

Daniel

 



Creating a Character for your Game Using Inkscape

In todays video session we will be showing you the basics of how to Create a Game Character using Inkscape.  If you have never used Inkscape or think you cant draw, then this session is for you.



Using Spine for GameMaker Sprites

This post is a work in progress to figure out workflow….

 

DaveSpine1DavesSpine2DavesSpine3



GameOn Event Resources

If you attended the GameOn event where we showed you how to build Windows 8 Games using GameMaker Studio, I mentioned a few resources that will help you on your journey to building your first game.  You will find these resources below.

 

GameOn Sample Timeline to get you from start to finish before the June 30th Deadline.

If  you are new to Game Maker and designing games, you may find it helpful to have a rough schedule of how to section out your time so that you can finish your game before the deadline of June 30th.   Since you will need the game to be in the Windows 8 Store by June 30th to be part of the contest you will need to plan accordingly. This gives us 15 days to work with.

 

Day 1: Ideas, Story / Character building,

On this first day, there are some decisions you need to make.

  • What type of game you are going to make (Platformer, Endless Runner, Maze, Physics Game)
  • What is the “Story” of your game.  Who is the good guy, who is the bad guy/s, possible obstacles, etc.
  • Maybe testing code fragments to see how long an idea might take to code

In this phase you want to make sure you are realistic in your idea.  Make sure it is something you can complete in the days required. (Don’t try to recreate DOOM in 5 days)

 

Days 2,3 & possibly 4: Sample artwork,  building the engine, testing everything as you go

In this phase you are working with “placeholder” artwork while you figure out your game.  Whether you download it or create stick figures, you will use this to do testing. In addition,  you will start with possibly one level, making sure characters work as expected (jumping, sliding, running etc) and that characters react the way they should in collisions, that bullets fire correctly (if you have any), that gravity is set correctly, etc.  This is where you will be testing out your gameplay.  This MAY turn into your first level OR just might be a testbed for things, it all depends.  Make sure you have all of the Windows 8 specific features implemented during this section.  Most, if not all, can be copied from many of the starter kits that can be found on the Resource Page.

 

Day 5&6: Assets GFX, sound, etc, which will have been planned in day 1

In this phase you will be creating the assets for your game.

  • Characters
  • Ground
  • Backgounds
  • Sound Effects

Make sure you are creating characters that can be worked on quickly.  Like a 4 image walk cycle instead of a 12, etc… this will keep you within scope

 

 

Days 7, 8 & possibly 9: Level design, building rooms

Once you have built your engine, you will want to start building multiple\levels or rooms.  This of course will depend on the type of game, for an endless runner you would plan on how to make it increasingly difficult and engaging.  You will use the pieces you built in the last section to pump out rooms quickly.   After doing one or two rooms, settle on how many rooms you think you can do in the time allotted (don’t over estimate) because you want to make sure the last room has some closure for the user.  Did they rescue someone, or win a prize, or escape to another world (Where you can add more levels later on with and update)

 

Day 9/10: Testing

This is both, testing for bugs and testing for good gameplay.  Have other people play the game and give you feedback (Incorporate what you have time for).  If you want to use Debug mode you will have to test using “Windows” mode instead of Windows 8 (Native or JS)  this will help you pinpoint and track little oversights by letting you follow variables more closely.

 

Day 10 – 15 Submission to the Windows 8 store.

In this section, you will package up your game (information on resource page) and submit to the Windows 8 store.  You will want to give it pleanty of time so that it is approved and in the store by June 30th.

 

Good Luck and happy Gaming.

 

 



Adding Windows 8 Features to a Game Maker Game

Download Template Here : Win8Template.gmx

As is true for any platform, you want to make sure that you take advantage of the possibilities that the platform gives you.  Building Games for Windows 8 is no different.  In Windows you will sometimes NEED and sometimes WANT to add Windows 8 Functionality.

For example, when a game is put into snap mode (As shown below) you will need to handle this event, pause your game, and show an image in the snap mode.  This is unless you plan on allowing users to play while in snap mode.  I can see that for only a few possible games.  For most you will want to pause.

Screenshot3

 

 

 

 

 

 

 

In addition to this, if you plan on having your application connect to the internet for things like online leader boards. You will need to implement a privacy policy ( which is located on the settings charm).  Your application will fail certification if this is not included.

Screenshot4

 

 

 

 

 

 

 

This, of course goes hand in hand with declaring Internet Use in your Global Game Settings.

Screenshot5

 

 

 

 

 

 

 

 

 

Finally, there are a couple of things you may WANT to implement. First is the Share Charm.  This can be used to share a screenshot, text, URL or other items that can help your app get more exposure by having users share your application with friends.

ScreenShot0

 

 

 

 

 

 

 

And last but not least.  Is implementing an app bar in your game.  You can place many of your Game Commands here if you wish.
ScreenShot1

 

 

 

 

 

 

 

I have included these items in a sample project and have commented it extensively.  If you need help implementing this in your own game or need further clarification on anything in the game, just ask.

Download Template Here : Win8Template.gmx

 

 

 

 

 



Day 5 – Photo A Day May (Zoom)

OK… for those of you that are following this to see some creativity in photography, you may be disappointed in todays picture.

Ever since I looked at the list for May and saw that day 5 was Zoom… I  could not get out of my head the song from a T.V. Show that I used to watch when I was a kid call, what else, Zoom.  It was a poor mans Mickey Mouse Club.  It had a weird language called something like Ubby Wubby Nubby Tubby or something like that and had kids to a bunch of funny skits (well funny to me because I was a kid).

I really couldn’t find a good picture so to keep it in the theme, I took a pic of my screen :)

Day5

 

 

 

 

 

 

 

 

 

 

 

You can see the opening credits here :  http://www.youtube.com/watch?v=I3AzdiWHEuc

It really brought back some memories. Does anyone else remember this?

Tomorrow is High Angle… .So I will get back to the “real” shots.

Daniel



Day 4 – Photo a Day May (Black & White)

Since this day was “Black and White”  it could have really been anything so I took some liberty with this one.

Day4

 

 

 

 

 

 

 

 

 

 

 

Although now that I think of it, I should have done a Starwars themed pic…  Oh Well.

Tomorrow is Zoom…. Zoom, Zoom,  Zoom……

 



Day3 – Photo A Day May (Glass)

It’s getting pretty hot here in Southern California but the mornings have been awesome.  The early morning and late afternoon sun sets the stage perfectly for my Glass picture.  Bonus points if you can tell me what the bottles were in their previous life.

Day3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Tomorrow is Black and White… that leaves it pretty wide open.  We will have to see :)

Daniel

 

Bonus Pics:  I had a little friend visit while I was shooting.  I think he was thirsty.

Day3Bonus

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Day3Bonus2