Endre Pálfi blogs http://endrepalfi.com/blog en Big Data Technologies http://endrepalfi.com/blogs/big-data-technologies <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>Here is a brief overview of the leading open source Big Data technologies currently available on the market and their functions.</p> <p><strong>Hadoop</strong> - A scalable distributed file system that a great majority of the giant media companies use nowadays. It uses mapping and reducing algorithms to process search requests by first parallel mapping (searching) the data across the cluster and then reducing the results by merging the findings into a finished table. Hadoop has a new type of file system called HDFS (similar to the proprietary Google File System or GoogleFS). HDFS is a highly fault-tolerant scalable file system written in Java. It normally sits on a bunch (can be thousands) of inexpensive computers with cheap drives, with files and directories scattered everywhere and operates as a cluster. HDFS can support up to 4500 servers and 200 petabyte addressible file space for a partition. A petabyte being about 1000 terrabytes (TBs), that is a total capacity of about 200,000 TBs or 200 million gigabytes (GBs). The MapReduce framework functionality is able to search the entire cluster in milliseconds and locate the file or data. Access to the HDFS file system is with the use of specialized hdfs commands (Ex. "hadoop fs -mkdir /user/hadoop/dir1 /user/hadoop/dir2" or "hadoop fs -ls /user/hadoop/dir1/filename.txt").</p> <p>There is of course a lot more to Big Data than a cluster of file servers running a virtual file system. Below are the most popular Big Data database technologies and concepts that I'm familiar with, but first let's clear up what Big Data databases are about. Big Data databases are primarily focused on storing non-relational data. NoSQL  is a concept that describes the process of storing non-relational data. The various data types are, document (ex. MongoDB, CouchDB), graph, key and value (ex. Riak), and wide-column hybrid (ex. Cassandra).  At the highest level in a big data system all data is in the form of keys and values, where they keys are the indexes and the values are complex structures (documents, hashes, graphs, etc).<br /> </p> <p>The various leading database technologies (in no particular order) are:</p> <ul><li><strong>NoDB</strong> - A NoSQL Java API that creates in-memory eventual consistency (EC) object repository for an application to use. Basically you can store, update, delete, and retrieve anything that the application has access to from any source as long as it can be stored in a Java object. Capacity and performance depends on the machine or cloud platform it's on. The objects are defined by the application commonly as classes with serialized values.<strong> </strong></li> <li><strong>Riak KV</strong> - A distributed NoSQL Key and Value store database. Similar to Cassandra just much simpler, offers fault tolerance, high availability, and scalability.<strong></strong></li> <li><strong>Riak TS</strong> - Same ideas as Riak KV, just used for time series data.<strong></strong></li> <li><strong>Riak S2 (Riak CS)</strong> - Same ideas as Riak KV and Riak TS, but geared toward large object storage such as video, software packages, backups, etc.<strong></strong></li> <li><strong>Cassandra</strong> - A distributed database system with a hybrid design that uses both relational and non-relational data concepts for its data model. The servers are distributed with high availability, fault tolerance, and high performance across the cluster. Intended uses are large number of low cost servers containing massive amounts of data scattered or clustered over large geographical regions. Supports clusters and data replication across multiple datacenters with very high best in class throughput. Nodes are identical in role so there is no single point of failure in the cluster. Performance and capacity goes up linearly with each new node added. The eventual consistency levels (essentially the QOS) are adjustable. Integrates with Hadoop.<strong></strong></li> <li><strong>CouchDB</strong> - NoSQL document database like MongoDB. Stores data as JSON documents. API calls are made using HTTP and the actual queries are done using JavaScript.<strong></strong></li> <li><strong>MongoDB</strong> - Most popular NoSQL document database. Uses dynamic schemas. Can be clustered and provides high availability replicating across multiple servers. Load balances using sharding where each shard contains a part of the data and consists of a master and multiple slave components. Anything such as JSON, XML, etc is supported. Can be used as a load balanced grid file system (GridFS) also. MongoDB GridFX implementations are used in NGINX and lighthttp web servers.<strong></strong></li> <li><strong>Pig</strong> - Used to run Hadoop MapReduce programs written in Pig Latin.</li> <li><strong>Pig Latin</strong> - programming language used to write Pig programs, essentially search functions. Integrates (can call) routines from common programming languages like Java, Python, and Java Script.<strong></strong></li> <li><strong>Apache Hive</strong> - Distributed database platform built into the Hadoop platform. Supports batch querying of text files (flat files), Key and Value pair format files (SequenceFiles), and Record Columnar Files (RCFiles). It's intended use is storing and managing large data sets located in distributed storage on the Hadoop File System (HDFS). Used the Hive query language HiveQL (simillar to SQL) which are compiled into MapReduce jobs. Ideal uses would be storing archive data of time series (ex. Stock prices) and running analysis on the data set frequently or storing tons of log files from a large number of sources for analytical or archival purposes. It's not intended for Online Transaction Processing (OLTP) applications.<strong></strong></li> <li><strong>Apache HBase</strong> - Just like Hive, Apache HBase is also made for the Hadoop platform and since it's on the HDFS it's also distributed, scalable, and fault tolerant. Unlike Hive which is intended more as a storage, and batch processor of data analysis sorts of jobs, HBase is made for real-time querying big data similarly to the way SPLUNK works, by storing data in massive tables with billions of records, and querying the tables frequently in a matter of seconds. HBase is modeled after Google's proprietary BigTable data storage system which Google runs on its own distributed Google File System (GFS) applications. Essentially Apache HBase is the big transactional database system for Hadoop the way BigTable is for the Google systems running GFS. HBase is intended to be used for data tables that contain upwards toward a billion or more records. Anything less and you're better off with an RDBMS system like MySQL or Oracle since the querying and reporting capacity of HBase isn't as advanced as the existing RDBMS systems out there.</li> </ul></div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/big-data-technologies"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/big-data-technologies%20Big%20Data%20Technologies%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Thu, 10 Dec 2015 21:56:15 +0000 endre 169 at http://endrepalfi.com http://endrepalfi.com/blogs/big-data-technologies#comments To Hadoop or not to Hadoop the enterprise? http://endrepalfi.com/blogs/hadoop-or-not-hadoop-enterprise <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>Got money for cheap computers and need a highly scalable Big Data storage system? Your answer might just be Hadoop!</p> <p><object class="video-filter video-youtube vf-xwgdny19yq4" type="application/x-shockwave-flash" width="580" height="326" data="http://www.youtube.com/v/xWgdny19yQ4&amp;fs=1&amp;rel=0"> <param name="movie" value="http://www.youtube.com/v/xWgdny19yQ4&amp;fs=1&amp;rel=0" /><param name="wmode" value="transparent" /><param name="allowFullScreen" value="true" /></object> </p></div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/hadoop-or-not-hadoop-enterprise"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/hadoop-or-not-hadoop-enterprise%20To%20Hadoop%20or%20not%20to%20Hadoop%20the%20enterprise%3F%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Fri, 19 Sep 2014 06:18:56 +0000 endre 168 at http://endrepalfi.com http://endrepalfi.com/blogs/hadoop-or-not-hadoop-enterprise#comments Farewell to the hard-driving days of Digital Insight http://endrepalfi.com/blogs/farewell-hard-driving-days-digital-insight <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>It's been a month now since I parted ways with <a href="http://www.endrepalfi.com/portfolio-company/intelliswift-intuit-digital-insight-ncr">Digital Insight</a> and I'm finally getting around to dropping some notes to my blog. Haven't had really any time off since I started there so it was refreshing to take a break, which wasn't really all that much of a break because it involved moving and sorting out my living space. Whatever it was, it wasn't like the previous 18 months, and things were definitely much less stressful. But as expected the bills keep coming and it's time to start taking the job search seriously before I begin falling behind, which is never a good thing. For me it seems Labor Day marked the end of this vacation which came way too soon and now it's time to start going hard in the paint again with the job search. Do you know any good businesses that are hiring or have a specific job opportunity in mind? Feel free to contact me, I'd like to hear about it.</p> </div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/farewell-hard-driving-days-digital-insight"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/farewell-hard-driving-days-digital-insight%20Farewell%20to%20the%20hard-driving%20days%20of%20Digital%20Insight%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Wed, 03 Sep 2014 08:14:32 +0000 endre 167 at http://endrepalfi.com http://endrepalfi.com/blogs/farewell-hard-driving-days-digital-insight#comments Stock pumping explained http://endrepalfi.com/blogs/stock-pumping-explained <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>The other day my mother brought a pink sheet or OTC stock to my attention saying she is considering investing and how the price of this stock is going through the roof. First I just tried to tell her this type of investment with no or very low liquidity is for suckers and experts. OTC companies have no or little financial reporting requirements. If you're dumb enough to be suckered into investing into one without verifying what is behind the claims made then it's your fault if you loose your money. On the other hand if you're an expert investor with the tools and the time to do your due dilligence and verify everything then you can find some good deals and new upcoming businesses with fine management and great products.</p> <p>I was hoping that just telling her that would discourage her sufficiently to leave the idea alone and forget about it completely. As it turns out I was not so fortunate. Today I wake to get a message from her that says I should look into it again for her.</p> <p>So I'm like, Okay. Let's take a second look. Here is what I see.</p> <p><img src="/sites/default/files/big.chart-TAGG%20-%2010_25_12.gif" alt="October 25, 2012" title="October 25, 2012" width="550" height="318" /></p> <p>Most people don't see what's going on here, but some will notice the abrupt change in volume that coincides with change in price. Okay basically this situation of having no volume and then big volume and price growth on a microcap stock is the telltale sign of the stock being advertised (pumped) heavily to unsophisticated buyers that fall for the break out claims which are being made. The activity of false advertising and making phoney claims itself is illegal and regulated, but at the end of the day it's Buyer Beware!</p> <p>There is another opinion on this activity that I found on the Internet, which explains what is going on and demonstrates how stocks usually collapse after the pumping is done and how the investors that got dumped to loose the value of their investments quite quickly. It's by Kaloyan Gadzhev and the article is called <a href="http://www.hotstocked.com/article/39872/taglikeme-corp-pink-tagg-pump-finally-heats-up.html" target="_blank">TagLikeMe Corp. (PINK: TAGG) Pump Finally Heats Up</a>. He's more familiar with the actual situation than me, but it verifies my point, which is that this volume is highly pumped up.</p> </div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/stock-pumping-explained"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/stock-pumping-explained%20Stock%20pumping%20explained%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Fri, 26 Oct 2012 06:39:44 +0000 endre 161 at http://endrepalfi.com http://endrepalfi.com/blogs/stock-pumping-explained#comments Insurance Sales http://endrepalfi.com/blogs/insurance-sales <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>I'm thinking about getting into insurance sales. It sounds pretty lucrative and there are some big names such as Prudential, Aflac, New York Life, and TransAmerica that have been trying to get me on board to sell their products, but it's straight commission which as a sales person it's lucrative, but how will that cover my bills this month or the next? My current dilemma...</p> </div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/insurance-sales"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/insurance-sales%20Insurance%20Sales%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Wed, 17 Oct 2012 03:29:42 +0000 endre 160 at http://endrepalfi.com http://endrepalfi.com/blogs/insurance-sales#comments Ad System Upgrade http://endrepalfi.com/blogs/ad-system-upgrade <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>Ads will be down due to an upgrade for 12-24 hours accross all sites. Doing a clean install that should clear up some of the existing delivery issues we had.</p> </div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/ad-system-upgrade"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/ad-system-upgrade%20Ad%20System%20Upgrade%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Wed, 17 Oct 2012 03:23:28 +0000 endre 159 at http://endrepalfi.com http://endrepalfi.com/blogs/ad-system-upgrade#comments Job hunting http://endrepalfi.com/blogs/job-hunting <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>Hustling for work now more than ever because rent will be due soon, the fridge is almost empty, the Caddy needs gasolina, and Uncle Sam is convinced that it is time to begin making student loan payments again, or they gonna roll me an interest snowball the size of the moon, with its own gravitational pull. Oh yeah. But, let's look on the bright side, shall we? I have some great friends and business partners that are being completely supportive of me through these days, the Russel 3000 is at Y2K levels, unemployment is dropping in Cali, and real estate prices are still down. Plus we have an election year which means everyone is looking at the road ahead, and concerned with finding solutions to real life problems. Hmmmmm.... Who is going to be the next president I wonder...? Romney, Obama, Paul, me? I'm being facetious of course. But who knows? Do people get hit by lightning? According to NOAA, about 360 people in the US are hit by lightning every year. So miracles can happen. Not exactly the type of miracle one is looking for, but I think it helps drive the point home, that much stranger things have happened before than finding good employment on a seemingly short notice.</p> </div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/job-hunting"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/job-hunting%20Job%20hunting%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Sun, 24 Jun 2012 05:41:32 +0000 endre 158 at http://endrepalfi.com http://endrepalfi.com/blogs/job-hunting#comments Gal Holiday and the Honky Tonk Revue @ Joe's - Burbank, CA - January 4, 2012 http://endrepalfi.com/blogs/gal-holiday-and-honky-tonk-revue-joes-burbank-ca-january-4-2012 <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>A small contribution to my friend Vanessa (Gal Holiday) and Dave's band marketing efforts.</p> <p><object class="video-filter video-youtube vf-3kgjak84ope" type="application/x-shockwave-flash" width="580" height="326" data="http://www.youtube.com/v/3kGjAK84opE&amp;fs=1&amp;rel=0"> <param name="movie" value="http://www.youtube.com/v/3kGjAK84opE&amp;fs=1&amp;rel=0" /><param name="wmode" value="transparent" /><param name="allowFullScreen" value="true" /></object> <br /><span class="long-title"></span></p></div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/gal-holiday-and-honky-tonk-revue-joes-burbank-ca-january-4-2012"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/gal-holiday-and-honky-tonk-revue-joes-burbank-ca-january-4-2012%20Gal%20Holiday%20and%20the%20Honky%20Tonk%20Revue%20%40%20Joe%27s%20-%E2%80%A6%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Wed, 13 Jun 2012 03:44:56 +0000 endre 157 at http://endrepalfi.com http://endrepalfi.com/blogs/gal-holiday-and-honky-tonk-revue-joes-burbank-ca-january-4-2012#comments Gal Holiday and the Honky Tonk Revue @ Viva Cantina, Burbank, CA - January 5, 2012 http://endrepalfi.com/blogs/gal-holiday-and-honky-tonk-revue-viva-cantina-burbank-ca-january-5-2012 <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p><object class="video-filter video-youtube vf-24chrv1mo" type="application/x-shockwave-flash" width="580" height="326" data="http://www.youtube.com/v/-24_CHRv1mo&amp;fs=1&amp;rel=0"> <param name="movie" value="http://www.youtube.com/v/-24_CHRv1mo&amp;fs=1&amp;rel=0" /><param name="wmode" value="transparent" /><param name="allowFullScreen" value="true" /></object> </p></div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/gal-holiday-and-honky-tonk-revue-viva-cantina-burbank-ca-january-5-2012"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/gal-holiday-and-honky-tonk-revue-viva-cantina-burbank-ca-january-5-2012%20Gal%20Holiday%20and%20the%20Honky%20Tonk%20Revue%20%40%E2%80%A6%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Wed, 13 Jun 2012 03:43:36 +0000 endre 156 at http://endrepalfi.com http://endrepalfi.com/blogs/gal-holiday-and-honky-tonk-revue-viva-cantina-burbank-ca-january-5-2012#comments RayBones - Let Me Know - Video http://endrepalfi.com/blogs/raybones-let-me-know-video <div class="field field-name-body field-type-text-with-summary field-label-hidden"><div class="field-items"><div class="field-item even" property="content:encoded"><p>My college buddy Raymond rapping a solo on this video. I didn't create the video, but it's close enough to home that makes it post worthy.</p> <p><object class="video-filter video-youtube vf-9mmsnficta" type="application/x-shockwave-flash" width="580" height="326" data="http://www.youtube.com/v/9m_MSNFictA&amp;fs=1&amp;rel=0"> <param name="movie" value="http://www.youtube.com/v/9m_MSNFictA&amp;fs=1&amp;rel=0" /><param name="wmode" value="transparent" /><param name="allowFullScreen" value="true" /></object> </p></div></div></div><ul class="links inline"><li class="fb_social_like_fb_like_default first"><span><div class="fb_social-like-plugin"><fb:like layout="button_count" show_faces="1" width="25" action="like" font="verdana" colorscheme="light" href="http://endrepalfi.com/blogs/raybones-let-me-know-video"></fb:like></div></span></li> <li class="tweet_Twitter last"><a href="http://twitter.com/home?status=http%3A//endrepalfi.com/blogs/raybones-let-me-know-video%20RayBones%20-%20Let%20Me%20Know%20-%20Video%20" class="tweet" rel="nofollow" target="_blank">Tweet</a></li> </ul> Tue, 02 Aug 2011 20:39:29 +0000 endre 154 at http://endrepalfi.com http://endrepalfi.com/blogs/raybones-let-me-know-video#comments