InfoQ Homepage Big Data Content on InfoQ
-
Yahoo Open-Sources DataSketches for Faster Operations Over Streams
Yahoo has open-sourced DataSketches, a library written in Java for stochastic streaming algorithms. DataSketches is able to perform traditionally expensive operations, like counting distinct occurrences of a variable within a stream, using a fraction of time and memory and with a predictable error margin.
-
Riley Newman on How Airbnb Uses Data Science
Riley Newman, head of data science at Airbnb, recently published an article describing how the Californian startup defines and uses data science. He explains that data can be seen as the voice of the customers, and data science as an act of interpretation. He also details several initiatives that have been particularly important for scaling data science.
-
MongoDB Hits 3.2 and Becomes Enterprise Ready
MongoDB recently announced the newest version of its NoSQL database synonymous product. Building upon the new features introduced in 3.0 release, 3.2 is expanding and solidifying MongoDB’s interest towards the corporate world.
-
IBM Commits to Advance Apache Spark
Earlier last month in Las Vegas, at IBM Insight 2015, IBM announced a major commitment to the Apache Spark project. Referring to it as “potentially the most significant open source project of the next decade” tells a lot about how important IBM believes Apache Spark is. With IDC reporting that 80% of cloud applications in the future will be data intensive, Apache Spark can unlock previously...
-
DMTK, a Machine Learning Toolkit from Microsoft
About the same time Google announced open sourcing TensorFlow, Microsoft has pushed to GitHub DMTK, a Distributed Machine Learning Toolkit. While Google has released a one-machine version of TensorFlow, DMTK runs on a cluster of machines.
-
TensorFlow: Google Open Sources Their Machine Learning Tool
TensorFlow is a machine learning library created by the Brain Team researchers at Google and now open sourced under the Apache License 2.0. TensorFlow is detailed in the whitepaper TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. The source code can be found on Google Git.
-
Teradata Announces New Software for Real-Time Analysis of Internet of Things Data
At its 2015 Partners User Group Conference, Teradata announced two new software capabilities for real-time ingestion and analysis of massive streams of IoT data. While the Teradata Listener software enables "listening" to multiple, diverse IoT data streams in real time, the new Teradata Aster Analytics on Hadoop software provides scalable analysis of massive IoT data streams.
-
DistributedLog at Twitter for High Performance Logging
Twitter is using replicated logs for high performance data collection and analysis of its systems. DistributedLog is the system developed at Twitter for this purpose. Twitter has developed a distributed key-value database, Manhattan. Manhattan can trade consistency for latency in reads following the eventually consistent data model. We examine Twitter's design and tradeoffs for DistributedLog.
-
Amazon Announces QuickSight - Business Intelligence for Big Data on AWS
Amazon has announced QuickSight at AWS Re:invent conference. QuickSight a complete Business Intelligence solution to help customers gain insights from the data they have stored in AWS.
-
Salesforce Enters IoT Market
At Salesforce’s recent Dreamforce conference, the company announced an upcoming IoT platform that will allow for the ingestion of real time data and turn it into actionable tasks across its suite of cloud based services.
-
Hortonworks Addresses the IoAT with DataFlow Based on NiFi
Hortonworks has quietly made available the DataFlow platform which is based on Apache NiFi and attempts to solve the processing needs of the IoAT.
-
SpringXD being Re-architected and Re-branded to Spring Cloud Data Flow
Pivotal announced a complete re-design of Spring XD, its big data offering, during last week’s SpringOne2GX conference, with a corresponding re-brand from Spring XD to Spring Cloud Data Flow. The new product is focussed on orchestration.
-
Splunk for DBAs
The DBA’s primary job is to ensure that the business’s information is always available, with performance coming in at close second. We’ve already talked about optimizing distributed queries in Splunk and map-reduce queries in Hunk. In this report we expand upon that with more information that a DBA needs to know about Splunk databases.
-
Optimizing Distributed Queries in Splunk
Optimizing queries in Splunk’s Search Processing Language is similar to optimizing queries in SQL. The two core tenants are the same: Change the physics and reduce the amount of work done. Added to that are two precepts that apply to any distributed query.
-
Big Data Architecture: Push, Pull, or Search in Place?
A surprisingly common theme at the Splunk Conference is the architectural question, “Should I push, pull, or search in place?”