homehome Home chatchat Notifications


'Data Smashing' algorithm might help declutter Big Data noise without Human Intervention

There’s an immense well of information humanity is currently sitting on and it’s only growing exponentially. To make sense of all the noise, whether we’re talking about apps like speech recognition, cosmic body identification or search engine results, highly complex algorithms that use less processing power by hitting the bull’s eye or as close as […]

Henry Conrad
October 29, 2014 @ 8:34 am

share Share

There’s an immense well of information humanity is currently sitting on and it’s only growing exponentially. To make sense of all the noise, whether we’re talking about apps like speech recognition, cosmic body identification or search engine results, highly complex algorithms that use less processing power by hitting the bull’s eye or as close as possible are warranted. In the future, such algorithms will be comprised of machine learning technology that gets smarter and smarter after each information parse; this will most likely employ quantum computing as well. Until then, we have to make use of conventional algorithms and a most exciting paper detailing such a technique was recently reported.

Smashing data – the bits and pieces that follow are the most important

Big Data

Credit: 33rd Square

Called ‘data smashing’, the algorithm tries to fix one major flaw in today’s information processing. Immense amounts of data are currently being fed in and while algorithms help us declutter, at the end of the day companies and governments still need experts to oversee the process and grant a much need human fine touch. Basically, computers are still pretty bad at solving complex patterns. Sure, they’re awesome for crunching the numbers, but in the end, humans need to compare the outputted scenarios and pick out the most relevant answer. As more and more processes are being monitored and fed into large data sets, however, this task is becoming ever more difficult and human experts are in low supply.

[ALSO READ] Breakthrough in computing: brain-like chip features 4096 cores, 1 million neurons, 5.4 billion transistors

The algorithm, developed by Hod Lipson, associate professor of mechanical engineering and of computing and information science, and Ishanu Chattopadhyay, a former postdoctoral associate with Lipson now at the University of Chicago, is nothing short of brilliant. It works by estimating the similarities between streams of arbitrary data without human intervention, and even without access to the data sources.

Basically, data is being ‘smashed’ with one another to tease out unique information by measuring what remains after each ‘collision’. The more info stands, the less likely it is it originated from the same streams.

Data smashing could open doors to a new body of research – it’s not just helping experts sort through data easier, it might also actually identify anomalies that are impossible to spot by humans in virtue of pure computing brute force. For instance, the researchers demonstrated data smashing using data from real-world problems, including detection of anomalous cardiac activity from heart recordings and classification of astronomical objects from raw photometry. Results showed that the info was on par with the accuracy of specialized algorithms and heuristics tweaked by experts to work.

share Share

Drinking Coffee at Night Could Be Making You More Impulsive and Reckless

The implications are especially important for people who work overnight shifts.

A Century-Old Lung in a Jar Yields Clues to the Spanish Flu’s Lethal Surge

Scientists decode how the 1918 flu rapidly adapted to humans—much earlier than thought.

This Common Ingredient in Chocolate May Outperform Tamiflu Against the Flu In New Drug Combo

Researchers uncover a potent, resistance-proof flu treatment—starting with bacteria and ending in mice.

Why Are Some Doctors Pretending to Do CPR? You Should Know About 'Slow Code'

Although it sounds wrong, performative CPR is sometimes the most humane thing to do.

Scientists Created a 3D Printing Resin You Can Reuse Forever

The new resin can be reused indefinitely without losing strength or quality.

Did Neanderthals Survive the Ice Age by Eating Rotting Meat and Maggots?

You may find it hard to digest, but Neanderthals may have loved their meat rotten, and full of maggots.

What Would Happen If Everyone in the World Turned On The Lights At the Same Time?

Power grids could likely handle the surge of demand, but all that light would pollute dark zones nearby.

A Massive Fraud Ring Is Publishing Thousands of Fake Studies and the Problem is Exploding. “These Networks Are Essentially Criminal Organizations”

Organized misconduct is rapidly poisoning the global scientific record.

Scientists Spied on Great Tits All Winter and Caught Them Drifting Apart Toward Divorce

Bird couples drift apart long before they split, Oxford study finds.

A Digital Artist Rebuilt the Shroud of Turin. Turns Out The Shroud Might Not Show a Real Body at All

New 3D analysis suggests the Shroud of Turin was imprinted from sculpture, not a human body.