Method enables machine learning from unwieldy data sets

December 16, 2016

When data sets get too big, sometimes the only way to do anything useful with them is to extract much smaller subsets and analyze those instead.

Those subsets have to preserve certain properties of the full sets, however, and one property that's useful in a wide range of applications is diversity. If, for instance, you're using your data to train a machine-learning system, you want to make sure that the subset you select represents the full range of cases that the system will have to confront.

Last week at the Conference on Neural Information Processing Systems, researchers from MIT's Computer Science and Artificial Intelligence Laboratory and its Laboratory for Information and Decision Systems presented a new algorithm that makes the selection of diverse subsets much more practical.

Whereas the running times of earlier subset-selection algorithms depended on the number of data points in the complete data set, the running time of the new algorithm depends on the number of data points in the subset. That means that if the goal is to winnow a data set with 1 million points down to one with 1,000, the new algorithm is 1 billion times faster than its predecessors.

"We want to pick sets that are diverse," says Stefanie Jegelka, the X-Window Consortium Career Development Assistant Professor in MIT's Department of Electrical Engineering and Computer Science and senior author on the new paper. "Why is this useful? One example is recommendation. If you recommend books or movies to someone, you maybe want to have a diverse set of items, rather than 10 little variations on the same thing. Or if you search for, say, the word 'Washington.' There's many different meanings that this word can have, and you maybe want to show a few different ones. Or if you have a large data set and you want to explore -- say, a large collection of images or health records -- and you want a brief synopsis of your data, you want something that is diverse, that captures all the directions of variation of the data.

"The other application where we actually use this thing is in large-scale learning. You have a large data set again, and you want to pick a small part of it from which you can learn very well."

Joining Jegelka on the paper are first author Chengtao Li, a graduate student in electrical engineering and computer science; and Suvrit Sra, a principal research scientist at MIT's Laboratory for Information and Decision Systems.

Thinking small

Traditionally, if you want to extract a diverse subset from a large data set, the first step is to create a similarity matrix -- a huge table that maps every point in the data set against every other point. The intersection of the row representing one data item and the column representing another contains the points' similarity score on some standard measure.

There are several standard methods to extract diverse subsets, but they all involve operations performed on the matrix as a whole. With a data set with a million data points -- and a million-by-million similarity matrix -- this is prohibitively time consuming.

The MIT researchers' algorithm begins, instead, with a small subset of the data, chosen at random. Then it picks one point inside the subset and one point outside it and randomly selects one of three simple operations: swapping the points, adding the point outside the subset to the subset, or deleting the point inside the subset.

The probability with which the algorithm selects one of those operations depends on both the size of the full data set and the size of the subset, so it changes slightly with every addition or deletion. But the algorithm doesn't necessarily perform the operation it selects.

Again, the decision to perform the operation or not is probabilistic, but here the probability depends on the improvement in diversity that the operation affords. For additions and deletions, the decision also depends on the size of the subset relative to that of the original data set. That is, as the subset grows, it becomes harder to add new points unless they improve diversity dramatically.

This process repeats until the diversity of the subset reflects that of the full set. Since the diversity of the full set is never calculated, however, the question is how many repetitions are enough. The researchers' chief results are a way to answer that question and a proof that the answer will be reasonable.
-end-
Additional background

PAPER: Fast mixing Markov chains for strongly Rayleigh measures, DPPs, and constrained sampling

ARCHIVE: Making big data manageable

ARCHIVE: "Shrinking bull's-eye" algorithm speeds up complex modeling from days to hours

ARCHIVE: To handle big data, shrink it

ARCHIVE: Collecting just the right data

Massachusetts Institute of Technology

Related Diversity Articles from Brightsurf:

More plant diversity, less pesticides
Increasing plant diversity enhances the natural control of insect herbivory in grasslands.

Insect diversity boosted by combination of crop diversity and semi-natural habitats
To enhance the number of beneficial insect species in agricultural land, preserving semi-natural habitats and promoting crop diversity are both needed, according to new research published in the British Ecological Society's Journal of Applied of Ecology.

Ethnolinguistic diversity slows down urban growth
Where various ethnic groups live together, cities grow at a slower rate.

Protecting scientific diversity
The COVID-19 pandemic means that scientists face great challenges because they have to reorient, interrupt or even cancel research and teaching.

Cultural diversity in chimpanzees
Termite fishing by chimpanzees was thought to occur in only two forms with one or multiple tools, from either above-ground or underground termite nests.

Bursts of diversity in the gut microbiota
The diversity of bacteria in the human gut is an important biomarker of health, influences multiple diseases, such as obesity and inflammatory bowel diseases and affects various treatments.

Underestimated chemical diversity
An international team of researchers has conducted a global review of all registered industrial chemicals: some 350,000 different substances are produced and traded around the world -- well in excess of the 100,000 reached in previous estimates.

New world map of fish genetic diversity
An international research team from ETH Zurich and French universities has studied genetic diversity among fish around the world for the first time.

Biological diversity as a factor of production
Can the biodiversity of ecosystems be considered a factor of production?

Fungal diversity and its relationship to the future of forests
Stanford researchers predict that climate change will reduce the diversity of symbiotic fungi that help trees grow.

Read More: Diversity News and Diversity Current Events
Brightsurf.com is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to Amazon.com.