Saturday, April 10, 2021

Fact Check

"Sunday was never used before as a voting day in Georgia.”

Behind a sign listing early voting dates for Fulton County, Georgia, a line forms outside of the State Farm Arena for early voting on...

“Report: Biden administration busing ‘thousands’ of illegals to red states to get permanent status.”

A migrant family crosses the border into El Paso, Texas, in Ciudad Juarez, Mexico, on Feb. 26, 2021. (AP)Source link

Says Georgia “has more day-of voting rights than Colorado.”

A voter casts a ballot in a dropbox Nov. 3, 2020, in Denver. (AP Photo/David Zalubowski)Source link

No direct link between ‘gun control’ and killings by authoritarian regimes

If Your Time is shortThere is no evidence that suggests “gun control” was a causal factor in these events.See the sources for this fact-checkA...

"Over the past decade, drug prices have risen at three times the rate of inflation."

Prescription drugs at a hospital dispensary. (AP)Source link

NOT REAL NEWS: A look at what didn’t happen this week

A roundup of some of the most popular but completely untrue stories and visuals of the week. None of these are legit, even though...

US Politics
Latest

Rep. Crenshaw says he will be temporarily blind after emergency eye surgery

Crenshaw said the news was “terrifying” and the prognosis “very bad.” “Anyone who knows the history of my injuries knows that I don’t have...

‘Election integrity’ becomes key buzzword in VA GOP race

‘Election integrity’ becomes key buzzword in VA GOP race Source link

Republicans trod well-worn path to court to sue Biden

WASHINGTON (AP) — These are busy days for Republican state attorneys general, filing repeated lawsuits that claim President Joe Biden and his administration are...

Wisconsin governor seeks legal fees from Trump, Republicans

MADISON, Wis. (AP) — Wisconsin Democratic Gov. Tony Evers is seeking more than $250,000 in legal fees from former President Donald Trump and a...

Lawsuits over execution of 2020 census near conclusions

Two legal challenges to the Trump administration’s execution of the 2020 census neared conclusions this week in the face of changes brought by President...

LGBTQ

A clinical trial for a HIV vaccine has been an overwhelming success / LGBTQ Nation

A newly developed vaccine candidate for HIV recently completed its first clinical, in-human trial, and the results are being widely regarded as exciting and...

Lil Nas X’s “Satan shoes” are being recalled. Do they really think people will send them back? / LGBTQ Nation

Lil Nas X with the Satan shoesPhoto: Screenshot/YouTubeLil Nas X’s “Satan shoes” are being voluntarily recalled as part of a settlement between Nike and...

Peacock is rebooting “Queer As Folk” for an 8 episode series / LGBTQ Nation

The cast of Queer As FolkPhoto: promo photoQueer As Folk is getting a third rendition as a television series. Eight episodes of a reboot created...

Tech

UK Deliveroo riders strike over pay, gig work conditions

LONDON (AP) — Riders for the app-based meal delivery platform Deliveroo held a strike in London Wednesday over pay and working conditions, part of...

Hackers, glitches mar France’s mass return to online school

PARIS (AP) — French children, parents and teachers are battling with internet connection problems across the country after an abrupt nationwide switch to online...

AI conferences use AI to assign papers to reviewers

Must read

"Sunday was never used before as a voting day in Georgia.”

Behind a sign listing early voting dates for Fulton County, Georgia, a line forms outside of the State Farm Arena for early voting on...

Rep. Crenshaw says he will be temporarily blind after emergency eye surgery

Crenshaw said the news was “terrifying” and the prognosis “very bad.” “Anyone who knows the history of my injuries knows that I don’t have...

UK Deliveroo riders strike over pay, gig work conditions

LONDON (AP) — Riders for the app-based meal delivery platform Deliveroo held a strike in London Wednesday over pay and working conditions, part of...

‘Election integrity’ becomes key buzzword in VA GOP race

‘Election integrity’ becomes key buzzword in VA GOP race Source link

The Conference on Neural Information Processing Systems, held in 2019 in Vancouver, Canada, is the largest in the discipline of artificial intelligence.

Khari Johnson/VentureBeat

Artificial intelligence (AI) researchers are hoping to use the tools of their discipline to solve a growing problem: how to identify and choose reviewers who can knowledgeably vet the rising flood of papers submitted to large computer science conferences.

In most scientific fields, journals act as the main venues of peer review and publication, and editors have time to assign papers to appropriate reviewers using professional judgment. But in computer science, finding reviewers is often by necessity a more rushed affair: Most manuscripts are submitted all at once for annual conferences, leaving some organizers only a week or so to assign thousands of papers to a pool of thousands of reviewers.

This system is under strain: In the past 5 years, submissions to large AI conferences have more than quadrupled, leaving organizers scrambling to keep up. One example of the workload crush: The annual AI Conference on Neural Information Processing Systems (NeurIPS)—the discipline’s largest—received more than 9000 submissions for its December 2020 event, 40% more than the previous year. Organizers had to assign 31,000 reviews to about 7000 reviewers. “It is extremely tiring and stressful,” says Marc’Aurelio Ranzato, general chair of this year’s NeurIPS. “A board member called this a herculean effort, and it really is!”

Fortunately, they had help from AI. Organizers used existing software, called the Toronto Paper Matching System (TPMS), to help assign papers to reviewers. TPMS, which is also used at other conferences, calculates the affinity between submitted papers and reviewers’ expertise by comparing the text in submissions and reviewers’ papers. The sifting is part of a matching system in which reviewers also bid on papers they want to review.

But newer AI software could improve on that approach. One newer affinity-measuring system, developed by the paper-reviewing platform OpenReview, uses a neural network—a machine learning algorithm inspired by the brain’s wiring—to analyze paper titles and abstracts, creating a richer representation of their content. Several computer science conferences, including NeurIPS, will begin to use it this year in combination with TPMS, say Melisa Bok and Haw-Shiuan Chang, computer scientists at OpenReview and the University of Massachusetts, Amherst.

AI conference organizers hope that by improving the quality of the matches, they will improve the quality of the resulting peer reviews and the conferences’ published literature. A 2014 study suggests there’s room for progress: As a test, 10% of papers submitted to NeurIPS that year were reviewed by two sets of reviewers. Of papers accepted by one group, the other group accepted only 57%. Many factors could explain the discrepancy, but one possibility is that at least one panel for each paper lacked sufficient relevant expertise to evaluate it.

To promote good matches, Ivan Stelmakh, a computer scientist at Carnegie Mellon University, developed an algorithm called PeerReview4All. Typically, a matching system maximizes the average affinity between papers and reviewers, even if it means some papers get really well matched reviewers and others unfairly get poorly matched reviewers. PeerReview4All instead maximizes the quality of the least good match, with an eye toward avoiding poor matches and increasing fairness.

Last year, Stelmakh experimented with using PeerReview4All at the International Conference on Machine Learning (ICML), and reported results in February at another, the Association for the Advancement of Artificial Intelligence (AAAI) conference. The method improved fairness significantly without harming average match quality, he concluded. OpenReview has also begun to offer a system aimed at increasing fairness, called FairFlow. NeurIPS will try at least one of these this year, says Alina Beygelzimer, a computer scientist at Yahoo and the NeurIPS 2021 senior program chair. “NeurIPS has a long history of experimentation.”

These systems all match a known set of papers to a known set of reviewers. But as the field grows, it will need to recruit, evaluate, and train new reviewers, conference organizers say. A recent experiment led by Stelmakh explored one way, which did not rely on AI, to ease those tasks. At last year’s ICML, he and collaborators used emails and word of mouth to invite students and recent graduates to review unpublished papers collected from colleagues; 134 agreed. Based on evaluations of those reviews, the team invited 52 to join the ICML reviewer pool and assigned them a senior researcher who acted as a mentor. In the end, the novice’s ICML reviews were at least as good as those of seasoned reviewers, as judged by metareviewers, Stelmakh reported at the AAAI meeting. He says organizers could potentially scale up the process to recruit hundreds of reviewers without too much burden. “There was a lot of enthusiasm from candidate reviewers” who participated in the experiment, Stelmakh says.

Matching systems that use affinity to measure reviewer expertise also let prospective reviewers bid on papers to review, and some recent work has attempted to address potential bias in this approach. Researchers have heard tales of bidders picking only their friends’ papers, essentially hacking the algorithm. A preprint posted on the arXiv server in February describes a countermeasure that uses machine learning to filter out suspicious bids. On a simulated data set, it reduced manipulation–even when potential cheaters knew how the system operated—without reducing match quality. Another algorithm, presented at NeurIPS last year, essentially penalizes bidders who place bids on papers outside of their field of expertise; researchers demonstrated the method’s effectiveness in reducing manipulation using a combination of simulated bids and real data from a previous conference.

One problem with the tools is that it’s difficult to evaluate how much they outperform alternative methods in real-world settings. Hard evidence would require controlled trials, but there have been none, says Laurent Charlin, a computer scientist at the University of Montreal. In part, that’s because many of these tools are new.

As they evolve, methods like these could also one day help journal editors outside computer science find peer reviewers—but so far uptake has been limited, says Charlin, who developed the TPMS affinity-measuring tool about 10 years ago. (Meagan Phelan, a spokesperson for AAAS, which publishes the Science family of journals, says they do not use AI in assigning peer reviewers.)

But in AI, Charlin says, “We are quite comfortable as a field with some level of automation. We have no reason not to use our own tools.”

Source link

- Advertisement -

More articles

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest article

"Sunday was never used before as a voting day in Georgia.”

Behind a sign listing early voting dates for Fulton County, Georgia, a line forms outside of the State Farm Arena for early voting on...

Rep. Crenshaw says he will be temporarily blind after emergency eye surgery

Crenshaw said the news was “terrifying” and the prognosis “very bad.” “Anyone who knows the history of my injuries knows that I don’t have...

UK Deliveroo riders strike over pay, gig work conditions

LONDON (AP) — Riders for the app-based meal delivery platform Deliveroo held a strike in London Wednesday over pay and working conditions, part of...

‘Election integrity’ becomes key buzzword in VA GOP race

‘Election integrity’ becomes key buzzword in VA GOP race Source link

US judge tosses claims by Chinese-born professor over arrest

PHILADELPHIA (AP) — Lawyers for a Temple University professor said Friday they plan to appeal a federal court ruling that threw out most of...