A brief preamble: please to note that these posts are about how I do PW stuff. Other editors may work very differently.
I was going to do starred reviews next, but it makes more sense to first talk about how I decide which books to review, since that's the first level of quality control.
The simplest criterion is this one: I have to get two copies of the galley, three months or more before the title is due to be published. If books aren't sent to me, I don't review them. If books are sent late, I don't review them (with a very few exceptions, about which more below). If I only get one copy of the galley, I might email the publisher and request a duplicate, but that's somewhat time-consuming and I only do it if there's plenty of lead time, the book looks really interesting, and the publisher clearly doesn't know about the two-galley requirement. Repeat offenders in this area eventually get ignored, on the principle that if you're that sloppy or cheap about promoting your books, you're probably that sloppy about choosing which books to publish. (Publishers take note: poor behavior on the part of your publicity staff reflects badly on your entire company. Yes, we pay attention to this sort of thing.)
Assuming that first hurdle is successfully crossed, the next question is whether my shelf is the right one for them to be on. I may get biographies of SF authors, for example, which I generally wouldn't cover with a full review. Some of those books go in our monthly "notes" column, where they get quick blurbs rather than full reviews. Others are passed along to the editors of relevant sections (e.g. speculative poetry would go to our poetry section editor, mass market first editions go to the mass market editor). For the most part, though, the books I get are trade paperback and hardcover originals of speculative fiction novels, collections, and anthologies: just the sort of thing I want to cover.
At this point I need to determine whether a book is good enough, interesting enough, or important enough to review. I review good books because our readers count on us to tell them about the good books. I review interesting books because I like drawing attention to them and they make for good reviews. (I never forget that PW lives and dies by the quality of its reviews.) I review important books--books by major authors, lead titles, books that are going to get a lot of press--because our book-buying readers care about our opinion and will want to have it to compare with other review venues, and also because it's a service to the publishers. Maintaining good relationships with publishers is vital to our business and I wouldn't dream of pretending otherwise. Of course it's also vital to maintain our independence, which is why I will almost always review an important book but I will never guarantee a favorable review.
So, the next step is to ask a series of questions about the books to determine whether they meet one of those criteria. Again, I want to emphasize that this is how I do things. Other PW editors undoubtedly have their own criteria.
1) Who's the publisher and what do I know about them?
The answer to this question can vary a lot, but in general there are four groups of publishers in my head: those who can be relied on to provide quality books, those who have some hits and misses, those who mostly publish poor quality books, and those I don't know well enough to judge. Books from publishers in group 1 get reviewed. Books from publishers in groups 2 and 4 get further consideration. Books from publishers in group 3 get a quick going-over (because every once in a while you scrape up gold from the bottom of the barrel) and if I don't see something that really grabs me, I reject them.
This has nothing to do with my personal tastes, incidentally. By "quality books" or "good books" I don't just mean books I like, but books that are well written (even if I don't like what they say) and appealing to readers (even if I'm not in their target audience).
2) Who's the author/editor and what do I know about them?
Authors and editors fall into the same groups as publishers, and I treat them the same way.
At this point, if I'm still undecided about whether to review a book, it's some combination of hit-or-miss and simply unfamiliar. This means it's probably not important (though I will keep an eye out for a promo letter that hints otherwise) and it has about a fifty-fifty chance of being good or interesting.
3) What does the promotional copy say?
I take this with a great big grain of salt, but it is useful for some things. It will give me a sense of subgenre, which can be useful if, say, I know that the the author writes very good epic fantasy and very bad hard SF. Often there's a letter from the publisher, editor, or publicist, telling me why they think the book is awesome, and something in there may catch my attention or make me roll my eyes. Blurbs are useful for categorization; a blurb from Laurell K. Hamilton indicates one sort of book, a blurb from Greg Egan indicates another sort, and blurbs from people and publications I've never heard of will still tell me something about the author's publishing history. It's all data.
4) What does the galley look like?
This is a bit of a tricky one, but in very broad terms it will help me figure out how to classify an unknown publisher. If there's no publisher or imprint logo on a bound galley, for example, I will suspect either a very new publisher or thinly disguised self-publishing; ditto certain styles of cover design. I'll also check whether the page count and ISBN are displayed on the cover of the galley. Sometimes we receive manuscripts (publishers, if you send us mss, please run down to Kinko's and have them spiral-bound rather than sending a heap of paper with a rubber band around it!), in which case I'll look at whether there's evidence of layout or it's all in Courier 12 with one-inch margins, which will hint at the publisher's editorial process.
5) How's the prose?
I'll look at the first page or two and then flip to somewhere in the middle and read another page or two, to get a sense of the writer's style, the plot, and the characters.
Now I have a much better sense of whether the book is likely to be good or interesting. If it is, I'll send it off for review, carefully selecting a reviewer who's open to books from unfamiliar sources (on the assumption that if I haven't heard of an author or publisher, my reviewer probably hasn't either) and willing to give a nuanced review of a book that will probably have both significant merits and significant flaws. If not, I'll take a pass.
If a book arrives late but looks spectacularly good, interesting, or important, I'll either rush the review--which means choosing a reviewer who isn't necessarily ideal but can reliably turn a review around in a short time--or put it on our website. In the long run, it doesn't really matter whether a review appears on the web or in the magazine; either way, it's a PW review. I tend to be reluctant to commission reviews that I know are going to go on the web, but I should probably get over that.
That's quite long enough for one post. Starred reviews next, I promise!