Janet Yellen's first big task as Federal Reserve chairwoman isn't deciding when to raise interest rates. It is deciding how.

The Fed's old system for moving interest rates up or down looks increasingly unsuited for the postcrisis financial system, so officials are rewriting their strategy for replacing it. But first they need to resolve big...