U.S. bank tellers conduct close to 10 billion transactions a year, any one of which might cement a lifetime of loyalty — or end a customer relationship for good.

But turning more of those experiences into the positive kind, or at least cutting down on the number of experiences that end badly, isn't always up to the staff at the branches.

"Some of it is about training our employees and investing more in them, but a lot of it is investing more in our infrastructure," said Brad Dinsmore, head of North American retail banking at Citigroup Inc. He said millions of dollars would have to be spent on technology upgrades, branch refurbishments and improvements to the "policies and processes that tend to get in the way of employees trying to provide great service."

To that end, Citi recently refined its lineup of checking account options, reducing confusion for customers and for staff. Other banks are expanding evening and weekend branch hours, reforming policies on fees and penalties even beyond what new regulations require, or developing smarter systems that can tailor the ATM experience based on a customer's profile.

But banks are still reticent to make the large-scale investments that will be needed to once and for all consolidate customer information across different lines of business, so that the data that branch employees might prefer to have at their fingertips is no longer spread around the disparate systems barely bound together by the architecture through which they run.

Until then, good luck making quick work of an address change for a customer who wants the updated information applied all at once to, say, a checking account, a credit card, a mortgage and a small-business loan.

Bankers also may frustrate customers impatient about credit decisions. In general, there is little that can be done at the branch level to hurry along a process that, in addition to having become increasingly centralized over the years, has also become all the more thorough in the wake of the financial crisis.

But if there is a crisis in customer service at banks, it has less to do with infrastructure and more to do with the field, where there tends to be a lack of leadership, motivation, training and coaching, said Richard Henry, the president of Millennium Consulting Group, which specializes in training for financial services companies.

"Go into a branch with a great leader, and you will find engaged people who are taking their skills to the next level and getting things done with their clients and teammates," Henry said. Visit a branch with "a lousy leader, and you hear lots of complaints about infrastructure."

Even when behind-the-scenes problems do exist, banks can offset some of the negative impact if their customer-facing staff is prepared. Problematic infrastructure "doesn't prevent me [as a branch employee] from asking better questions, from being a better listener; it doesn't typically prevent me from placing basic products" in customers' hands, Henry said.

U.S. Bancorp in Minneapolis, which has been on an acquisitive streak of late, has made an art out of avoiding the disruptions that can accompany a change in branch ownership. When branches of a failed bank are purchased from the Federal Deposit Insurance Corp., for example, "we will have branch managers from our branches in the acquired bank's region the first morning that we open the doors," said Mary Blegen, U.S. Bancorp's executive vice president for employee engagement and leadership development.

Bankers from the acquired properties are assigned "buddies" in other branches, who check on them and are available to help them navigate the company's systems and procedures.

Blegen's group includes 148 trainers across 24 states. They have been constantly on the move, visiting U.S Bancorp's 3,000 branches to educate new hires and to introduce new products or regulations. "My guys have been in the field more in 2010 than we have for many, many years," Blegen said.

It's ironic that ever more resources are being devoted to improving customer service in the branches of U.S. banks when there are ever more ways for customers to bank outside of the branch confines. But in some ways, that trend has put even more pressure on branch employees to get it right — because what they're dealing with tends to go well beyond basic deposit posting and the like.

"When customers have a choice to use the ATM or use online, mobile or telephone banking, more than 90% of their transactions are being done day to day in these alternative channels," Citi's Dinsmore said. "But when they have a problem, when they have what I call 'the moment of truth,' more often than not they want to deal face to face with a banker, and that's where we've got to figure out ways to serve them better."

Dinsmore, a former Bank of America Corp. retail executive who joined Citi in March 2009, organized a recent training program he hoped would mark the moment, for employees and for customers, when Citi got serious about improving its customer service. Responding to unsettling low scores in customer service surveys, he ordered Citibank's U.S. branches to close early on the last Saturday of September and sent 10,000 employees to training sessions conducted concurrently at 58 sites around the country.

At the end of the four-hour program, the training leaders, having promised to share the No. 1 secret to providing terrific service, instructed the employees to open the boxes that had been placed on the tables in each of the meeting rooms where the sessions occurred. Inside each box was a mirror. The No. 1 secret, the employees were told, is you.

Hokey? Perhaps.

But when you're a Citibank employee who has just spent four hours of a Saturday afternoon being told that you're a valued part of a crucial team, that you're empowered to take matters into your own hands when the standard playbook can't solve a customer's problem, that even simple efforts can make the difference between good enough and great, the mirror isn't so hokey.

"It was like a revival in there," said Stephen Fusco, Citibank's area director for the Upper East Side of Manhattan, and a leader of one of the Saturday afternoon training meetings. The program included role-playing exercises, instructional videos and pledges from one colleague to another to commit to service improvements.

Employees were reminded to welcome customers, to acknowledge them when they are waiting in line, to address them by name when they can — basic stuff, but things that have been proven to make a difference in the minds of consumers.

If banks can make those things happen every time, customers will get the signal that they are dealing with an employee base that is engaged. Add to that a back-office system that allows for smooth, responsive problem-solving, and banks will have done something far more important for customers than just making them feel as though they've stepped into the bar on "Cheers."

"Customers may want to get a very personal experience, but what I think they really want is a consistent experience, and something that expresses their value to the bank," said Jim Eckenrode, research executive for banking at the consulting firm TowerGroup. "And if you remember my kids' names, too, then great.