JLab Computing Cluster Tops 100 Teraflops

100 Teraflops
The Scientific Computing group paused for a photo in front of the new Hadron computing system that uses graphics processing units, or GPUs. Pictured from left to right in the front row are: Jie Chen, Troy Mattox, Sandy Philpott, Ying Chen and Lawrence Sorrillo. Pictured from l. to r. in the back row are: David Rackley, Christopher Larrieu, Christopher Hewitt, Chip Watson, Michael Barnes and Robert Edwards.

The fastest computer system in Hampton Roads has booted up with more than 100 Teraflops of processing power. Located at Jefferson Lab, the cluster computer system was recently upgraded with video game components to assist scientists in modeling the smallest bits of matter in the universe.
"Our resources crossed 100 Teraflops of sustained processing while running a science application. We finally hit the peak in late September," said Chip Watson, manager of Jefferson Lab's High-Performance Computing group in the Information Technology Division.

The supercomputer, called Hadron, was built with components that were purchased and cobbled together over the last year. It runs on both video game graphics cards and ordinary computer processors. About 90 percent of its computing power comes from the video game graphics processing units, or GPUs.

The Hadron system contains 480 GPUs and 266 CPUs mounted inside computing chassis and assembled into side-by-side racks. Two parts run mainly on GPUs, dubbed 9g and 10g ('g' for GPU).

"We bought two types of GPUs. We bought gaming cards, and we bought graphics cards in the professional line for supercomputing. These cards are similar to gaming cards, but they are configured somewhat differently and have error-correcting code built-in," Watson explained. "Some calculations we can do equally well on either card, and we do those calculations on the gaming cards. There are some calculations that involve much more, and those have to be on the professional-quality cards."

Hadron is being used to compute how the building blocks of matter, quarks, build the protons, neutrons and other particles that makeup everyday matter.

"The GPUs are being used solely for propagator calculations – computing how a quark will move from one point in space to another point," said Robert Edwards, a senior staff scientist in Jefferson Lab's Theory group.

100 Teraflops
Chip Watson (right), manager of the High-Performance Computing group, and Robert Edwards, a senior staff scientist in the Theory group, examine a computer box that when complete will hold four graphics processing units and two CPUs. The GPUs are stacked in front of the cardboard. The Hadron system contains dozens of similar boxes in the racks behind Edwards.

Once those calculations are done, the scientists then take the results and use them on a CPU-powered cluster computer. "We take the results from the GPUs and we tie this group of quarks together to make mesons or baryons. From this, we can compute the mass of these particles and their excited states," Edwards explained.

Watson used about $1 million of a nearly $5 million grant received as part of ARRA (American Recovery and Reinvestment Act) funding under the auspices of the Department of Energy's USQCD (US Quantum Chromodynamics) collaboration to purchase the 352 gaming cards and 128 professional graphics cards and associated hardware currently installed in the Hadron system. A chunk of the grant also went toward funding the effort to create computer code to adapt the GPUs for scientific computing.

Watson said that some of the GPUs and CPUs purchased for Hadron are still undergoing testing before being added to the system. "When we get all of the GPU systems running, the GPUs alone will exceed 100 Teraflops," he said.

A teraflop is a term used to describe a computer's performance. It indicates that a system can perform a million-million operations per second.

In the meantime, Watson said he and his team continue to monitor the marketplace for the next big leap in technology.

By Kandice Carter
Science writer

Editor's note: An article about the first portion of the Hadron system (9g) to come online is available at this link: http://www.jlab.org/news/OnTarget/2010/2010-06/Story1.html

China's State Councilor Meets With U.S. High Energy Physics Group

China Visit
This is a photo of the U.S. High Energy Physics delegation with State Councilor Liu Yandong and other government officials. Liu Yandong is in the center of the front. Jefferson Lab Director Hugh Montgomery is in the center of the second row, and Timothy Hallman, associate director for Nuclear Physics within DOE's Office of Science, is in the front row, third from the left. Photo by Liu Weibin, Xinhua.

On Nov. 2, at the Beijing Great Hall, State Councilor Liu Yandong met with the U.S. Delegation of High Energy Physics, which included Jefferson Lab Director Hugh Montgomery. The delegation was in China for the 31st Joint Committee Meeting on China-U.S. High Energy Cooperative Program.

According to a Chinese government news announcement, Liu Yandong praised the contributions of Noble Prize Laureate, T. D. Lee of Columbia University, citing his more than 30 years of encouraging cooperation between China and the U.S. in high-energy physics, in promoting the development of science and technology in China, and in speeding up the training of scientific talent in China. Liu cited China-U.S. cooperation in high-energy physics as a good example of China-U.S. collaborations in science and technology. She also expressed the hope that high-energy scientists from both countries further enhance the collaboration, continue to achieve new results, and make new and more significant contributions in promoting the common development of scientific research and friendship between the people of the two countries. The news posting is at: http://www.gov.cn/ldhd/2010-11/02/content_1736656.htm.

Liu Yandong is an official of the Communist Party of China currently serving on the Politburo and holds the post of State Councilor. She is the highest-ranking female politician in the party, and the only woman with a seat on the Politburo.

Doty Coordinates Accelerator Work for 12 GeV Upgrade

Chuck Doty
Chuck Doty
12 GeV Upgrade Accelerator Work Coordinator

Chuck Doty, Jefferson Lab's 12 GeV Upgrade accelerator work coordinator, is at home wearing a palm-tree festooned Hawaiian shirt. After all, Hawaii was his last duty station with the Navy before his retirement in 2005 as an engineering duty officer. A native of Albuquerque, he had been enthralled the first time he saw the Naval Academy at Annapolis on a trip with his dad, who was an air traffic controller. When he graduated and chose a career in the nuclear Navy, his first – and, eventually last – stop was at Pearl Harbor.

He first served on submarines, but that career course was derailed by a brain tumor when he was in his late 20s. Ultimately, it took three surgeries over a number of years to remove all of the tumor, leaving him with permanent vocal cord damage and deafness in his left ear. Although the Navy could have let him go, his skills were valued so much that he was sent to Penn State, where he earned a degree in nuclear engineering in 1986.

He often jokes that one of the reasons he's a good engineer is because his brain is stronger than most since after his last surgery his head was rebuilt with a piece of quadriceps muscle from his thigh.

Throughout the rest of his Navy career, he worked at shipyards in Charleston, Guam, Washington state and Newport News. In fact, he was stationed here when the lab was being built. "I watched it as it grew," he recalled, "and I always thought that someday I'd love to work here."

Although Doty and his wife of 30 years, Peg, would have loved to stay in Hawaii when he retired, they came east, and bought a house in Hampton; and he took a position with a small shipyard in Chesapeake. Years of fighting the Hampton Roads tunnel traffic spurred Doty to keep his eye open for a job closer to home. As luck would have it, his skills fit perfectly with the opening he spotted on the Jefferson Lab website, and he came on board in January.

"It's a wonderful group of people here," he said. "The 12 GeV Upgrade is envisioned by the physicists but the techs and the engineers are critical to making it happen."

And, he pointed out, the work environment is not so different from a submarine: "There's only so much room in the tunnel, and everything that needs to be done has to be done in that area."

There's a lot to be done. With its projected requirement of 10,000 man days, Doty's job of coordinating all the required tasks is a challenge, but one he's up for. In a sense, he noted, it's like working at a shipyard where the employees he directed didn't actually work for him, according to the organizational chart. "You have to be able to motivate people who don't work for you to do what needs to be done," he said.

What's very different, though, is what he called the "language of the lab."

"For the first three months I was here, I'd write down things I heard and didn't understand, and then go home and look them up and study," he confessed. "This place provides a great opportunity to learn and to meet new people."

He's found the culture at the lab to be not only fulfilling, but inspiring as well. "Everyone's attitude is that we're all in it together, and our goal is to make it work," he said. "You couldn't ask for a better place to work. The people are wonderful and the work is challenging."

He and Peg, who's a certified public accountant, have two daughters – Phoebe, a freshman at the College of Charleston and Abby, a freshman at Peninsula Catholic High School. The family's been very involved over the years with the girls' sports, band and swimming. The family keeps a 33-foot sloop, Alternative, at Fort Monroe. Experienced sailors, they found the boat in Ft. Lauderdale and sailed it up to Annapolis; they even had it shipped to Washington so they could sail it in the Puget Sound.

Among the other leisure pursuits that he's not very good at, Doty confessed, are golf and bowling. In fact, when he retired from the Navy, his co-workers gave him a plaque that says "100 – 100."  That, he noted, stood for his hope to someday shoot less than 100 in golf and more than 100 in bowling.

By Judi Tull
Feature writer

Lab Policy Promotes Energy Awareness

Bill Mooney
Bill Mooney
Energy Manager

Energy and water conservation, waste management and sustainability initiatives are top priorities for all federal agencies, and especially within the Department of Energy's national laboratories. Jefferson Lab is responsible for compliance with a variety of legislation, DOE Orders and directives, all of which include challenging goals to improve energy efficiency, reduce energy consumption, and, to be addressed soon, reduction of greenhouse gas emissions.

Over the years, Jefferson Lab has strived to be a good steward of the resources it uses; and during 2009 the lab put an Energy Conservation Policy in place. The policy is designed to provide a safe, productive and energy efficient workplace for all JLab staff and visitors.

Bill Mooney, Facilities Management and Logistics, is the laboratory's energy manager. His responsibilities include implementing the lab's energy policy. He also has duties related to the achievement of the lab's ongoing energy efficiency improvement goals, and compliance with DOE and other agency efficiency requirements through development of energy reduction and sustainability strategies and programs.

During the past several years, the lab has invested in energy-efficient equipment, technology and practices, and has achieved significant progress towards meeting many of its energy efficiency objectives. Further, all new construction and major renovation projects include a very high level of energy-efficiency design and technology (to qualify for LEED Gold certification), according to Mooney.

Facilities Management and work groups around the lab are identifying and evaluating energy-saver and energy-consumer activities and processes in order to determine where improvements can be made. They are making efforts to upgrading utilities; designing energy-efficiency into new construction projects; metering and upgrading environmental comfort systems (HVAC); turning off electronic equipment and appliances when possible; automating lighting; studying options for electric and bio-fuel-powered fleet vehicles; using renewable energy sources and improving building occupancy.

Currently, Jefferson Lab is installing a site-wide Energy Metering and Management System designed to monitor critical electric loads, as well as natural gas and water consumption. The EMMS installation is scheduled for full operation in the next few months to provide data necessary to better measure and manage energy efficiency initiatives, according to Mooney.

What can an individual do to make Jefferson Lab more energy efficient?

Take a few minutes to review the lab's Energy Conservation Policy; it is available on the Facilities Management webpage: http://www.jlab.org/fm/energymanagement/.

Make sure that thermostats in your areas are set to the policy standards (68 -70 degrees Fahrenheit in the heating season, and 74-76 degrees F in the cooling season).

Questions regarding the lab's Energy Conservation Policy, and especially ideas or suggestions for energy, water and waste conservation, may be directed to Bill Mooney at ext. 5461 or via e-mail: mooney@jlab.org.

BEAMS: JLab's Science, Math Outreach Program Seeks Your Help

BEAMS students learn about energy transfer, insulators and scientific procedure during the Cold Stuff activity.

Becoming Enthusiastic About Math and Science – or BEAMS – Jefferson Lab's long-running math and science enrichment program needs your help. Staff, users and students interested in assisting with student activities conducted at the lab during normal business hours are encouraged to contact Christine Wheeler, Science Education.

BEAMS supports Newport News inner-city public school students as they progress from sixth through eighth grade. Nearly 1,500 students and their teachers visit JLab between two and five days each school year to participate in science and math activities conducted by scientists, engineers, technicians and administrators. Volunteers spend about 75 minutes in a BEAMS classroom (VARC, Building 28, rooms 72A & 72B), either leading an activity or assisting students as they carry out an activity.

Volunteers are also needed to provide 15-minute Role Model visits in their office or other appropriate work area for the students. "If you aren't available for a longer activity, Role Model visits are a great way to help out," Wheeler notes.

The 2010-2011 round of BEAMS class visits began in early October and will run through May 2011. Usually two classes attend at a time and participate in two to four separate activities during each day-long visit.

Volunteers are currently needed to lead or help with the following activities:
• Go-Far Cars: Determine how the height of an inclined ramp (potential energy) impacts the distance a model car travels.
• Hot Stuff: Study how heat (energy) transfer works and test materials for their heat absorption or reflecting capabilities.
• Ultra-Violet Detectives: Learn about ultra-violet light and make your own UV detector key chain.
• Cold Stuff: Study how heat (energy) transfer works and test materials for their insulation capabilities.

Later in the school year volunteers will be needed to lead or help with these and other activities:
• Hurricanes: Learn how hurricane strength is measured and use longitude and latitude readings to track a storm's path on a grid map.
• Thermometers: Learn how to measure temperature changes and build a thermometer.
• Solar System: Learn about the solar system and build a scale model of the solar system.

"JLab staff, students and users are critical to the success of JLab's science education programs, especially BEAMS," Wheeler says. "Our BEAMS volunteers have been sharing their passion and excitement for math, science and technology with students for more than 18 years. Students get to learn about you and your job and you get to have fun working with lots of smiling faces. Everybody wins!

"From right here at the lab, you have a chance to make a difference in the lives of these students. We're always happy to welcome back volunteers who have helped in the past, and we are eager to welcome first-time volunteers," she adds.

"Want to help, but not sure how? That's what we're here for," Wheeler explains. "Science Education staff will provide the training you need. Members of the education group are happy to go over any activity whether you are new to the program or if you'd like a refresher. If you want to observe a classroom activity to see if it's something you want to do, we can schedule that."

For additional information, class visit schedules and activity opportunities, contact Wheeler at ext. 7560 or email wheelerc@jlab.org.

The program's goals include: motivating students to boost their learning; strengthening students' math and science skills with hands-on activities, and having them interact with individuals who use math and/or science in their daily work environment. BEAMS also provides teachers with classroom activities based on JLab science and technology.

You may view several of the activities' work sheets at the following pdf links:
Cold Stuff: http://education.jlab.org/cold_stuff.pdf
Go Far Cars: http://education.jlab.org/go_far_cars.pdf
Hot Stuff: http://education.jlab.org/hot_stuff.pdf
UV Detective: http://education.jlab.org/ultraviolet.pdf
Solar System: http://education.jlab.org/solar_system.pdf
Thermometers: http://education.jlab.org/thermometers.pdf
Hurricanes: http://education.jlab.org/hurricane.pdf

Lab Offers Virtual Tour on Webpage, DVD

Individuals interested in seeing Jefferson Lab facilities may now visit them via a virtual tour. The tour may be accessed from a JLab Visitor's Center webpage: http://www.jlab.org/visitors/virtualtour.html.

The web menu for the tour features an aerial view of the campus and includes a still photo and printed information about each tour stop. The tour may be viewed by selecting individual locations or viewed in its entirety.

The Public Affairs office also has the virtual tour available on DVD. The video lasts about 12 minutes and can be used to complement the lab video. A link to the lab video "Exploring the Nature of Matter" is also available from the virtual tour webpage.

Lecture Features 'Exploring Our World With Particle Accelerators'

Jefferson Lab's final 2010 Fall Science Series lecture will be Tuesday, Nov. 23, and feature James E. Brau, University of Oregon, presenting "The Mysterious Universe - Exploring Our World with Particle Accelerators."

The universe is dark and mysterious, more so than even Einstein imagined, Brau says. While modern science has established an understanding of ordinary matter, scientists now know that previously unidentified elements (dark matter and dark energy) dominate the structure of the universe, its behavior and its destiny. Brau will discuss how researchers are working address these unknowns with experiments at particle accelerators. Results of this research may revolutionize our view of nature as dramatically as the advances of Einstein and other quantum pioneers, according to Brau.

He will introduce the questions being advanced, the research facilities being used to study these mysteries and discuss our current understanding of the underlying science. The presentation will be at an introductory level, appropriate for anyone interested in physics and astronomy.

The lecture is free and open to students and adults with an interest in science. It will begin at 7 p.m., in the CEBAF Center auditorium located at 12000 Jefferson Ave, Newport News, and last about an hour. Seating in the auditorium and overflow area is available on a first-come, first-served basis and is limited to about 300 people. People arriving once capacity has been reached will be turned away.

All those under age 16 must be accompanied by a parent or responsible adult. For security purposes, enter at Jefferson Lab's main entrance (Onnes Drive.). Everyone over 16 is asked to carry a valid photo ID. Security guards may perform ID, parcel and vehicle checks.

For directions and additional information about Jefferson Lab public lectures, visit http://education.jlab.org/scienceseries/index.php, or contact Christine Wheeler, e-mail wheelerc@jlab.org or call 757-269-7560.

Lectures are added to the video archive for on-demand viewing upon approval from the presenter. The archive link is at: http://education.jlab.org/scienceseries/archive.html.

DOE Announces Winners of 2010 Energy and Water Management Awards

The U.S. Department of Energy recently announced that more than 30 individuals, teams and organizations across the federal government were selected to receive Federal Energy and Water Management Awards for outstanding and innovative efforts to implement sustainable strategies that improve energy, water and vehicle fleet efficiency.

These awards demonstrate the commitment by federal agencies to invest in efficiency measures that save money for taxpayers, reduce carbon pollution and create a stronger economy for the American people.

The 31 initiatives that received awards in October saved taxpayers almost $42 million in energy and water costs in fiscal year 2009 and kept the equivalent of about 190,000 metric tons of carbon dioxide from being released into the atmosphere. These efforts are also helping to move cutting-edge clean energy technologies into the marketplace, create new American jobs and strengthen our national security.

"As the single largest consumer of energy in the country, the federal government has a responsibility and an opportunity to lead by example in implementing clean energy projects that save taxpayers money, create new jobs and reduce carbon pollution," said Secretary of Energy Steven Chu. "Today's award winners show what is possible when it comes to implementing energy efficiency and renewable energy projects in the Federal Government and beyond."

The 2010 Federal Energy and Water Management Awards winners were selected from nominations submitted by 17 federal agencies. Included among the award winners are employees from the U.S. Air Force, Army, Marine Corps, and Navy; the Departments of Defense, Energy, Homeland Security, Interior, Transportation, and Veterans Affairs; the Environmental Protection Agency; General Services Administration and the National Aeronautics and Space Administration.

Data for fiscal year 2009 shows that the federal government has decreased energy consumption per square foot of building space by approximately 13.1 percent compared with fiscal year 2003, surpassing the goal of 12 percent for the year. The government also reported purchasing or producing 2,331 Gigawatt-hours of electricity from renewable sources, equivalent to 4.2 percent of their electricity use, surpassing the goal of 3 percent for fiscal year 2009.

This awards program is one of several held each year in October in conjunction with Energy Awareness Month to highlight the critical importance of energy efficiency and renewable resources and federal efforts to lead by example in energy management. To see the complete list of this year's winners visit the 2010 Federal Energy and Water Management Award Winners webpage.

The complete DOE news release is posted at: http://energy.gov/news/9659.htm.

DOE Releases Report on Offshore Wind Power in the U.S.

U.S. Energy Secretary Steven Chu announced recently the release of a report from the Department of Energy's National Renewable Energy Laboratory, which comprehensively analyzes the key factors impacting the deployment of offshore wind power in the U.S.

The report, "Large-Scale Offshore Wind Power in the United States: Assessment of Opportunities and Barriers," includes a detailed assessment of the nation's offshore wind resources and offshore wind industry, including future job growth potential. The report also analyzes the technology challenges, economics, permitting procedures and the potential risks and benefits of offshore wind power deployment in U.S. waters.

"Clean, renewable energy development that capitalizes on the nation's vast offshore wind and water resources holds great promise for our clean energy future and our economy," said Secretary Chu. "Today's (Oct. 7) report will help guide our efforts in the coming years to support the offshore wind industry, create new clean energy jobs, and develop environmentally responsible energy resources."

The NREL report finds that harnessing even a fraction of the nation's potential offshore wind resource, estimated to be more than 4,000 gigawatts, could create thousands of jobs and help revitalize America's manufacturing sector, reduce greenhouse gas emissions, diversify U.S. energy supplies, and provide cost-competitive electricity to key coastal regions. The report also reaches the conclusion that while significant challenges remain, effective research, policies and market commitment will enable offshore wind to play a significant role in the country's energy future. More information is available in the Executive Summary and full NREL report.

More information on DOE's efforts to promote responsible offshore renewable energy deployment can be found on the Wind and Water Power Program's website.

The complete DOE news release is posted at: http://energy.gov/news/9663.htm.

Milestones for September-October 2010


Andrew Burrill, Superconducting Radiofrequency Production Engineer, Accelerator Division
Mary Campbell, Conference Coordinator, Chief Operating Officer
Adam Carpenter, Unix/Linux System and Network Administrator, Accelerator Division
Pashupati Dhakal, Post Doctoral Fellow in SRF Materials Application, Accelerator Division
Hovanes Egiyan, Hall D Staff Scientist, Physics Division
Jesse Hales, Hall B Student Intern, Physics Division
Lawrence Hurt, Safety Systems Electronics Technician, Engineering Division
Marie Ivanco, Mechanical Engineer Co-op Student, Engineering Division
Scott Kaneta, Electronics Engineer, Physics Division
Seung Joon Lee, Radiation Detector and Imaging Group Biophysicist, Physics Division
Richard Kenney, 12 GeV Safety Inspector, 12 GeV Upgrade Project
Johnny Leung, SRF Institute Quality Manager, Environment, Safety, Health & Quality Division
Russell Mammei, Injector Scientist, Accelerator Division
Michael Pennington, Associate Director, Theoretical and Computational Physics Center
Charlene Polk, Procurement Administrator, Chief Finance Officer
Dru Renner, Lattice QCD Staff Scientist, Theoretical and Computational Physics Center
Todd Satogata, Accelerator Physicist, Accelerator Division
Alcatrous Johnson, Hall D FDC Technician, Physics Division
Kristin Grinnell, Procurement Administrator, Chief Finance Officer
Shuo Yang, Mechanical Engineer, Engineering Division
Jiawei Mei, Hall C Post Doctoral Fellow, Physics Division

Brandon Holland, Facilities Management and Logistics
Joan Holloway, Property, Facilities Management and Logistics, retired
Kristin May, Administrative Support, 12 GeV Upgrade
Nathaniel Zukerman, Facilities Management and Logistics

These Milestone entries, listed alphabetically, are full-time, term, casual and student actions posted by Human Resources for September and October 2010.

Jefferson Lab is currently seeking qualified individuals for a range of engineering positions as well some scientific, managerial and technical positions. More than 20 JLab employment opportunities are posted at: http://www.jlab-jobs.com/.



Subashini De Silva
Subashini De Silva, a graduate research assistant in JLab's Center for Advanced Studies of Accelerators, placed second in the poster competition at LINAC10. The 25th International Linear Accelerator Conference was held in Tsukuba, Japan, Sept. 12-17. The second place award came with a cash prize equivalent to about $600.


The On Target newsletter is published monthly by the Thomas Jefferson National Accelerator Facility (Jefferson Lab), a nuclear physics research laboratory in Newport News, Virginia, operated by Jefferson Science Associates, LLC, for the U.S. Department of Energy's Office of Science. Possible news items and ideas for future stories may be emailed to jlabinfo@jlab.org, or sent to the Jefferson Lab Public Affairs Office, Suite 15, 12000 Jefferson Avenue, Newport News, VA 23606