In February 2015, a post to Micronet, a UC Berkeley mailing list for campus IT professionals, led to a BRC Consulting engagement with The Institute of Transportation Studies’ Connected Corridors program. Greg Merritt, a lead developer on Connected Corridors, engaged Aaron Culich of Research IT to help wrangle serious threats to the program’s progress posed by increasing computing requirements, aging high-maintenance hardware, staff turnover, and constrained funding.
Merritt asked: “Does anyone know of a way to access the free trial Amazon Web Services [AWS] without entering a credit card?” His near-term goal was to explore the possibility of offloading some of the Institute’s infrastructure to the cloud. Culich answered the Micronet post, then scheduled time to explore Connected Corridors’ research requirements and computational needs under the auspices of the Berkeley Research Computing (BRC) program’s consulting service. During the initial consulting session, Culich discovered that Connected Corridors needed much more than the ability to pay for Amazon services with a campus chartstring.
Connected Corridors is a combined research and production technology-based integrated transportation corridor management program. It uses a data- and model-driven approach to support real-time decision making for traffic management in the State of California. The program combines contributions from Professors Alexandre M. Bayen (PI, EECS & CEE), Roberto Horowitz (ME), Adib Kanafani (CEE), Alexey Pozdnukhov (CEE), Alexander Skabardonis (CEE), and Pravin Varaiya (EECS). The Connected Corridors system faces challenges that include high volume/high throughput data with very high activity for short bursts (traffic event management), and the requirement to provide real-time responses within minutes.
The system architecture is complex (see accompanying diagram), and includes components originally deployed on legacy hardware, alongside other components in active development. The project’s rapid development cycle needed to continue uninterrupted during a transition from legacy infrastructure to the cloud, and faced a June 2015 deadline to demo the system to sponsors.
In assisting the project’s professional staff to address this critical transition, BRC Consulting provided:
- a primary point of contact to help facilitate access to the AWS cloud platform, leveraging the campus-negotiated AWS Enterprise Customer Agreement -- including the ability to pay monthly charges through the campus invoicing system;
- assistance in planning for costs;
- advice for writing and submitting AWS Research Grants;
- training for Connected Corridors’ research and technical staff in best practices and re-usable cloud templates; and,
- coordination with campus partners to configure a virtual private network through AWS Direct Connect for access to Oracle databases in the campus data center and reduced data egress charges (campus partners included Isaac Orr and Erik McCroskey, in the IST Telecommunications & Networks group; as well as Walter Stokes and Adam Fuchs, in the IST Enterprise Data Warehouse group).
The first phase to develop the proof of concept in the cloud was successfully completed on-time. As Merritt observed:
“BRC consulting has accelerated this transition ahead of all expectations, by contributing guidance in best practices, planning, technical training for research support staff, and assistance with navigation of associated University and Amazon bureaucratic processes.”
The next phase, now under way, includes BRC Consulting support of Connected Corridors developers as they work with AWS Solutions Architects to review the proof of concept and architect the next iteration in the project’s evolution as a secure, reliable, scalable cloud-based platform for research into traffic simulations, service of Caltrans’ long-term operations and planning objectives, and providing real-time decision support for up to fifty transportation corridors across the state.
While Connected Corridors is an example of a complex research project with a professional IT support team, BRC welcomes projects large and small to schedule consultations to help match research requirements with computing resources that fit the project’s needs. BRC supports research with a coordinated set of services across a range of computation and data analysis needs, including Cloud Computing. Contact us at research-it@berkeley.edu.