Carnegie Mellon Robotics Academy (CMRA) and the RAND Corporation (RAND) led a multi-disciplinary team to develop a more responsive model for curriculum development, deployment, PD, and credentialing that can both address the immediate ML training need and catalyze future development. The project emphasized development of an efficient and responsive training production and delivery system that can maintain parity with the rapidly evolving state of ML in technician work. We started with an experienced STEM education research and development team, capable community partners, and a “shovel ready” curriculum concept tailored to the hypothesized needs of the learner population. We then engaged in Design-Based Implementation Research (DBIR) to improve the production system itself through a transdisciplinary coordination of high-level interorganizational logic (e.g. stakeholder representation, efficient collaboration routines, alignment to regional technician ecology) with micro-level production processes (e.g. user-centered design, agile software development). The resulting system was exceptionally capable at adapting emerging insights about ML and AI’s relationship with technician work into relevant training that connects future workers to technical careers. DBIR coordinated this work, and codified and validated underlying principles as research findings. 

Figure 1: Production mockup for a user interface that emphasizes central concepts of ML/AI processes (e.g. representation of data, training vs. running, neural networks, confidence ratings) without requiring a strong mathematical background.

Intellectual Merit

This project advances knowledge at the interface of two perspectives that traditionally consider the same systems through different units of analysis – organizational perspectives model stakeholders engaged in joint work, while product-oriented perspectives emphasize contextualized micro-level design decisions. Each has an accounting for phenomena central to the other and thus may achieve similar results over time, but separately do not keep pace with rapidly advancing knowledge domains like ML and AI.

The best results were achieved by “burning the candle at both ends” – conducting design research on means by which to fuse the “organizations-inward” and “product-outward” approaches into a single efficient enterprise. This approach furthers convergence of literatures, methodologies, and action. It integrates theories and practices from boundary theory (sociology), user-centered design (human-computer interaction), microcredentialing (assessment), and agile software development (software engineering). Work is coordinated by connecting collaboration-focused Design-Based Implementation Research methodology (organizational and policy research) with product-focused Design-Based Research (learning sciences and instructional design). The workflow merged two initially separate lines of work into a single joint enterprise with intertwined goals.

Broader Impacts

In addition to compounding effects of dissemination of findings around convergence and the acceleration of production, the collaboration itself builds long-term social capital among participants. Developed curriculum is available for free online. Innovative simulations and user interfaces were open sourced for future designers to build upon. 

The Rapid Dissemination of AI Microcredentials through Hands-on Industrial Robotics Education (RD-AIM-HIRE) concept began as a training concept built around a 6-week bootcamp  adapted from a set of 5 existing microcertification-based courses, and extended through the integration of a Machine Learning/Artificial Intelligence (ML/AI) microcredential module to prepare vulnerable populations in the Pittsburgh region for careers in advanced robotics manufacturing. Research and collaborative design activities were pursued to facilitate collaboration of the core team (“improvement” line of work in Phase I) and assess the feasibility of elements of the original concept (“production” line), including the bootcamp model, focus on AI/ML; understanding barriers for local trainees, employers, and other potential partners; and informing the scalability of the program beyond the Pittsburgh region. These processes were designed to occur in parallel to improve the concept and the team’s collaboration.

During Phase 1, RAND conducted 43 interviews with various stakeholders in advanced manufacturing in the Pittsburgh region and peer geographies (See Table 1). The goal of the interviews was to better understand each stakeholder’s role, involvement in and perceptions of advanced manufacturing locally and outside of Pittsburgh, their experiences related to technical training, and their feedback on the prototype program as it evolved. Early interviews by RAND contributed to the design of some collaborative routines, and CMRA gathered feedback about the curriculum and supports that would need to be designed.

Trainer interview data and NSF coaching around NSF Meeting 1 led to the conclusion that the original 6-week bootcamp format was not well suited to support the at-risk portion of the population it was originally intended to reach. The financial and existing-job risk incurred by a 6-week time commitment combined with the lack of guaranteed job upon completion forced the design team to rethink this implementation. Moreover, industry interviews revealed 1) that reaching these at-risk populations required basic employability/workplace competencies training and 2) training in affordable ML/AI technologies was critical to overall adoption.

With the help of the Manufacturing Institute’s (MI) Center for Manufacturing Research, companies in MI’s network were contacted to ask if they were willing to partake in an interview that would take 30 minutes to an hour. Initial sampling criteria were to include vendors of AI technologies, and companies of different sizes that had active or past ML/AI efforts (successful or unsuccessful). Ultimately, no small manufacturers were included because we were unable to locate any with current or near-future plans for implementing AI technologies. This paper shares the results of Carnegie Mellon Robotics Academy’s (CMRA) interviews with six companies that range from AI technology vendors to manufacturers.
Individual interviewees were employees of each company with knowledge of the company’s floor operations, current technology used in the manufacturing process, plans for their workforce, and plans (if any) of their use of AI. Most companies connected us to heads of operations, Chief Technology Officers, floor managers, or combinations of individuals in those roles.

Our work in this program resulted in several prototype programs and microcredential modules. These are further explored in this section.

1. Phase II Factory Forward Prototype:
The full project team worked to map the technician training workforce ecology in the region and collaboratively design and stress test prototype ideas that stakeholders would realistically utilize. A convening of ~20 local stakeholders from training providers and workforce development agencies provided focus group feedback on the then-prototype plan. Stakeholders expressed interest on either the bootcamp portion of the process or the technology advancement aspect, but not both. This eventually resulted in the splitting of BootUP from SkillUP, so that the two ideas audiences found most interesting could be presented and developed with parity. 

Attention turned to program implementation at scale.  A survey of literature on the advanced manufacturing and workforce development landscape suggested that the NIST MEP network and Goodwill Industries International were ideal partners with aligned interests, programs, services, and national reach through their networks. The team began leveraging the prototype designs and accompanying visuals to explain the shape of a program that did not yet exist. The Factory Forward program prototype is explained in this video:

2. Robotics Technician Capstone Project Prototype:
In response to concerns, CMRA reconceptualized the bootcamp to better demonstrate collaboration, which employers valued, and reduce its length. The bootcamp shifted from a 6-week uniform experience to a 2-week engagement in which trainees in teams took different courses and combined expertise. A third week of specialized training was added in response to additional employer concerns that the skills demonstrated were not relevant enough to their specific company. A pilot using this format was scheduled with a local provider, but canceled due to COVID-19 concerns, which emerged at that time. 

3. BootUP Smartphone App:
An individual participant in the BootUP program has access to a great number of program features, including curriculum materials, wraparound services, and career navigation coaching. A simple smartphone app designed for the BootUP program organizes these into simple, useful views including progress tracking, contact information for support services, and coach chat. It also provides training reminders and access to curriculum materials.

4. Topology Optimization SkillUP Microcredential
As part of a SkillUp microcredential module, CMRA collaborated with Westmoreland County Community College (WCCC) to create a Topology Optimization prototype. Topology optimization is a process that begins with a 3D object and removes material within it to achieve a design that maintains structural integrity while reducing material costs. The process has far-reaching benefits for manufacturers as it presents the opportunity to reduce waste and save time and money while maintaining component quality. 

5. Predictive Maintenance / IoT SkillUP Microcredential
As part of a SkillUp microcredential module, CMRA created a Predictive Maintenance and Internet of Things Prototype. This module uses low-cost electronics to demonstrate how a variety of sensors can be used to continuously collect data from a remote, autonomous system. Sensor data can trigger notifications to a human operator under failing conditions, and even be fed into a machine learning model to predict future failure of the system. The technique has broad applicability to manufacturers as it presents a low-cost, high-return opportunity to improve the "uptime" of their existing equipment. 

Our work in this program resulted in several newly created microcredential modules, enhancements to existing modules, as well as the generation of professional development materials necessary for program implementation. These are further explored in this section.

Machine Vision Primer Microcredential

In this microcredential, learners experience how Machine Vision can be integrated into robotics systems to automatically perform tasks such as image detection and quality control. Learners get hands-on experience configuring a vision system for a low-cost Arduino system and transporting the vision data over the web. Learners then train a vision-based quality control system in a simulation that leverages the vision data in machine learning modules. This microcredential was developed for the SkillUP incumbent worker training program.  Access on CS2N

Arduino with Vision System

Machine Vision / Quality Control Simulation

Robotics Integration Microcredential

Robotics Integration introduces students to situations where technicians receive multiple components of a robotics system that require assembly, installation, and debugging. Students learn how to integrate components such as a vision sensor (camera) system, breadboard, servo motors, and embedded microprocessor from multiple hardware vendors. The learner will "unpack and test" components and refine “robot navigation programming" through this curriculum. This is 1 of 5 microcredentials that was enhanced in preparation for the BootUP bootcamp program. Access on CS2N

Software Foundations Microcredential

Software Foundations is an introduction to programming concepts. Students who complete this curriculum demonstrate an understanding of the software engineering process through repeated planning, testing, and iteration throughout the units. Students also learn basic robot movement, how to use feedback from different kinds of sensors, and how to create complex programs using loops and decision-making logic. This is 1 of 5 microcredentials that was enhanced in preparation for the BootUP bootcamp program. Access on CS2N

Mechanical Foundations Microcredential

Mechanical Foundations focuses on mechanical concepts such as structural design, weight distribution, drivetrains, fastening, the relationship between speed and torque, and alternate methods of transferring motion such as linear slides and belts and pulleys. This curriculum familiarizes students with the foundational skills needed to understand how components come together and different use cases for creating motion. This is 1 of 5 microcredentials that was enhanced in preparation for the BootUP bootcamp program. Access on CS2N

Electrical Foundations Microcredential

Electrical Foundations focuses on the foundational concepts around basic electricity and how circuits work. In this course, students learn how to use multimeters to measure various parts of a circuit. Students learn how to control signals using a microcontroller, how to utilize binary sensors like Limit Switches, and analog sensors like an Ultrasonic Sensor. The culminating project is an e-panel consisting of all of the components found in a typical robotic system. This is 1 of 5 microcredentials that was enhanced in preparation for the BootUP bootcamp program. Access on CS2N

Fabrication Foundations Microcredential

Fabrication Foundations introduces students to hand tools to cut, drill, and file down multiple materials to create a robot chassis and motor mount. Students also learn how to use additive manufacturing (3D printing) to create a sensor mount. The skills taught include safety, basic measurement, reading and interpreting drawings, basic hand tool use, and handling materials. This is 1 of 5 microcredentials that was enhanced in preparation for the BootUP bootcamp program. Access on CS2N

1. Rapid Prototyping with Unity and React

Web applications are great way to quickly reach a lot of users without the hassle of native client installs and play store downloads. Web frameworks such as React allow page components to communicate and respond to user interaction much like traditional applications. Being web-focused, React also offers exceptionally strong support for adaptive and responsive layout, e.g. smoothly re-flowing the page content for a vertically-oriented mobile device or browser window.

• Access the full blog article here: https://medium.com/carnegie-mellon-robotics-academy/rapid-prototyping-with-unity-and-react-f3f12b5d7707
• Access the full code repository here: https://github.com/cmroboticsacademy/cra-template-unity-react

2. Deploying a Machine Learning Model to a Serverless Backend with SAM CLI

Over the last year at CMRA, we have been incorporating more machine learning models into our applications. There are plenty of data science blogs about developing models. However, there are very few tutorials about deploying models into production. Fortunately, there are options available for this and it continues to get easier.

Popular options for deploying models developed in Python include using services written in Flask or Django for client applications to interface with. AWS, Azure, and Google Cloud each offer their own brand of Machine Learning as a Service (MLaaS) products. These services offer convenient workflows, cloud based notebooks, and marketplaces for models. As nice as both options are there are drawbacks when it comes to costs and scaling.

• Access the full blog article here: https://medium.com/carnegie-mellon-robotics-academy/going-serverless-for-your-ml-backend-with-sam-cli-5332912019ef
• Access the full code repository here: https://github.com/cmroboticsacademy/serverless_ml_tutorial

3. How We Used Transfer Learning to Create an Image Classifier in the Browser

We recently released an interactive activity that provides an introductory understanding of how an Image Classifier might be used in a factory context. The goal of this activity is to produce as many ‘good’ widgets as possible while also minimizing ‘defective’ widgets. The robot can be trained to classify any part coming down the assembly line and can also be told to remove an object that is identified as ‘defective’.

• Access the full blog article here: https://medium.com/carnegie-mellon-robotics-academy/how-we-used-transfer-learning-to-create-an-image-classifier-in-the-browser-57a0abb4ccd3

• A national partnership is critical to creating an efficient, rapid, and sustainable training program for technical workers in advanced manufacturing.
• An emphasis on transitional technology and foundational, forward-looking skills like computational thinking in entry-level training, as opposed to the original, explicit focus on ML and AI
• The need to de-risk and facilitate employer and trainee participation through early asynchronous employment training, wrap-around services, and work-based micro-trainings at employers
• Thinking about the SMM as the unit of intervention as opposed to workers or training providers
• The focus in which employees and leaders shift their thinking toward Industry 4.0 in parallel. The cohort and peer learning system that are part of BootUP and SkillUP build on MEP efforts to consult with SMM firm leadership to include training for new and incumbent employers.

• Advisory Committee on Student Financial Assistance (2012), Pathways to Success: Integrating Learning with Life and Work to Increase National College Completion, Washington, D.C.  https://files.eric.ed.gov/fulltext/ED529485.pdf
• Andrew, Megan, Timothy Marler, Jesse Lastunent, Hannah Acheson-Field, and Steven W. Popper, An Analysis of Education and Training Programs in Advanced Manufacturing Using Robotics, Santa Monica, Calif.: RAND Corporation, forthcoming, June 28, 2020.
• Baird, Matthew D., John Engberg, Gabriella C. Gonzalez, Thomas Goughnour, Italo A. Gutierrez, and Rita Karam (2019), Effectiveness of Screened, Demand-Driven Job Training Programs for Disadvantaged Workers: An Evaluation of the New Orleans Career Pathway Training, Creative Commons Attribution 4.0 International Public License, RR-2980-DOL. As of May 12, 2020: https://www.rand.org/pubs/research_reports/RR2980.html
• Brown, A. L., & Campione, J. C. (1996). Psychological theory and the design of innovative learning environments: On procedures, principles, and systems. In L. Schauble & R. Glaser (Eds.), Innovations in learning: New environments for education (p. 289–325). Lawrence Erlbaum Associates, Inc.
• Feldman, Martha & Pentland, Brian. (2003). Reconceptualizing Organizational Routines As a Source of Flexibility and Change. Administrative Science Quarterly - ADMIN SCI QUART. 48. 94-118. 10.2307/3556620.
• Fuchs, Erica R. H. (2014), “Global Manufacturing and the Future of Technology” Science, Vol. 345, No. 6196, pp. 519–520.
• Giffi, Craig, Paul Wellener, Ben Dollar, Heather Ashton Manolian, Luke Monck, and Chad Moutray (2018), 2018 Deloitte and the Manufacturing Institute Skills Gap and Future of Work Study, Deloitte Insights and the Manufacturing Institute. As of July 17, 2019: https://www2.deloitte.com/content/dam/insights/us/articles/4736_2018-Deloitte-skills-gap-FoW-manufacturing/DI_2018-Deloitte-MFI-skills-gap-FoW-study.pdf
  
• Holzer, Harry J. (2015), “Sector-Based Training Strategies: The Challenges of Matching Workers and Their Skills to Well-Paying Jobs,” Future of Work Paper Series, p. 48.
• Malkus, Nat (2019), “The Evolution of Career and Technical Education, 1982-2013.” Washington, D.C.: American Enterprise Institute. As of March 16, 2019: https://www.aei.org/wp-content/uploads/2019/04/RPT-Malkus-Evolution-of-CTE-5-FINAL.pdf
• Modestino, Alicia Sasser (2016), “The Importance of Middle-Skill Jobs,” Issues in Science and Technology, Vol. 33, No. 1, pp. 41–46.
• National Academies of Sciences, Engineering, and Medicine (2017b). Securing advanced manufacturing in the United States: The role of Manufacturing USA: Proceedings of a workshop. Washington, DC: The National Academies Press. Available at https://doi.org/10.17226/24875.
• National Academies of Sciences, Engineering, and Medicine (2017a). Advancing Concepts and Models for Measuring Innovation: Proceedings of a Workshop. C.D. Mackie, rapporteur. Committee on National Statistics. Washington, DC: The National Academies Press. Available: https://doi.org/10.17226/23640.  
• National Research Council (2000), Surviving Supply Chain Integration: Strategies for Small Manufacturers. Washington, D.C.: The National Academies Press.
• North, D. C. (1992), “Institutions and Economic Theory.” The American Economist, 36(1), pp. 3–6. https://doi.org/10.1177/056943459203600101
• Organisation for Economic Co-operation and Development (2017), Getting Skills Right: Skills for Jobs Indicators, Paris: OECD Publishing.
• Penuel, W. R., Allen, A. R., Coburn, C. E., & Farrell, C. (2015). Conceptualizing research–practice partnerships as joint work at boundaries. Journal of Education for Students Placed at Risk (JESPAR), 20(1-2), 182-197.
• Creative Commons Attribution 4.0 International Public License (2019), Programs for Disadvantaged Workers: An Evaluation of the New Orleans Career Pathway Training, RR-2980-DOL. As of May 12, 2020: https://www.rand.org/pubs/research_reports/RR2980.html
  
• Ryburn, B., Brown, D. and Meier, R. (2011), Exploration of technical and nontechnical employee attitudes toward participation in training. Perf. Improv., 50: 33-40. doi:10.1002/pfi.20239
• Strada Education Network (2020), “Public Viewpoint: COVID-19 Work and Education Survey Work Impact and Education Perspectives by Job Field.” Indianapolis, IN: Strada Center for Consumer Insights.  As of May 6, 2020: https://www.stradaeducation.org/wp-content/uploads/2020/05/Public-Viewpoint-Charts-Week-6.pdf
  
• U.S. Congressional Research Office (2018), The Manufacturing Extension Partnership. Washington, D.C. As of May 5, 2020: https://crsreports.congress.gov/product/pdf/R/R44308/15
• Waldman, P., & Mulvany, L. (2020, March 5). Farmers Fight John Deere Over Who Gets to Fix an $800,000 Tractor. Retrieved from https://www.bloomberg.com/news/features/2020-03-05/farmers-fight-john-deere-over-who-gets-to-fix-an-800-000-tractor
  
• Zaber, Melanie A., Lynn A. Karoly, and Katie Whipkey (2019), Reimagining the Workforce Development and Employment System for the 21st Century and Beyond. Santa Monica, CA: RAND Corporation. https://www.rand.org/pubs/research_reports/RR2768.html.