See the attached pdf for article need Select 2 article and 1 page of review for each article: Total 2 pages No specific formatShould include brief summary, discussion, insights, implications to you,

See the attached pdf for article need Select 2 article and 1 page of review for each article: Total 2 pages No specific formatShould include brief summary, discussion, insights, implications to you,
Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=hhci20 Human –Computer Interaction ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/hhci20 ML Lifecycle Canvas: Designing Machine Learning- Empowered UX with Material Lifecycle Thinking Zhibin Zhou, Lingyun Sun, Yuyang Zhang, Xuanhui Liu & Qing Gong To cite this article: Zhibin Zhou, Lingyun Sun, Yuyang Zhang, Xuanhui Liu & Qing Gong (2020) ML Lifecycle Canvas: Designing Machine Learning-Empowered UX with Material Lifecycle Thinking, Human –Computer Interaction, 35:5-6, 362-386, DOI: 10.1080/07370024.2020.1736075 To link to this article: https://doi.org/10.1080/07370024.2020.1736075 Published online: 30 Apr 2020.Submit your article to this journal Article views: 738View related articles View Crossmark dataCiting articles: 1 View citing articles ML Lifecycle Canvas: Designing Machine Learning-Empowered UX with Material Lifecycle Thinking Zhibin Zhou a,b,c , Lingyun Sun a,b,c , Yuyang Zhang b,c, Xuanhui Liu a,c, and Qing Gong a,b,c aInternational Design Institute, Zhejiang University, Hangzhou, China; bKey Laboratory of Design Intelligence and Digital Creativity of Zhejiang Province, Hangzhou, China; cInternational Design Institute, Zhejiang University, Hang Zhou, China ABSTRACTAs a particular type of artificial intelligence technology, machine learning (ML) is widely used to empower user experience (UX). However, designers, especially the novice designers, struggle to integrate ML into familiar design activities because of its ever-changing and growable nature. This paper proposes a design method called Material Lifecycle Thinking (MLT)that considers ML as a design material with its own lifecycle. MLT encourages designers to regard ML, users, and scenarios as three co-crea- tors who cooperate in creating ML-empowered UX. We have developed ML Lifecycle Canvas (Canvas), a conceptual design tool that incorporates visual representations of the co-creators and ML lifecycle. Canvas guides designersto organize essential information for the application of MLT. By involving design students in the “research through design ”process, the development of Canvas was iterated through its application to design projects. MLT and Canvas have been evaluated in design workshops, with completed propo- sals and evaluation results demonstrating that our work is a solid stepforward in bridging the gap between UX and ML. ARTICLE HISTORYReceived 16 June 2019 Revised 25 February 2020Accepted 25 February 2020 KEYWORDSDesign method; machinelearning; user experience; design tool 1. Introduction As a particular form of artificial intelligence (AI) technology, machine learning (ML) not only provides opportunities for the user experience (UX) community , but also raises several desi gn challenges because of its malleable nature (Gillies, Fiebrink, Garcia, et al., 2016 ; Kuniavsky & Churchill, 2017 ). ML can assist designers in extracting patterns from complex information (Witten et al., 2016 ), such as understanding semantics from speech (Deng & Li, 2013 ). Thus, ML can be used to empower the UX of various interactions. However, the intelligence of ML comes from the learn ing of massive labeled datasets like audio clips annotated with semantic tags. The learning/developm ent process of ML involves the annotation of data, the construction and training of an ML model, predicti on, and so on. This constant growth means that the performance of ML might evolve across the learning p rocess.Thus,wethinkofMLasbeinglikeagrowable “tree ”instead of “wood ”with fixed attributes. The static “wood ”is easily carved into a desired form. When faced with a “tree ”, however, the designers have to consider its entire lifecycle and growth process. The malleable nature of ML hinders designers, especially novice ones, from using it in their regular design activities (Dove et al., 2017 ;Yang, 2017 ). To cultivate the desired UX, a designer is supposed to behave like a qualified gardener who would influence the growth ofthetreebyfertilizing ,trimming,andotherwise caring for it. Designers are supposed to understand the growth process of ML and know how to create the final UX with proper resources and treatment. For example, the training data could be regarded as the “fertilizer ”that promotes the growth of ML, the application scenario could provide the required “sunlight and physical environment ”, and iterative training could be considered as the process of “trimming the CONTACT Lingyun Sun [email protected] Key Laboratory of Design Intelligence and Digital Creativity of Zhejiang Province, Hangzhou, China © 2020 Taylor & Francis Group, LLC HUMAN –COMPUTER INTERACTION 2020, VOL. 35, NOS. 5 –6, 362 –386 https://doi.org/10.1080/07370024.2020.1736075 leaves ”that crafts the performance of ML. The growth of ML is closely related to users and scenarios, so these three factors should be considered simultaneously following a specific method in the conception phase. However, there are currently few design methods that assist novice designers in fully understanding ML technology, users, and application scenarios to addr ess these development issues in UX design. Firstly, UX designers lack a technical background and are not well prepared to leverage ML capabilities (Dove et al., 2017 ), because ML attributes such as accuracy and speed constantly change during the development process. Secondly, the potential value of users is ignored to improve the performance of ML and bring users better experience (Zhu et al., 2018 ). When faced with the unfamiliar growable nature of ML, designers are unable to involve the available user inputs in the development process of ML (De Choudhury & Kiciman, 2018 ; Yang, Suh et al., 2018 ). Thirdly, designers are confused about the application scenarios of ML. For example, designers need to consider the restrictions of the envi ronment, laws, and regulations when proposing an ML solution in the real world (Amershi, Weld et al., 2019 ). As a result of the above challenges, designers rarely envision how ML might empower UX within current conceptual design activities, which leads to UX issues remaining unresolved. Therefore, designing ML-empowered UX demands methods that support designers in consider- ing the malleable nature of ML, users, and the scenario at the conceptual design stage. Although typical design concepts like service design thinking (Bitner et al., 2008 ) illustrate how all stake- holders/co-creators can be taken into account, conceptual design methods for empowering UX with ML should be tailored to the attributes of ML. This is because the malleable nature of ML makes it quite complex to consider all co-creators (ML, user, scenario) in a dynamic development process. Designers are supposed to participate in such development processes with design thinking that they are familiar with (Rozendaal et al., 2018 ; Yang, 2017 ). However, existing design methods barely consider the malleable nature of the material, which prevents designers from taking advantage of the familiar design thinking to conceive design proposals for ML-empowered UX. Our work aims to incorporate the development process of ML into typical conceptual design activities in which the designers systematically consider ML, use rs, and the scenario as a whole. As the performance of ML can evolve during its development process, the more involvement a designer has with this process, the more likely the design proposal of ML-empowered UX i s to suit the demands of scenarios and users. To this end, we present the basic concepts and mechanism of ML to designers instead of offering a black-box. We also adapt the concepts from design thinking (e.g., service design, mate rial driven design) to integrate the existing research on ML –human interaction, helping designers t o understand the malleable nature of ML and to consider ML, users, and the scenario in a holistic way. In this paper, we propose a design method called Material Lifecycle Thinking (MLT) that regards ML as a constantly growing material with an iterative “lifecycle. ”MLT involves designers in the development process of ML with the consideration of co-creators who cooperate in creating the UX. We also present ML Lifecycle Canvas, a conceptual design tool derived from MLT mainly for novice designers. The evaluation and reflection of MLT are conducted by applying Canvas to design activities. The Canvas creates a visual schematic incorporating the perspectives of ML, users, and the scenario, detailing the ML lifecycle from data annotation to ML model update. For this purpose, we first study the characteristics of ML by identifying associated UX issues and summarizing the related factors through a literature review (see Figure 1 ). We then present a prototype of Canvas and Figure 1. Roadmap of our work. HUMAN –COMPUTER INTERACTION 363 iterate it according to the collected feedback. The final version of Canvas is compared to a typical conceptual design tool in workshops involving 32 participants. Analysis of completed design proposals and evaluation results reveals that our system is effective in enhancing the understanding of ML and tackling ML-related UX issues. Our contribution consists of three parts. First, the proposed MLT method encourages designers to reflect on the whole lifecycle of a malleable material like ML and cultivate it into the desired form by considering the co-creators. Second, we developed Canvas, a conceptual design tool for applying MLT and inspiring novice designers to conceive ML-empowered UX proposals without extensive ML skills. By representing ML, users, and the scenario in a visual way, the interplay between them can be charted and analyzed throughout the entire ML lifecycle. Finally, we share the lessons learned in developing Canvas, which may offer insights for future research on the integration of other AI technologies with UX design. 2. Related work As an important branch of AI, ML is defined as a series of computer programs that learn from experience and improve performance without using explicit instructions when performing a specific task (Koza et al., 1996 ; Mitchell, 1999 ). A typical ML system is a data-driven deep neural network that includes data annotation, neural network building, model training, and model prediction. In recent years, the broad application of ML has promoted considerable research on democratizing ML in the field of design. 2.1. Lowering the barrier to ML for novice designers Existing practices aimed at the democratization of ML have provided programming tools, educa- tional material, and courses for novice designers, enabling them to understand basic concepts and practically construct ML prototypes. Current programming tools assist novice designers in creating ML prototypes with simple code. These tools can be categorized as MLaaS (machine learning as a service platform) (Ribeiro et al., 2016 ), such as IBM Watson; open-source toolkits like TensorFlow (Abadi et al., 2016 ) that require programming skills; and non-programming tools including Yale and Wekinator (Fiebrink & Cook, 2010 ). Additionally, hardware platforms like AIY (Google, 2019b ) provide easy-to-use tools to assemble physical ML artifacts. Platforms such as Google ’s autoML (Google, 2019a ) or IBM ’s autoAI (IBM, 2019 ) help engineers and designers to easily construct or train high-quality ML models according to their needs. These facilitate designers in prototyping through appropriate simplification of the abstract technical details and processes of ML. However, they do not really help designers with little technical background to understand the mechanisms of ML and ideate ML- empowered UX when faced with the unfamiliar growable ML technology. Existing practice in ML-related design courses provided us with possible course module settings (McCardle, 2002 ), design processes (Rozendaal et al., 2018 ), and a tangible tool platform (Vlist et al., 2008 ). Moreover, current educational material (Hebron, 2016 ) and text books for developing statistics literacy (Kruschke, 2011 ; McElreath, 2018 ) guide designers toward a preliminary under- standing of ML ’s capabilities and working mechanisms. Although these practices and tools inspire interesting design cases, there has been little attempt to understand the malleable nature of ML and consider the interests of the stakeholders/co-creators (i.e., users and scenarios). The studies mentioned above lower the barrier to accessing ML and simplify the workflow of manipulating ML technology. However, these practices do not focus on the conceptual design, which is indispensable for novice designers, and do not incorporate users and scenarios. Thus, there is a need for ML-related design methods and tools that support conceptual design activities while considering both users and scenarios. 364 Z. ZHOU ET AL. 2.2. Existing conceptual design tools Conceptual design is a series of activities that outline and draft the key characteristics of an experience (Atasoy & Martens, 2011 ; Hanington & Martin, 2012 ). An essential part of the early design process is to initiate creative reflection and plan subsequent phases (Kim & Ryu, 2014 ). To envision hypothetical experiences, designers tend to create novel assemblies of known things and conduct reflective con- versations with design materials, users, and scenarios (Schon & Desanctis, 2005 ). Typical conceptual design tools (Tschimmel, 2012 ) used in the conception phases cover empathy, problem definition, idea generation, and experimentation (see Figure 2 ). Some of them are used in virtually every stage of design. The above tools work well in terms of understanding users and scenarios when designers have a tacit understanding of the applied technologies, but they are not tailored to the unique malleable nature of ML. Hence, they might fail to guide designers to gather crucial ML-related information and conduct subsequent conceptual design activities. To overcome this deficiency, using the above foundation for transforming complex information into design insights through a visual approach (Snyder et al., 2014 ), we modify the existing conceptual design tools in the context of integrating ML and UX. 2.3. Research on integrating ML and UX Due to the malleable nature of ML, misalignments occur between the function and interpretation of ML systems (Baumer, 2017 ), which affects UX. Most designers know little about how a given technology functions and only use what they do know to extrapolate possibilities (Bucher, 2017 ), which might not align with the real situation. Thus, recent research integrating ML and UX regards ML as a design material and identifies its main challenges (Dove et al., 2017 ) and design values (Yang et al., 2018 ). Several theoretical attempts have been made to apply design thinking concepts like human- centered design with the aim of seeking novel design opportunities about ML. Some studies advocated “power to the people ”(Amershi, Cakmak et al., 2014 ; Hand, 1998 ; Patel et al., 2008 ) and pursued human-centered ML (Gillies, Fiebrink, Tanaka et al., 2016 ) and interactive ML (Yang, Phase Tool Brief summary Empathy: gather substantial information to gain an empathic understanding of the user and scenario Personas A tool to create a fictional character whose attributes can represent a group of users. It can make the abstract idea of the user more personal and reveal insights. (Pruitt & Adlin, 2010) Problem Definition: to identify the core problems by analyzing the gathered information User Journey Map A visual representation of a user’s journey through a service, showing all the touchpoints in the journey. Touchpoints are points of interaction where the user conducts transactions in the service, exchanges informatio n with products, and so on (Stickdorn et al., 2011) . Idea Generation:envision solutions based on the insights gained in the first two stages Brainstorm A collaborative tool whereby participants mainly generate ideas based on emotions and intuition ra ther than rati onal thinking. The main objective is to produc e a large number of ideas in a short time Experimentation: evaluate alternative solutions Service Blueprint A visualization tool that show s the relationship between all stakeholders, including the users, service offers, and so on. It consists of more stakeholders than just the user, so it can bridge cross-department efforts and identify opportunities (Bitner et al., 2008) . Figure 2. Typical conceptual design tools. HUMAN –COMPUTER INTERACTION 365 Suh et al., 2018 ). For instance, Amershi, Cakmak et al. ( 2014 ) and Wood ( 2014 ) allowed users to freely modify the personalized labels so that ML systems could better understand user-preferences. The Google UX community (Jess, 2017 ) encourages designers to involve users in the ML process as a way of gaining design insights. Other recent research has drawn inspiration from the typical design process (Rozendaal et al., 2018 ; Yang, 2017 ). Yang, Scuito et al. ( 2018 ) demand that designers identify the problem that ML is intended to solve, validate the technical feasibility, and then iterate the solution to craft the ML functionality. Different design strategies have also been proposed to improve the UX of malleable, unpredictable system. The theoretical strategy applies numerous theories from behavioral and social sciences to help understand how users behave (Murnane & Counts, 2014 ; Recasens et al., 2013 ). The speculative strategy involves extrapolating from existing circumstances to imagine possible futures and is less constrained by technical feasibility (Baumer, Ahn, Bie et al., 2014 ; Dunne & Raby, 2013 ). The participatory strategy involves people who might be impacted by the system in the design process (Kuhn & Muller, 1993 ; Muller, 2007 ). Eric Baumer ( 2017 ) developed a human-centered algorithm design process that incorporates the above strategies. Moreover, some design guidelines have investigated specific ML issues, such as how to design interactions that may produce unpredictable results. Amershi et al. ( 2019 ) developed 18 human –AI interaction design guidelines that are organized into categories based on when the user interacts with AI. These guidelines require designers to clarify the capabilities of AI, the context, relevant social norms, and so on. Additionally, Google ( 2019c ) provided suggestions for specific UX issues such as trust, failure, and data collection. Although the above guidelines are helpful, they are not well connected to the theoretical explorations or design strategies that inspire designers to make trade-offs between different guide- lines based on the real-life situation and to explore other feasible possibilities. To bridge this gap, designers require a methodological framework that allows them to use practical design guidelines while following the abovementioned design thinking concepts and strategies. 2.4. Summary Existing prototyping tools and design education have certainly lowered the barrier to ML. However, as current conceptual design tools are not tailored to the ML context, there is still no design method or tool that assist designers in ideating novel and practical approaches using ML. Moreover, recent research has suggested that there is a gap between detailed guidelines and theoretical recommenda- tions like “power to the people ”and “retain the typical design process ”. Moreover, there is a lack of research on encouraging designers to participate in the growing process of ML with design strategies and consider ML, users, and the scenario as a whole. Therefore, what is required is a method that can bridge the gap between theoretical foundations and detailed guidance. To construct such a method, we need to seriously consider the malleable nature of ML and holistically incorporate the demands of the users and scenarios. 3. Material lifecycle thinking In this section, we introduce the MLT conceptual design method, which considers malleable tech- nologies like ML as design materials with iterative lifecycles. MLT also encourages users and scenarios to collaborate with materials to co-create UX. In this way, designers are able to shape ever- changing materials through user engagement and scenario arrangement, rather than simply using the material as a static black box. 366 Z. ZHOU ET AL. 3.1. Why MLT? We originally intended to apply the “material driven design ”(MDD) (Karana et al., 2015 ) method to study ML as a literal material. MDD allows designers to firstly characterize the material both technically and experientially, and then envision the material ’s role when embodied in a product. The designers then imagine how the users experience the material, rather than using intuition. Finally, the designers integrate all findings into the final product concept. MDD and similar design methods work well with static physical materials whose characteristics might not change significantly over time. However, such methods ignore other resources involved in the development process and would be constrained when dealing with ever-changing materials. An ideal method for a malleable material should guide designers to understand the basic mechanisms of the material ’s lifecycle and the factors that influence its growth. MLT draws appropriate inspiration from existing design methods, as recent studies have suggested that the design method should not change significantly in the context of ML (Rozendaal et al., 2018 ; Yang, 2017 ). Essentially, it is an extension of existing design methods such as MDD, human-centered design, value-sensitive design, and green design. Most of these methods focus on design materials with fixed attributes, which can not cope with the challenges caused by the malleable attributes of ML. In contrast, MLT is tailored to malleable design ma terials and focuses on the development process of materials. For example, the idea of “lifecycle ”was borrowed from green design, such as lifecycle assessment (Russell et al., 2005 ), to assess the environmental impacts associated with all stages of a product ’slife,from raw material processing to maintenance and recycling. In MLT, the concept of “lifecycle ”implies that the material is “growable ”and describes the development process of a material whose attributes vary over different evolutionary cycles. Regarding ML, lifecyc le refers to the specific learning process because the performance of ML can change on each iteration, inevitably affecting the UX. 3.2. What is MLT? MLT encourages designers to participate in the main development process of the material while considering the users and scenarios involved in creating UX. Figure 3 illustrates MLT, with the five Figure 3. Brief introduction to MLT. HUMAN –COMPUTER INTERACTION 367 main steps presented in a cyclic manner. The steps are as follows: (1) Understanding the technical and experiential characteristics of the material, which helps to describe the design goal; (2) Identifying the co-creators and understanding their technical and experiential characteristics. Here, the term “co- creators ”is used to describe subjects that cooperate with the material to create UX during its lifecycle. The material itself can also be regarded as a co-creator; (3) Identifying touchpoints where the co- creators exchange resources with each other and have an impact on the material ’s development; (4) Envisioning possible UX by considering the trade-offs between co-creators with various characteristics to gain design insights, embodying the insights through the touchpoints of material for generating potential UX solutions; (5) Monitoring the material, as its characteristics might evolve because of the mutual influences among the co-creators and result in different UX. We use the breeding process of potted plants as an example to explain how MLT deals with the growable material (see Figure 3 ). The process starts with a malleable material (i.e., ML) and ends with a cycle that continually improves UX. In the case of ML-empowered UX, we regard the ML, users, and scenarios as co-creators with different characteristics and interests. We consider ML as a malleable material that requires the cooperation of users and scenarios to support UX. During each loop of the ML lifecycle, a human user “teaches ”ML through interaction inputs and their preferences, while the scenario provides resources to support the ML performance and brings restrictions. Designers gain design insights by resolving the conflicts resulting from the co-creators ’own demands. This process regards ML and the scenario as synergistic partners. The objective of MLT is to help designers propose feasible and suitable conceptual UX solutions, without requiring them to practically implement ML systems. 4. Development of ML lifecycle Canvas The ML Lifecycle Canvas, a conceptual design tool featuring the holistic visualization of cooperation among ML, users, and scenarios during the ML lifecycle, is intended to enhance design practices with the MLT method. The development process includes the “Research Through Design ” (Hanington & Martin, 2012 ) approach, which is a reflective process consisting of conception, prototype, evaluation, and iteration to discover insights that will subsequently be incorporated into the Canvas. As well as conducting a literature review, we invited designers to contribute their ideas for improving the Canvas in a participatory way. 4.1. Phase 1: identify the design issues and key factors The first step in applying MLT to grow ML-empowered UX is to understand ML as a malleable material. We identified the characteristics of ML by reviewing the literature on ML-related design challenges and influencing factors. 5. Design issues Using search terms such as “AI, ”“ ML, ”“ UX, ”and “challenges, ”we collected recent (past three years) publications that mentioned ML-related UX issues from the field of human-computer inter- action (HCI). We set the scope to the past three years because of time and resource constraints, but the summary of previous work in state-of-the-art publications also covered earlier work and offered inspirations. We searched in the HCI community instead of the broader computer science domain because almost all work in HCI has some relationship to people (though not only related to ML), and is thus connected to UX design (Yang et al., 2018 ). From the work of Holmquist ( 2017 ); Yang, Scuito, et al. ( 2018 ); Forlizzi, Zimmerman, Mancuso et al. (2007 ); Dove et al. ( 2017 ), and our previous work (Zhou, Gong et al., 2019 ), which cover most UX issues related to ML, we identified potential detailed challenges. Some of these issues were specific to a particular scenario, e.g., the distrust of automotive driving and unpredictable changes in adaptive UIs. These issues were extended to the universal scenario, e.g. the distrust to AI systems. Three of the 368 Z. ZHOU ET AL. authors extracted the above-mentioned issues from the collected publications, transformed descriptions of these issues into post-it notes and pasted them on the board to construct an affinity diagramming process. They sorted the descriptions into different clusters, and named each cluster with the most representative description. After an iterative process of reorganizing and merging, we obtained six clusters, each representing one of the typical issues listed in Figure 4 . Each issue represents several associated topics. For instance, transparency represents several concerns including explainable AI (Wang et al., 2019 ), the interpretability of results, and the visualization of decision-making processes. The factors that may influence these UX issues were determined through a literature review of HCI publications and social, cultural, and psychological studies. We also reviewed current work on design guidelines for human –AI interaction (Amershi, Weld et al., 2019 ; Google, 2019c ). After the same affinity diagramming process, we identified the nine key factors (excluding the user category) listed in Figure 5 . We divided the factors into the categories of ML and scenario. Regarding the user category, existing conceptual design toolboxes provide tools such as Personas and Empathy Maps to collect user demands and preferences. 5.1. Phase 2: conception of Canvas To apply MLT in conceptual design, we crafted Canvas to present a visual form. Canvas enables designers to complete the conceptual phase of t he design according to MLT in a cyclic manner. The conception of Canvas according to the fi ve specific steps of MLT is illustrated in Figure 6 . The first step of MLT is to understand the attributes of ML. Thus, we conceived Issue cards to represent the six specific issues caused by ML (see Figure 6a ), which can be considered as the material attributes. The factors related to ML (see Figure 5) offer tips for understanding the material and describing the design goal. Secondly, Canvas uses different rows to represent the co-creators (ML, user, and scenario; see Figure 6b ). The relationships among the three co-creators are somewhat ambiguous, but we clearly assigned distinct parts to each co-creator to avoid pre-defining relationships and restricting the flexibility of Canvas. Canvas does not intend to imply any relationships among the three co-creators, leaving designers with the freedom to mine potential relationships using the advantage of MLT. We used different approaches to help designers understand these co-creators. In particular, we custo- mized the Persona to promote empathy with users. Our Persona format contains the pain points, needs, and behavioral demographic information related to ML, which aids user studies in an ML Issues Brief summary Unpredictability The issue of managing the unpredictable conclusions given by ML systems according to given data no matter how well the ML system is trained. Transparency The issue of designing the product whose workings nobody can quite explain. And how does this affect qualities like trust and confidence in the system? Learning The issue of using available user input to not only evolve the performance of ML with unobtrusive interaction but also avoid the bias in data. Control The issue of designing ML systems that allow the co-control with the user, so that users can truly cooperate with ML system in decision making. Anthropomorphism The issue of determining the anthropomo rphic degree that can impact on how people perceive the intelligent system like perceived task suitability, engagement with agents, and perceived attractiveness. Interactivity The issue of supporting the interaction between user feedback and newly learned ML models. Many ML models are built to offer improvements in accuracy and may not run quickly enough for real-time interaction. Figure 4. Specific UX issues caused by ML. HUMAN –COMPUTER INTERACTION 369 context. As for the scenario and ML system, the related factors in Table 3 provide guidance for determining the questions worth considering. In stage 3, the different columns of Canvas are used to describe different touchpoints. The touchpoints at which the users, scenario, and ML system exchange resources are shown in Figure 6c . We used the stages of the ML process (i.e., data annotation, training, inference) to pre- define the touchpoints, reducing the cognitive burden on the designers. The most important step is to envision the potential UX and embody the design insights about users and scenarios in specific steps of the ML lifecycle (touchpoints). To this end, the information collected in stages 1 –3 is organized in a visual form in Canvas to enable the identification of design insights (see Figure 6d ). The three co-creators have their own goals and optimal interests, but sometimes they have to make compromises and concessions for the final experience. For example, acquiring more data can certainly help to train a better-performing ML application, but the data size might be limited by the user ’s willingness and regulations in the scenario. As discussed above, the mutual relationships of the co-creators vary greatly with the type of ML system, the user, and the scenario. Therefore, Canvas provides a template guiding designers to think about issues worth considering for such ambiguous relationships, instead of clearly mapping the factors in Table 3 to specific issues or touchpoints. Figure 6. Conception of Canvas. Category Brief summary of the factors Scenario Societal impact: Aspirations concerning culture, poli tical system and safety. (Cuevas et al., 2007) Ta s k t y p e s : Cognitive type, control type and perceptual aids type, etc. Physical environmen t:Light, humidity, noise, location etc. ML system Automation level: Predictability and stability of automation (Merritt et al., 2013) Capability: Types of tasks that the ML system can support. Aesthetics: Aesthetic appearance, which is related to automation level and instructions (Li & Yeh, 2010). Co-control Mode: Fixed, adjustable (e.g., set by the us er), or adaptive (e.g., accounts for user’s state) (Hancock & Chignell, 1988). Hardware: Internet, power supply (e.g., battery), computing capability (e.g., GPU), etc. Learning ability: Amount and quality of required data, the difficulty of iterative training, and the required computing resources etc. Figure 5. Factors influencing the scenario and ML system. 370 Z. ZHOU ET AL. Finally, the attributes of ML might change after each loop of the lifecycle following the involve- ment of the users and scenario. The design process supported by Canvas is iterative, and designers thus need to pay close attention to the evolution and growth of the ML system to revise the design proposal accordingly. 5.2. Phase 3: Canvas Prototype The Canvas Prototype consists of Canvas I, Canvas II, Issue cards, and Persona cards. Their usage is illustrated in Figure 7 . To seek potential improvements, the prototype was used in the conceptual phase of a design project, with essential technical support provided. 6. Participants Two instructors and 30 junior (third-year) undergraduate design students majoring in industrial design (M: 17, F: 13) were involved in a design project with the theme of “ML-empowered product design ”(see Figure 8 ). The project was part of a compulsory course and students received credit for the course. We selected the students who had long-term (at least 3 months) internship experience and had done at least 2 practical design projects to ensure that they could be regarded as novice designers. These design students had been trained in the same way. They had already mastered mainstream design tools, but lacked a deep understanding of ML, which helps us to obtain objective results. The participants were split into six groups, with the two instructors offering guidance to each group as required. The instructors explained the usage of Canvas and offered technical-related suggestions, but were not responsible for the design process. Figure 7. Prototype of Canvas. Figure 8. Working with the Canvas prototype. HUMAN –COMPUTER INTERACTION 371 7. Procedure First, the participants were introduced to UX issues, related factors, and the usage of Canvas. According to the topic assigned to each group, they then gathered essential information through user studies and online research. Furthermore, the participants entered the information into Canvas and discussed the trade-offs in the optional solutions to serve the application scenario and users. Finally, Canvas was used to exchange and evaluate design ideas. Building on the design tools discussed in Section 2 and other research on developing creative tools (Atasoy & Martens, 2016 ; Jacobs et al., 2017 ), we developed four evaluation criteria. Each participant took part in a 10 –15 minutes semi-structured interview. During the in-person interviews, we asked the laddering questions (Reynolds, & Gutman, 1998 ) that were adjusted for deeper investigation according to the participant’s answers. . Laddering refers to an in-depth interviewing technique for selecting the most appropriate level of question or researcher response to respondent dialogue, based on the premise that we arranged our questions in an order that starts with the least invasive and proceeds to deeper matters. The following were part of the laddering questions. Ease-of-use: Do you think the tool is easy to use? Why do you feel that it is (or is not) easy to use? What elements do you think are important for an easy-to-use tool? Why is this important to you? Creativity: Do you think the tool helps you to come up with diverse ideas? Why do you feel that it is (or is not) helpful for creativity? What kind of ideas does the tool help you to create? Why is the tool helpful for creating the mentioned ideas? Process: Do you think the tool helps you to integrate prior expertise in the design process? Why can you (or can you not) integrate prior expertise? Why do you feel that it is (or is not) helpful in the design process? How does this tool relate to the traditional design process? Understanding: Do you think the tool helps you improve your understanding of ML? Do you think the tool encourages you to think about issues and factors related to ML? What do you feel about that? What are the critical aspects of the tool in helping your understanding of ML? 8. Results At the end of the project, all six groups produced concept designs varying from modular devices for connecting distant families (Zhou, Jiang, et al., 2019 ) to a mobile app for assisting children in drawing comics. A completed Canvas is shown in Figure 9b , including detailed information and solutions for Cabe (see Figure 9a ), an AI robot that senses changes in emotion and reacts appro- priately to help users achieve a better learning experience. How the final version of Cabe dealt with the UX issues has been discussed in our previous work (Zhibin Zhou et al., 2019). We applied a content analysis process (Howitt & Cramer, 2010 ) to summarize the interview results. We took notes during the interviews and tagged them with the eight pre-defined codes related to the Figure 9. Cabe: one of the design outcomes of the six groups. 372 Z. ZHOU ET AL. strengths and weaknesses of four evaluation criteria. We used the audio recordings of interviews to avoid missing important information. One of the authors who is most familiar with the notes conducted the coding and other analysis processes. For example, “the cyclic nature of the ML lifecycle is not obvious in the Canvas (participant 5) ”was assigned the code of “negative comment to under- standing criterion ”while “it helps me learn more about the ML concepts and its process (participant 16) ”was assigned the code of “positive comment to understanding criterion. ”The coder was respon- sible for the whole project and designed all the questions for the interview, as well as taking the notes during the interviews. Thus, we considered him as the most suitable person for the coding task to ensure that the analyzer of the data understood the context of the entire conversation. From the analysis, we found that Canvas helped the majority of participants gain an increased understanding of ML-related factors and UX issues and imagine appropriate solutions. To identify the weaknesses of Canvas, the main negative comments are summarized in Figure 10 . 8.1. Phase 4: iteration of Canvas On the basis of the evaluation results, we made the following improvements to the Canvas. These improvements mainly addressed issues including overly simplified visualization layout, ML process (touchpoints) involving technical terms, and ambiguous factors. ● We transformed the tabular visualization layout implying a linear ML process into a iterativeCanvas is split into six fan-like sections, one to visually suggest a recyclable and realistic ML process. ● We combined Canvas I and Canvas II, isolated the Persona cards from Canvas, and rearranged its format to evoke empathy because users do not change their demands significantly at different stages of ML process. We also presented completed Canvas designs as examples and provided instructions that introduce the usage of Canvas. Evaluation criteria Brief summary Ease-of-use 1) Canvas had many ML technology terms and details that hinders participants from understanding touchpoints and the stages of ML process; 2) Participants learn little Creativity about the usage of Canvasthrough the Creativity 1) Some aspects of ML and scenario ar e ambiguous and complex so it was difficult for participants to ask appropriate questions; 2) Canvas offers large quantities of information combining ML technology with UX, while participants are accustomed to thinking about the design issues without much technical detail. Process 1) Canvas was kind of inconsistent with similar design tools because it was split into two parts; 2) Some participants complained that th e complexity of factors and ML process prevented them from ffreely using Canvas Understanding in design process. Understanding 1) ML process in Canvas was still quite abstract, especially when participants wanted to match the ML process to the corre sponding after stage o f product lifecycle; 2) The factors regarding the ML and scen ario were ambiguous so the participants might have different understandings; 3) Some participants ignored the cyclic nature of the ML process because the linear form of Canvas assumes a simple linear ML process. information transmitted by the tool itself without the explanation of attached cases. Figure 10. Interview results of the prototype Canvas. HUMAN –COMPUTER INTERACTION 373 ● We modified the description of touchpoints to make them less like the terminology of ML. We combined the data annotation and dataset processing stages into data collection and split the original inference stage into prediction and performance. ● We provided a Question list instead of an ambiguous description of factors, as it was difficult for designers to understand the given factors. In particular, the most typical questions of Question list were extracted from the information of the completed Canvas in phase 3 and existing guidebooks on human-AI interaction (Amershi, Weld et al., 2019 ; Google, 2019c ). The Question list contains problems worth considering in the conceptual phase. We provided a list because the evaluation showed that designers struggled to carry out the design process without explicit instructions. However, some questions might have implications for the ambiguous relation- ships among the three co-creators. Additionally, this list does not cover all aspects of ML technology. It serves as a self-checking list that can be freely modified whenever necessary, and designers are not required to consider every question. The aim is to inspire designers to think about the most valuable issues by answering questions. 8.2. Description of the final Canvas The final version of Canvas is split into six fan-l ike sections, each representing one stage of the ML lifecycle: data collection, construction, training, prediction, performance, and updating. Canvas can be filled in collaboratively, with gro ups of people using the Question list to sketch and model the various aspects of ML, users, and t he scenario. In each section, the green area (see Figure 11b ) of Canvas is used to record information ab out the scenario, while the pink area (see Figure11a ) is for ML-related information. Issue cards ( Figure 11f )arelocatedinthegrayarea (see Figure 11c ). The Persona card shown in Figure 11e allows designers to record users ’ attitudes and feedback regarding UX issues. Figure 11d presents the Question list, with questions categorized according to the relevant co-creator s and touchpoints. Brief usage instructions are presented below. (1) Answer questions related to ML in the list and add relevant information, such as the dataset, inference speed, and neural network types in the corresponding area. Identify the UX issues Figure 11. The final version of Canvas. 374 Z. ZHOU ET AL. to be addressed at each stage and put the corresponding Issue cards in the gray area, thus clarifying the design goal. (2) Conduct the user study and complete the Persona as required; identify the key features of potential users, fill in user preferences and core appeals for ML products. (3) Answer questions related to the scenario and add the corresponding information, thus constructing a basic understanding of the scenario in terms of the laws, regulations, noise, temperature and etc. Adjustments can be made to the Issue cards in this process. (4) Explore potential proposals based on the guidance of the MLT and collected information. To this end, identify the connections and relationships among the three co-creators and explore solutions to deal with the issues at different stages. At this point, the conflicts caused by the different interests of the co-creators should be resolved. (5) Summarize the final proposal and write it down on the Issue cards for iterative evaluation. 9. Evaluation of ML lifecycle canvas To test how well the iterated Canvas works in terms of the four criteria presented in the previous section, we organized two-week design workshops . As there were no conceptual tools specifically designed for ML, we compared Canvas with Service Blueprint (Blueprint) which has similar features and goals to those of Canvas. Moreover, we presented the design outcomes of Canvas to show its effectiveness in dealing with UX is suesrelatedtothemalleablenatureofMLin conceptual design. 9.1. Participants Thirty-two participants (M: 11, F: 21) were recruited to participate in the evaluation. All participants were junior (third-year) undergraduate design students with the same background as the participants in phase 3. They were randomly divided into six teams containing six or five participants. None of the participants had been involved in the project in phase 3; the instructors were the same. 9.2. Procedure We invited the 32 participants to an evaluation session to compare ML Canvas and Blueprint in two workshops (each lasted for two weeks) on the theme of ML-empowered UX. To ensure the participants would not be biased by the previous evaluated tool, there was a six-month interval between the two workshops. The group organization, design topics, and environment of the two workshops were the same. Thus, we regarded the collected data as two independent groups. Both the workshops were part of a compulsory course and students received credit for the course. Before each workshop, the design students were introduced to the malleable nature of ML technology, including the mechanisms of ML, its attributes of unpredictability, data-centric nature, the related UX issues and etc. In the first workshop, the participants were given a two-hour introduction to the MLT method and Canvas (see Figure 12a ), whereas in the second workshop, they were introduced to service design thinking and Blueprint. After the introduction, all the participants were asked to complete two assignments. First, they were asked to use Canvas/Blueprint to illustrate an existing design case, then they were required to improve this case using Canvas/Blueprint (see Figure 12c ). During the workshop, there were several discussions where the instructors answered questions on the use of Canvas and Blueprint (see Figure 12b ). After using Canvas/Blueprint in each workshop, participants evaluated the four aspects of Canvas/Blueprint on a 5-point Likert scale, with 1 indicating strong disagreement and 5 indicating strong agreement. Each workshop was followed by the same procedure of in-person semi-structured interviews as those in Section 4.3.2, each lasting for 10 –15 minutes. The laddering questions regarding the four criteria HUMAN –COMPUTER INTERACTION 375 were also the same as those in Section 4.3.2. We did not show any expectancies regarding the performance of the participants in the workshops and the evaluation part. 9.3. Evaluation results The Shapiro-Wilk test indicated that the data from the four dependent variables (ease of use, creativity, understanding, process) was not normal distribution. Thus, we conducted a Mann – Whitney U test to compare Canvas and Blueprint with respect to the above four criteria. The analysis results showed that Canvas outperforms Blueprint regarding creativity and understanding. However, there was no significant difference between the two design tools regarding ease of use and process. The Boxplot is presented in Figure 13 . Figure 12. Workshops where the participants used the final version of Canvas. a) Introduction to Canvas; b) discussion; c) design with Canvas and Blueprint. Figure 13. Comparison between Canvas and Blueprint across the four criteria. 376 Z. ZHOU ET AL. In the corresponding in-depth analysis, we analyzed the advantages and limitations of Canvas compared with Blueprint from the results of the semi-structured interviews, using the same content analysis process as in Section 4.3.2. The following descriptions explain the reasons for the differences. 9.3.1. Ease of use There was no significant difference between Canvas ( Mdn = 3) and Blueprint ( Mdn = 3.5), U = 544.0, p=.645, r= .06. Many participants thought that Canvas was well-structured and friendly. The reasons include the following: (1) The visualization layout of Canvas “assisted us to organize information in a clear manner ”(P17) because of the “well-arranged blocks corresponding to the different co-creators (P5) ” and “concise symbols indicating different stages of ML process (P11) ”. They regarded Canvas as a good tool to record thoughts, present ideas, and review the proposals. (2) Several participants described Issue cards as “flexible ”because they are “quite easy to make adjustments ”(P5). For example, someone thought that they could easily remove or add some issues cards (P8) and take notes on the cards (P4) when discussing with partners to make sure that everyone was on the same page. (3) They also mentioned that the Question list was friendly because “some questions directly reminded us of problems that we had not considered ”(P22). By answering questions, participants were motivated to “collect the related information ”(P17) and “figure out if the concept design is feasible ”(P19). Despite all this, the negative evaluations were mainly caused by unfamiliarity with the ML concepts and usage method of Canvas. The participants thought that (1) the layout was too different from the existing design tools that designers are familiar with; (2) the visual template of Canvas did not intuitively indicate the order in which information should be entered, which complicated the usage method. In contrast, the two-dimensional chart of Blueprint is easier to understand. 9.3.2. Creativity The score was significantly higher for Canvas ( Mdn =4) than for Blueprint ( Mdn = 4), U = 318.0, p= .004, r= .36. Most of the participants thought that Canvas helped them to conceive diverse and practical UX solutions in an ML context. (1) The majority of participants commended Canvas for concerns about the malleable nature of ML. It allowed them to have an “open mind with regard to the six issues ” (P7, P14, P16) and offered them definite objects instead of seeking solutions blindly. (2) The consideration of co-creators helped them “come up with multiple solutions to a single issue ”(P7) and “figure out not only ML-related issues, but also issues involving other aspects ”(P20). (3) The division into six ML stages made the identification of solutions more in-depth. P17 said that he learned to pay attention to cultivating “a system for the common progress of co-creators. ”Canvas reminded designers to try to “strike a balance among the three co-creators (P12) ”, thus avoiding paying too much attention to one co-creator and damaging the interests of others. However, we also received the following complaints: (1) Several participants felt that the ideas derived through Canvas were more like incremental innovation rather than breakthrough innova- tion. They thought the main reason would be that the Canvas aims to provide a feasible design scheme that can specify the next steps like prototype, manufacture and so on. (2) Canvas lacked a mechanism to link scattered solutions into a complete scheme, so the design ideas corresponding to the issues are difficult to integrate into a comprehensive solution. For instance, P5 and P6 pointed out that their ideas were limited to the given stages or issues. (3) Several participants thought that the semi-structured Question list somehow restricted their ideas. P2 and P9 suggested adding more open-ended questions to inspire divergent thinking. (4) P15 and P21 suggested adding a blank Issue card to describe those UX issues that are not caused by ML. Though Canvas is designed to address ML-related UX issues, providing a way to visualize other issues would help participants to have overall control of the design proposal. HUMAN –COMPUTER INTERACTION 377 9.3.3. Process There was no significant difference between Canvas ( Mdn =3) and Blueprint ( Mdn = 3), U= 591.0, p= .256, r= .14. Several participants reported that they could easily integrate other design tools into the use of Canvas: (1) Some found that ideating with Canvas was “similar to service design thinking that requires comprehensive consideration of all stakeholders ”(P14). P16 drew an analogy between Canvas and Blueprint: “Data collection, performance, and update can be regarded as the frontstage in service design, while other stages are like the backstage. ”(2) Most participants spontaneously integrated typical design tools like brainstorming, interviews, and 5W2H into the process of completing Canvas, which shows the compatibility of Canvas with other design tools. (3) Canvas offered guidelines for conducting traditional design activities in the context of ML. Some steps of the traditional design process need to be adjusted by the participants to incorporate ML. For example, user research should involve new content about the UX issues that are related to ML. P13 mentioned that “the Issue cards are helpful when creating the outline of user interviews. ” There were some suggestions on optimizing the process. (1) Several participants believed that the layout of Canvas should be more like those of other traditional tools to enable better collaboration. (2) Some hoped that Canvas would provide detailed and formal guidelines on how to integrate design knowledge into the process of using Canvas. 9.3.4. Understanding The score was significantly higher for Canvas ( Mdn =4) than for Blueprint ( Mdn = 3.5), U= 249.5, p< .001, r= .48. Most participants stated that their understanding of ML had been improved by working with Canvas. (1) Canvas not only reminded designers of issues that had been ignored (P1, P10), but also “set up a framework to learn about ML in the way of design thinking ”(P5). (2) Some students said that Canvas changed their perception on the role of designer in ML-empowered UX design: “I realized that UX design can extend into the ML process like data collection and prediction ”(P7). Through using Canvas, they learned to enhance the performance of ML by appropriate UX design, e.g., “helping the ML system to collect user data in a more pleasant way ”(P18). However, Issue cards might not cover all ML-related UX issues(P15, P21). Additionally, Canvas does not clearly show the relationship among users, the ML system, and the scenario and “it is quite difficult to image the mutual influence between them ”(P21). 9.4. Design outcomes The completed proposals (Canvas group) are presented in Figure 14. The design outcomes of the six groups are all conceptual design proposals and not the actual implemented algorithm/systems. Te a m Brief summary 1. Hello Barbie Chat robot for accompanying children through their preferences 2. Clips Intelligent camera that automati cally records memorable moments of and their pets 3. Makeup Camera App A mobile app that helps users to vi rtually beautify themselves with virtual make-up in real time 4. Language Translator A website for academic translation tha t can translate sentences and words into different languages 5. Echo Dot Smart speaker tailored to the needs of children 6. E-shopping recommendation Shopping recommendation app based on social relationships and browsing history Figure 14. Description of completed projects. 378 Z. ZHOU ET AL. To illustrate how the participants engaged in the process of growing the ML-empowered UX iteratively over time, we present part of the design activities of team 4 as follows. The presented activities are derived from the completed Canvas, the presentation of team 4, and instructors ’records. 9.4.1. Understand the material Understand the attributes of ML models, including key elements such as the required dataset, the type of the required ML model, and the inference speed (see Figure 15d ). The ML models for text translation use a Recurrent Neural Network (RNN) that requires a large amount of computation and storage, thus cloud computing is usually adopted as a solution. Similarly, several studies are addressing the interpretability of RNN models and are transformed into practical applications. In addition, such models require a large amount of high quality, annotated data to improve system performance. 9.4.2. Understand co-creators (1) Produce Persona (see Figure 15c ) for researchers and college students who require high-quality scientific translation. They expect the platform to interpret the translation results, which helps to evaluate the credibility. Meanwhile, they possess academic translation skills and can provide profes- sional revisions. (2) Understand the scenario that mainly includes laws and regulations, social rules and other restrictions in this case (see Figure 15d ). For example, we may need to give relevant tips during the translation process for sensitive words and terminologies. In addition, academic activities require high confidentiality of text, and are sensitive to data privacy. 9.4.3. Identify touchpoint Team 4 filled the above-collected informatio ninto the corresponding areas of the ML process (touchpoints) and identified how the co-creators could have an impact on the material ’s growth. In this case, users revise the transl ation result so that the ML can learn from the data provided by the users. As the ML model is re fined, the accuracy of translation will be improved so users can benefit from the platform. Also, existing interpretable practices in Figure 15. Part of the design proposal that engages in the process of growing the ML-empowered UX. HUMAN –COMPUTER INTERACTION 379 RNN models can meet the user ’s demand for reliable results. However, the data privacy requirements in this scenario conflict with the ML system that currently only provides online services. 9.4.4. Embody The design team tried to eliminate conflicts between co-creators ’dem ands and then filled in the solutions of issues in the corresponding area. For example, using different colors to present the different operating status of the system (see Figure 15b ); using example sentence and word frequencytopresenttheinterpretationoftranslationresults(see Figure 15a ). Finally, they produced a preliminary conceptual design pr oposal by embedding these design solutions to the ML process. 9.4.5. Start another loop The design team continued to monitor how the co-creators influence each other and ensure that UX issupported by the continuous growth of ML. For example, they generated the mechanism of the medal symbolizing professionalism (see Figure 15a ) that motivates users to produce data to con- tinuously improve the ML. The system also differentiates data providers for their professionalism to quickly select high-quality samples, and so on. We also present novel ideas for coping with six UX issues from these proposals (see Figure 16 ). These ideas demonstrate how designers tackled the malleable nature of ML through addressing the six UX issues. Some of the completed Canvas is shown in Figure 16d . 9.4.6. Unpredictability Participants explored several ways to alleviate the confusion caused by the uncertainty of malleable ML applications. (1) Team 5 designed guiding questions for when the smart speaker cannot under- stand what the user has said. For example, the robot will guide users to repeat themselves or use other terms to interpret what has been said when the voice quality is poor. (2) Team 1 explored reply strategies for when their robot is not confident about the answers. For example, the robot will seek help from parents by saying “Let ’s ask mom together ”to the child. (2) Team 6 used easily under- stood tags to describe recommendations (see Figure 16b ). This allows users to quickly understand why an item is recommended and somehow reduces the feeling of unpredictability. Figure 16. Outcomes of the design workshop. 380 Z. ZHOU ET AL. 9.4.7. Transparency Various methods were conceived to expose the complexity and malleable mechanism of ML to users, which might reduce unpleasant situations. (1) Team 3 envisaged that the ML system could visualize feature point annotation of the face, whereupon users could adjust misplaced feature points to enhance the performance of ML, thus improving the UX (see Figure 16c ). Moreover, manual annotation may help the ML optimize its learning dataset. (2) Team 4 showed the current working status (working/completed) through the color of the results box. When users choose a particular vocabulary, the interpretation and other information (such as frequency of occurrence, reference source) will be displayed (see Figure 16e ) to illustrate the credibility of the results. 9.4.8. Learning The participants applied novel strategies to produce labeled data and promote the growth of the ML application, although only with the authorization of the users. (1) Team 1 spontaneously stimulated emotional expressions and obtained data, such as capturing the child ’s happy expression by telling jokes. (2) Team 4 used rewards (medal symbolizing professionalism) to facilitate users making sug- gestions on the translation results. Based on the users ’proficiency, the translation platform assigns different weights to different users, thus attaching more importance to high-quality samples. (3) All design teams applied different approaches to protect the privacy of users. For example, teams 1 and 2 allowed training, prediction, and so on to be performed locally without connecting to the Internet. 9.4.9. Control Participants came up with different strategies to balance the control exerted by the ML application and the user over the products. (1) Team 2 listed the recognized subjects on a mobile app and enabled users to remove subjects that should no longer be recognized. (2) Team 5 offered different modes for children of different ages. The complexity of the voice interaction will increase as the child grows up. (3) Team 6 offered a seamless browsing mode, allowing users to remove some behavior data from the training dataset so that the ML system will not be misled. 9.4.10. Anthropomorphism The participants explored various anthropomorphic styles to avoid the “uncanny valley ”while providing a pleasant impression. (1) Team 1 increased the anthropomorphism of their robot by using body language and voice characteristics instead of facial expressions, thus avoiding the “uncanny valley. ”(2) In contrast, team 5 was cautious about anthropomorphism. They customized certain wake words, but reduced the anthropomorphism. This avoids cognitive problems in children who should not be over-dependent on such products. (3) Team 2 retained the non-anthropomorphic appearance of Clips because it clearly makes itself known as a camera. To some extent, this protects users ’privacy. 9.4.11. Interactivity Participants tried to make the interaction smooth, and thus provided improved UX and promoted the users ’contribution to the growth of ML. (1) Team 6 simplified the process of collecting user performances. For example, the option for the user to specify that they “do not like this recommen- dation ”was very easy to select. To further enhance reliability, users can easily state why they do not like a particular recommendation. (2) To reduce the computation time used for prediction, team 5 decreased the complexity of the voice content, e.g., setting modal words with no special meaning to respond to 1 –3-year-old children and to encourage them to keep talking. (3) Team 1 used an ML accelerator called a neural computing stick to decrease the computation time, as all the computation is performed within the local hardware. HUMAN –COMPUTER INTERACTION 381 10. Discussion and future work The proposed MLT method focuses on the malleable nature of ML-empowered UX design and enables designers to participate in the development of the material. During the design process of Canvas, we obtained several insights and lessons that may inspire further research into the design methodology and tools unifying AI and UX. 10.1. Contribution to the integration of AI and UX design In this paper, we have described the characteristics of ML from the perspective of conceptual design and summarized the design challenges caused by the growable nature of ML. This might provide a foundation for follow-up UX research that considers AI technology as a growable “tree ”instead of static “wood. ” The MLT method encourages designers to reflect on the whole lifecycle of a malleable material like ML and cultivate it into the desired form. As an extension of existing design methods, MLT provides specific features for human-AI interaction design. In contrast with Human-centered Design, MLT pays equal attention to factors besides human users, especially the design material, and supports co-creation among users, ML, and the scenario. Compared with Value-Sensitive Design and participatory design methods, MLT focuses on the development process of the design material and how to intervene in the process to gain the desired UX. The Canvas provides a conceptual design template for applying MLT in growing ML-empowered UX and developing design proposals. By discussing issues about ML, users, and the scenario in a visual way, the interplay between them can be analyzed throughout the entire ML lifecycle. Our evaluation result illustrates that Canvas performs better than Blueprint in terms of creativity and understanding. However, we do not observe significant difference between the two design tools regarding ease of use and process. The interviews reveals that Canvas introduces ML-related concepts in a way that is consistent with design thinking, which is easy understandable for designers. Besides, the forms and compositions of Canvas offer meaningful creative opportunities for designers by revealing the malleable nature of ML. Additionally, the Canvas framework can be extended to other malleable design materials, enabling designers to fulfil the five stages of the MLT process. The Issue cards could be used to describe any kinds of malleable material (see Figure 17a ).TherowsandcolumnsoftheCanvas format can be used to present the co-creators and touchpoints of any other materials (see Figure 17b,c ). Designers can insert information into each space representing the different co- creators at specific touchpoints. Finally, desi gners could summarize the design insights from collected information (see Figure 17d ). The extended Canvas could easily be transformed to be applied to a similar technology. 10.2. Future work on improving MLT and Canvas For now, MLT and Canvas do not violate the traditional design process; rather, they complement original design methods because they deal with a kind of technology with malleable attributes that Figure 17. The extended material lifecycle Canvas. 382 Z. ZHOU ET AL. has not been widely applied as a design material before. We chose to retain typical design activities because designers prefer to learn about an unfamiliar material in a familiar design process. However, the design thinking process might evolve as designers better understand the inner workings of ML and prepare for working with ML. For instance, the prototyping process might disappear when AI- empowered products can iterate themselves to form better versions. The evaluation of MLT and Canvas was conducted in a university, but no implementation in a working environment has yet been investigated. Although instructors played the role of technical teams, the design students lack experiences (e.g., cooperating with engineers, applying other emer- ging technology in design) that might facilitate considerations of technical feasibility and thus potentially influence the evaluation result. Besides, the construction of evaluation and recruitment of participants may inevitably bring some biases including the educational setting of the evaluation, order effects, and etc. Future work should eliminate the possibility of order effects and other biases. We would also try to gain feedback in an industrial environment. The current Canvas platform has several limitations and requires further improvements. (1) The Question list may constrain the creative design activities, as it cannot cover all possible questions and inevitably presupposes some relationships among the three co-creators. Future practice might examine a way of providing sufficient guidance without implying these relationships. (2) Canvas should not be confined to designers, as it could also serve as a bridge for connecting experts in different disciplines. For example, Canvas might help engineers learn about design thinking, or offer a way for product managers to gain a comprehensive view of the lifecycle of ML-empowered products, thus improving communication efficiency and promoting cooperation. However, experts without a design background might require more time to learn Canvas before using it in their tasks. (3) Within the scope of our current research, Canvas is for the conceptual phase and cannot really be used in prototyping, such as helping designers to implement the ML systems. We look forward to extending Canvas to a toolbox that is not constrained to the conceptual phase. In the future, we would like MLT to be used in various industries and be used by experts in different disciplines. This would provide us with insights that motivate the further unification of UX and AI and generate new design guidelines or principles in this filed. 11. Conclusion In this paper, we have introduced MLT as a means of helping designers, especially novice designers, to explore and appreciate the malleable nature of ML, and encouraging them to consider the ML system, users, and scenario as co-creators of ML-empowered UX. The ML Lifecycle Canvas, a visual schematic for applying MLT, was developed in an iterative design process involving designers. Canvas visually represents three co-creators (ML, user, and scenario) and outlines notable questions within the ML lifecycle to enable the identification of crucial design possibilities. Through a comparison with Service Blueprint, we found that our approach guides novice designers to tackle the identified UX issues throughout the whole ML lifecycle and gain an increased understanding of the co-creators of ML- empowered UX. Though our work is at an early stage, it takes us one step closer to unifying UX and ML. We expect that the MLT method and Canvas will be improved through future use in design practice. Funding Project supported by the National Natural Science Foundation of China (No. 61672451), the China Postdoctoral Science Foundation (2018M630658), Zhejiang Provincial Key Research and Development Plan of Zhejiang Province (No. 2019C03137), and the National Science and Technology Innovation 2030 Major Project of the Ministry of Scienceand Technology of China (2018AAA0100703). HUMAN –COMPUTER INTERACTION 383 Notes on contributor Lingyun Sun is a Professor of Design at Zhejiang University. He ’scurrently a deputy director of International Design Institute. He has interdisciplinary research experiences, including artificial intelligence, computer graphics, design cognition, interaction design, ergonomics and drama. He is taking part in the MIT-SUTD Teach-the-Teacher program and teaching courses regularly at SUTD. He is the author or coauthor of over 20 journal or conference papers on Industrial Design, Computer Aided Sketching and Interaction Design. He holds patents on inventions relating to interaction and sketching technology. He is a member of the Industrial Designers Society of America, a member of thecouncil of Chinese Mechanical Engineering Society Industrial Design Institution, a member of China Computer Federation. References Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin M, Ghemawat S, Irving G, Isard, M., & Kudlur M (2016 ).Tensorflow: A system for large-scale machine learning [Paper presentation], Savannah, GA, USA. OSDI. Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. ( 2014 ). Power to the people: The role of humans in interactive machine learning. AI Magazine ,35 (4), 105 –120. https://doi.org/10.1609/aimag.v35i4.2513 . Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P.,Suh J, Iqbal S, Bennett PN, Inkpen K, &Teevan J. ( 2019 ).Guidelines for human-AI interaction [Paper presentation]. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, Uk. Atasoy, B., & Martens, J.-B. ( 2011 ).Crafting user experiences by incorporating dramaturgical techniques of storytelling [Paper presentation]. Procedings of the Second Conference on Creativity and Innovation in Design, Eindhoven, Netherlands.Atasoy, B., & Martens, J.-B. ( 2016 ). STORYPLY: Designing for user experiences using storycraft. In Collaboration in creative design (pp. 181 –210). Springer. Baumer, E. P. ( 2017 ). Toward human-centered algorithm design. Big Data & Society ,4(2), 2053951717718854. https:// doi.org/10.1177/2053951717718854 . Baumer, E. P. S., Ahn, J., Bie, M., Bonsignore, E. M., B, A., & Yip, J. ( 2014 ).CHI 2039: Speculative research visions [Paper presentation]. CHI ‘14 Extended Abstracts on Human Factors in Computing Systems, Toronto, Ontario, Canada. Bitner, M. J., Ostrom, A. L., & Morgan, F. N. ( 2008 ). Service blueprinting: A practical technique for service innovation. California Management Review ,50(3), 66 –94. https://doi.org/10.2307/41166446 Bucher, T. ( 2017 ). The algorithmic imaginary: exploring the ordinary affects of facebook algorithms. Information, Communication & Society ,20(1), 30-44. doi: 10.1080/1369118X.2016.1154086 Cuevas, H. M., Fiore, S. M., Caldwell, B. S., & Laura, S. (2007). Augmenting team cognition in human-automation teams performing in complex operational environments. Aviation Space & Environmental Medicine ,78(5), B63 –B70. https:// www.ingentaconnect.com/content/asma/asem/2007/00000078/A00105s1/art00010.De Choudhury, M., & Kiciman, E. ( 2018 ). integrating artificial and human intelligence in complex, sensitive problem domains: experiences from mental health. AI Magazine ,39(3), 69 –80. https://doi.org/10.1609/aimag.v39i3.2815 Deng, L., & Li, X. ( 2013 ). Machine learning paradigms for speech recognition: An overview. IEEE Transactions on Audio, Speech, and Language Processing ,21(5), 1060 –1089. https://doi.org/10.1109/TASL.2013.2244083 Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. ( 2017 ).UX design innovation: challenges for working with machine learning as a design material [Paper presentation]. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, Colorado, USA.Dunne, A., & Raby, F. ( 2013 ).Speculative everything: Design. Fiction, and social. The MIT press. Fiebrink, R., & Cook, P. R. ( 2010 ).The Wekinator: A system for real-time, interactive machine learning in music [Paper presentation]. Proceedings of The Eleventh International Society for Music Information Retrieval Conference (ISMIR 2010), Utrecht. Forlizzi, J., Zimmerman, J., Mancuso, V., & Kwak, S. ( 2007 ).How interface agents affect interaction between humans and computers [Paper presentation]. Proceedings of the 2007 conference on Designing pleasurable products and interfaces, Helsinki, Finland.Gillies, M., Fiebrink, R., Garcia, A. T., Bevilacqua, F., Heloir, A., Nunnari, F., Mackay W, Amershi S, Lee B, &d’Alessandro N. ( 2016 ).Human-centred machine learning [Paper presentation]. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, San Jose, California, USA.Gillies, M., Fiebrink, R., Tanaka, A., Garcia, J., Bevilacqua, F., Heloir, A., Nunnari, F., Mackay W, Amershi S, Lee B, &d’Alessandro N. ( 2016 ).Human-centred machine learning [Paper presentation]. Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, San Jose, California, USA.Google. ( 2019a ).AI & machine learning products: Cloud AutoML . Retrieved from https://cloud.google.com/automl/ Google. ( 2019b ).Do-it-yourself artificial intelligence .https://aiyprojects.withgoogle.com/ 384 Z. ZHOU ET AL. Google. (2019c).People+ AI guidebook: Designing human-centered AI products .https://pair.withgoogle.com/intro/ Hancock, P. A., & Chignell, M. H. (1988). Mental workload dynamics in adaptive interface design. IEEE Transactions on Systems Man & Cybernetics ,18 (4), 647 –658.https://doi.org/10.1109/21.17382. Hand, D. J. ( 1998). Data mining: Statistics and more? The American Statistician,52(2), 112 –118. doi: 10.1080/ 00031305.1998.10480549. Hanington, B., & Martin, B. ( 2012). Universal methods of design: 100 ways to research complex problems. Develop Innovative Ideas, and Design Effective Solutions: Rockport Publishers . https://books.google.com/books/about/ Universal_Methods_of_Design.html?id=uZ8uzWAcdxEC Hebron, P. ( 2016).Machine learning for designers .O’Reilly Media. Holmquist, L. E. ( 2017). Intelligence on tap: Artificial intelligence as a new design material. Interactions,24 (4), 28–33. https://doi.org/10.1145/3085571 . Howitt, D, & Cramer, D. ( 2010). Introduction to qualitative methods in psychology (vol. In 1). UK: Pearson Education. IBM. ( 2019).AutoAI with IBM watson studio .https://www.ibm.com/cloud/watson-studio/autoai Jacobs, J., Gogia, S., Radom, M., & Brandt, J. R. ( 2017).Supporting expressive procedural art creation through direct manipulation [Paper presentation]. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, Colorado, USA. Jess, H. ( 2017).Human-centered machine learning .https://medium.com/google-design/human-centered-machine- learning-a770d10562cd Karana, E., Barati, B., Rognoli, V., Der Laan, V., & Zeeuw, A. (2015 ).Material driven design (MDD): A method to design for material experiences. Kim, J., & Ryu, H. ( 2014). A design thinking rationality framework: Framing and solving design problems in early concept generation. Human–Computer Interaction ,29(5 –6), 516– 553.https://doi.org/10.1080/07370024.2014. 896706 Koza, J. R., Bennett, F. H., Andre, D., & Keane, M. A. ( 1996). Automated design of both the topology and sizing of analog electrical circuits using genetic programming. In Artificial intelligence in design’96 (pp. 151 –170). Springer. Kruschke, J. K. ( 2011).Doing Bayesian data analysis: A tutorial with R and BUGS . Academic Press/Elsevier. Kuhn, S., & Muller, M. J. ( 1993). Participatory design. Communications of the ACM ,36(6), 24–29. https://doi.org/10. 1145/153571.255960 Kuniavsky, M., & Churchill, E. ( 2017).Designing the user experience of machine learning systems SS-17-04 . Li, Y.-M., & Yeh, Y.-S. (2010). Increasing trust in mobile commerce through design aesthetics. Computers in Human Behavior ,26 (4), 673 –684.https://doi.org/10.1016/j.chb.2010.01.004 . McCardle, J. R. ( 2002). The challenge of integrating AI & smart technology in design education. International Journal of Technology & Design Education ,12 (1), 59 –76.https://doi.org/10.1023/A:101308940 . McElreath, R. ( 2018).Statistical rethinking: A Bayesian course with examples in R and Stan . Chapman and Hall/CRC. Merritt, S. M., Heather, H., Jennifer, L. C., & Deborah, L. (2013). I trust it, but I don ’t know why: Effects of implicit attitudes toward automation on trust in an automated system. Human Factors,55 (3), 520 –534.https://doi.org/10. 1177/0018720812465081. Mitchell, T. M. ( 1999). Machine learning and data mining. Communications of the ACM,42(11), 30–36. https://doi. org/10.1145/319382.319388 Muller, M. J. ( 2007). Participatory design: The third space in HCI. In The human-computer interaction handbook(pp. 1087 –1108). CRC press. Murnane, E. L., & Counts, S. (2014 ).Unraveling abstinence and relapse: Smoking cessation reflected in social media [Paper presentation]. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto, Ontario, Canada. Patel, K., Fogarty, J., Landay, J. A., & Harrison, B. (2008 ).Examining difficulties software developers encounter in the adoption of statistical machine learning [Paper presentation]. Proceedings of the 23rd national conference on Artificial intelligence – Volume 3, Chicago, Illinois. Pruitt, J., & Adlin, T. (2010). The persona lifecycle: Keeping people in mind throughout product design . Elsevier. Recasens, M., Danescu-Niculescu-Mizil, C., & Jurafsky, D. (2013 ).Linguistic models for analyzing and detecting biased language [Paper presentation]. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.1016/S2213-2600(13)70187-5 Reynolds, T. J, & Gutman, J. J. ( 1998). Laddering theory, method, analysis, and interpretation. Journal of Advertising Research ,28(1), 11-31. Ribeiro, M., Grolinger, K., & Capretz, M. A. M. ( 2016).MLaaS: machine learning as a service [Paper presentation]. IEEE International Conference on Machine Learning and Applications. Miami, FL, USA, the DOI is 10.1109/ ICMLA.2015.152. Rozendaal, M. C., Ghajargar, M., Pasman, G., & Wiberg, M. ( 2018). Giving form to smart objects: Exploring intelligence as an interaction design material. In M. Filimowicz & V. Tzankova (Eds.), New directions in third wave human-computer interaction: Volume 1 – technologies (pp. 25–42). Springer International Publishing. HUMAN–COMPUTER INTERACTION 385 Russell, A., Ekvall, T., & Baumann, H. (2005). Life cycle assessment introduction and overview. Journal of Cleaner Production, 13(13–14), 1207 –1210.c Schon, D. A., & Desanctis, V. (2005 ). The reflective practitioner: How professionals think in action. Proceedings of the IEEE , 73(4), 845 –846. https://books.google.com/books?hl=zh-CN&lr=&id=ceJIWay4- jgC&oi=fnd&pg=PR7&dq=The+reflective+practitioner:+How+professionals+thin k+in +action&ots=q88QS_DWum&sig=LrVDHtR4asDzR7DiTkl_cXJElk4#v=onepage&q=The%2 0reflective%20practi- tioner%3A%20How%20professionals%20think%20in%20action&f=false. Snyder, J., Baumer, E. P., Voida, S., Adams, P., Halpern, M., Choudhury, T., & Gay, G. (2014 ). Making things visible: Opportunities and tensions in visual approaches for design research and practice. Human–Computer Interaction ,29 (5 –6), 451 –486.c Stickdorn, M., Schneider, J., Andrews, K., & Lawrence, A. (2011). This is service design thinking: Basics, tools, cases (Vol. 1). Wiley Hoboken. Tschimmel, K. ( 2012).Design thinking as an effective toolkit for innovation [Paper presentation]. ISPIM Conference Proceedings, Barcelona, Spain. https://doi.org/10.1094/PDIS-11-11-0999-PDN Vlist, B., Westelaken, R., Bartneck, C., Hu, J., Ahn, R., Barakova, E., Delbressine F, & Feijs, L. (2008 ).Teaching machine learning to design students [Paper presentation]. Proceedings of the 3rd international conference on Technologies for E-Learning and Digital Entertainment, Nanjing, China. Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019 ).Designing theory-driven user-centric explainable AI [Paper presentation]. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, Uk. Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. ( 2016).Data mining: Practical machine learning tools and techniques . Morgan Kaufmann. Wood, M. ( 2014).Hopes to serve up music in a novel way .http://nyti.ms/1fr8LY5 Yang, Q. ( 2017).The role of design in creating machine-learning-enhanced user experience [Paper presentation]. 2017 AAAI Spring Symposium Series on Designing the User Experience of Machine Learning Systems, Stanford, CA, USA. Yang, Q., Banovic, N., & Zimmerman, J. (2018 ).Mapping machine learning advances from HCI research to reveal starting places for design innovation [Paper presentation]. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. (2018 ).Investigating how experienced ux designers effectively work with machine learning [Paper presentation]. Proceedings of the 2018 Designing Interactive Systems Conference, Hong Kong, China. Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. ( 2018).Grounding interactive machine learning tool design in how non-experts actually build models [Paper presentation]. Proceedings of the 2018 Designing Interactive Systems Conference, Hong Kong, China. Zhou, Z., Gong, Q., Qi, Z., & Sun, L. ( 2019).ML-process Canvas: A design tool to support the UX design of machine learning-empowered products [Paper presentation]. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, Scotland, Uk. Zhou, Z., Jiang, H., Yang, C., Yang, J., Loy, Y. Y. W., & Sun, L. (2019 ).QianLi: A modular system for connecting distant family members through tacit interaction [Paper presentation]. International Conference on Human-Computer Interaction, HCII 2019, Orlando, Florida, USA. Zhu, H., Yu, B., Halfaker, A., & Terveen, L. ( 2018). Value-sensitive algorithm design: Method, case study, and lessons. Proceedings of the ACM on Human-Computer Interaction ,2 (CSCW), 1–23. https://doi.org/10.1145/3274463 386 Z. ZHOU ET AL.

Needs help with similar assignment?

We are available 24x7 to deliver the best services and assignment ready within 3-4 hours? Order a custom-written, plagiarism-free paper

Order Over WhatsApp Place an Order Online