Windows 3.1 vs. Windows 95 Quantification of Learning Time & Productivity
Usability Sciences Corporation
November, 1994
Overview
Usability Sciences Corporation was retained by Microsoft to quantify the learning time and productivity of users in the migration from Windows 3.1 to Windows 95. The study's objectives were to:
Overall, users were substantially more productive with Windows 95 than with Windows 3.1. After using Windows 95 for 1.5 hours:
There has been a great deal of speculation about the impact of Windows 95 on the current population of Windows 3.1 users. Microsoft wanted to create an objective study that would quantify the learning curve of Windows 95. Usability Sciences was retained as an independent third party usability testing organization that has done extensive work in comparing various PC software programs and quantifying their relative effectiveness with users.
Microsoft and Usability Sciences established the following objectives for the study:
Process
Test Participants
In order to ensure the projectability of the study results, 75 people were recruited to participate in the testing. All participants were recruited from the general business population of Dallas/Fort Worth. All participants were employees of corporations ranging in size from a minimum of 50 employees to over 10,000. No more than 3 individuals were tested from the same company and 43 different companies were represented in the study. One third of the test pool was advanced, one third intermediate, and one third was beginner users of Windows 3.1.
It was essential in the recruiting process to establish the skills of the users ranging from beginner to advanced. To do so, Usability Sciences developed a screening questionnaire that asked users to evaluate their Windows skills. This screener was used to classify users as follows:
Beginner Uses one or two applications in Windows and rarely performs any functions or operations outside these applications.
Intermediate Uses more than two Windows applications and also uses some facilities of the operating system such as File Manager or Control Panel.
Advanced Uses many Windows applications at an advanced level and has in-depth knowledge of most features of the operating system.
In four cases it was determined by Usability Sciences testing personnel that users had misrepresented their knowledge of Windows. In these cases the users were reclassified so that the final sample was evenly divided between beginner, intermediate, and advanced level users.
Task Selection
In order to establish the work patterns and experiences of the current Windows 3.1 user population, Microsoft commissioned a telephone market research study of over 200 corporate users. This research was conducted by Market Decisions Corporation, an independent market research company in Portland, Oregon.
Based on the results of the market research Microsoft proposed a set of tasks that closely mirrored operating system usage by beginner through advanced users. The tasks were divided into three sections, A though C for each operating system. Sections A and B were designed to reflect those procedures commonly encountered in day to day computer use. Section C was designed to reflect common advanced tasks.
Usability Sciences developed a new task set that was used for the study itself. The new task set consisted of Sections 1 through 4 for each operating system. Sections 1 through 3 were completed by beginner and intermediate users; advanced users were asked to complete section 4 as well during the testing.
Testing Procedures
When users arrived for the testing they were escorted to one of Usability Sciences' lab test rooms. A briefing sheet was developed and read to each participant so they understood the purpose of the testing and the test procedures. Users were then given the Windows 3.1 task set and asked to complete it to the best of their ability. Upon completing the Windows 3.1 tasks users were given exactly 20 minutes to familiarize themselves with Windows 95. They were asked to take the Windows 95 computer based tutorial (included with Windows 95) and then explore on their own for the remainder of the familiarization time. At the end of the 20 minutes, users were given 3 task sets to complete for Windows 95. The task sets were isomorphic (similar in function but not identical in program use and file name).
Users were observed and videotaped as they worked with the operating systems to perform the tasks. Users were not given assistance from the observers if they had difficulty. Users had the availability of the on-line Help and Users Guide documentation for each system. A maximum time limit of 5 minutes per task was enforced. If a user exceeded 5 minutes in a given task they were asked to move on to the next task. Users completed a 21 question satisfaction survey at the end of Windows 3.1 use and completed the same survey after using Windows 95. Users were interviewed at the completion of the test to gather their thoughts on the ease of use and learning of each operating system.
The testing was conducted on identical 486/33 Compaq computers with 8 MB of RAM. Each user was provided with two machines: one with Windows 3.1 installed and one with Windows 95 installed. Each operating system was set up in its default state with no non-default programs resident. Windows 95 M 6.4 was used for the testing.
Data Collection
Data was collected on the following:
Results
Task Completion Time
The users were timed for each task and an overall timing was taken for each system. The timings in seconds are the raw timings and do not show whether or not the users completed the task successfully, gave up, or performed it incorrectly. The Task Success Rate reflects the rate at which the users successfully completed the tasks, and is outlined in the next section of this summary.
Overall Task Timings
Average Task Timings for All 75 Users
Overall, current Windows 3.1 users took longer to complete the tasks in the first round of Windows 95 testing. However, the users' productivity drastically increased for the second round of Windows 95, and by the third round of Windows 95 testing, the users were 91% more productive than they were with Windows 3.1.
There was a notable difference in task timings for the beginner users versus the intermediate and advanced users. The beginner users actually completed the tasks faster in Windows 95 in their first time to work with it. The beginners finished the tasks more than 1 1/2 minutes faster and still completed more tasks correctly than in Windows 3.1. The beginners were not only more productive in their first time to use Windows 95, but got more of their work done correctly than in Windows 3.1.
Across all three categories of users, not only were they more productive with Windows 95, but they completed more tasks correctly in Windows 95 in less time. Even in the first round of Windows 95, users completed more tasks correctly.
Task Success Rate
The Task Success Rate shows how many tasks each user attempted and successfully completed within the time constraint of 5 minutes. A list of 17 tasks for each system were given to the users to complete. If a user exceeded the 5 minute time limit for a task, incorrectly completed the task, or gave up on the task, the task was considered missed and not counted towards the total number of tasks completed. The graph below illustrates the rate at which the tasks were successfully completed for each system.
Overall Task Completion
Average Task Success Rate for All 75 Users
In the first round of Windows 95 testing, the users completed more tasks than in Windows 3.1. The task completion rate consistently increased the more the users worked with Windows 95. By the third round with Windows 95, the users completed on average 1.5 more tasks out of 17 total tasks than in Windows 3.1.
User Satisfaction from Surveys
After testing each system, users ranked the system across 21 different factors that are important to users of operating systems. Factor scores were on a scale of 7-Very Satisfied to 1-Very Dissatisfied. In addition, users were asked to rank the importance of each of these factors as high, medium, or low. Based on Usability Sciences' experience with user product ratings, a 0.5 difference in ratings between two products on the 7-point scale shows a distinct preference for a product. A margin in ratings larger than 0.5 is considered significant.
The graph below illustrates the users' preferences for Windows 95 over Windows 3.1 for the 5 factors they rated as most important in their evaluation of the operating systems.
User Satisfaction Survey Results
Windows 95 vs. Windows 3.1
Overall, users were significantly more satisfied with Windows 95. This can be determined from the high ratings given to Windows 95 versus Windows 3.1. In addition, when breaking down the survey into the 21 separate categories, Windows 95 was ranked higher in 15 of the categories by more than 0.5, which shows that users notably preferred Windows 95 to Windows 3.1.
The areas of Windows 95 that users especially preferred over Windows 3.1 (with a difference in ratings of 0.85 or more) were the overall ease of learning of Windows 95, ease of locating applications in Windows 95, the terminology used in the menus and dialog boxes, and the new on-line Help system.
Users also ranked the areas in which they felt were most important for an operating system to perform well. In the 16 most important aspects of an operating system as ranked by the users, the users consistently preferred Windows 95 over Windows 3.1.
On the survey's 7-point scale, the average of all 75 users' ratings for Windows 95 was extremely high, and the difference in ratings for Windows 95 and Windows 3.1 was significant, with each area being ranked an average of 0.53 points higher.
User Preferences from Post-test Interviews
The users were interviewed at the end of the test to capture their opinions and overall preferences for each operating system. The questions were designed to obtain the users' preferences for each system in the areas of ease of use, productivity, and overall preference. The table below illustrates the results of the post-test interview sessions comparing Windows 3.1 and Windows 95.
Windows 95 vs. Windows 3.1
| Interview Question | Number of Responses for Win 95 | Number of Responses for Win 3.1 | Number of Responses for Both |
| Which operating system was easier to use? | 68% 51 | 32% 24 | 0% 0 |
| Which operating system were you more comfortable with? | 33% 25 | 64% 48 | 3% 2 |
| Which operating system were you more satisfied with? | 81% 61 | 13% 10 | 5% 4 |
| Which operating system allowed you to complete your tasks most effectively and quickly? | 84% 63 | 7% 5 | 9% 7 |
| Which operating system do you prefer? | 85% 64 | 12% 9 | 3% 2 |
| Would you upgrade to Windows 95? | 97% yes 73 | 3% no 2 | 0% 0 |
The interview results showed that the users generally preferred Windows 95 over Windows 3.1 across several areas. Of the six questions asked, Windows 3.1 was preferred over Windows 95 in only one area - how comfortable they felt using the operating system. This is understandable, and was expected, since all 75 users currently use Windows 3.1 on a regular basis.
In general, the users felt Windows 95 was easier to use and allowed them to complete their tasks most effectively and quickly. An overwhelming majority, 73 of the 75 Windows 3.1 users (97%), said they would definitely upgrade to Windows 95.
Conclusions
Overall, in analyzing the transition from Windows 3.1 to Windows 95 and evaluating the users' productivity after training, Windows 95 was the superior operating system. Windows 95 not only far exceeded Windows 3.1 in the areas of satisfaction and productivity, but the current Windows 3.1 users that tested the two operating systems preferred Windows 95 and said they would definitely upgrade from Windows 3.1 to Windows 95.
The Transition from Windows 3.1 to Windows 95
In this study users were allowed 20 minutes to explore Windows 95 including a 10 minute CBT. Combining this time and actual test time, the users worked on Windows 95 an average of one hour. One of the compelling results of this study was, in the period of one hour using Windows 95, users achieved a higher rating of task completion and were more satisfied with Windows 95. People also enjoyed working with the new operating system more than with Windows 3.1.
Discoverability played a key role in easing the transition between operating systems. With Windows 95 there are a number of ways to perform the same tasks. Users took advantage of this ability using their own learned experiences and logic even in the confines of this study.
Some of the new features that also directly impacted easing the users' transition to the new operating system were: the Task Bar, the new on-line Help system, the My Computer approach to disk and file management, the fuzzy search capabilities, the Start Button, and the ease of finding their applications. The users also liked the 'Open With...' and the Wastebasket features in Windows 95. These new features made Windows 95 more appealing and more exciting to the users.
Users perceived Windows 95 to be far better than Windows 3.1 in most of the areas surveyed.
Productivity Benefits
Based on the data gathered in this study, we feel that users will be considerably more productive with Windows 95. The users in this study achieved a 91% productivity increase after working with Windows 95 for less than one hour. Key contributors to this productivity are the new operating features: the Task Bar, the new on-line Help system, and the application menuing system.
Since Windows 95 offers users many different ways of performing the same task, the users found ways to complete their tasks more often in Windows 95 than in Windows 3.1. The beginner level users were more productive with Windows 95 in their first experience using it, and completed their first set of tasks with Windows 95 faster and with more success than with Windows 3.1, the operating system they currently use everyday. We also found that many advanced users enjoyed trying to discover new ways of performing their tasks in Windows 95 and took extra time to explore its possibilities.
Users perceived Windows 95 as having a better look and feel than Windows 3.1. We observed that as users experienced more success in completing their tasks, they were more confident in the system and more likely to try new things. In our experience, this tends to encourage users to explore and discover additional features and functionality within the system. As one user stated after working with Windows 95, "In Windows 95 you are just a click away from anything."
Based on the objective data gathered in this study, as well as the subjective observations of users' attitudes towards Windows 95, we feel comfortable in strongly endorsing the new operating system. Migrators to Windows 95 will be more satisfied, more productive and feel better about using their computer.