Q-Assessor News

Forced Selection of Poles During Q Sorts

Several investigators using Q-Assessor have inquired about why participants are forced to select first the most “positive” statements and then the most “negative” statements during the second level sort. This is confusing and undesirable, they say. Participants should just be able to place any statement anywhere in the grid at any point.

It’s a bit hard to understand how this is confusing, since Q-Assessor’s instructions to participants clearly tell them that this will be the actions they are to perform. Further, Q-Assessor’s mechanisms clearly guide participants to do precisely these steps. If Q-Assessor wasn’t directive, then it might be confusing. As it is, the expected actions are incontrovertible.

But more importantly, Q-Assessor imposes these precise steps because that is the way Q technique is supposed to be performed. Steven R. Brown in his Political Subjectivity book makes this very clear (page 196):

Procedurally, the subject is instructed to spread out all the “characteristic” statements, to read through them again, and to select those two which, of those available, he regards as most characteristic, and these are placed, one under the other, beneath the +4 label, as shown in the illustrative score sheet in figure 13. Next, the subject is instructed to spread out all the “uncharacteristic” statements and to select the two most uncharacteristic for placement under -4. Returning to the positive side, those three next-most characteristic items are selected for +3, followed by the next three-most uncharacteristic items for -3. By working back and forth between characteristic and uncharacteristic, the subject gradually approaches the middle.

Thus Q-Assessor’s approach actually already involves a simplification of official Q technique because it only enforces the polar choices for the first round of choices. We’ve done so with some reluctance given the clear nature of Brown’s description. However, since Q investigators these days chafe even at the level of direction Q-Assessor provides now, we’re going to stick with the current strategy.

Other computer-based and online systems don’t impose any such sequential behaviors upon participants, and this presumably is why investigators wonder why Q-Assessor does. However, these other systems are doing Q wrong — and even Q-Assessor isn’t doing it quite right. But Q-Assessor is closer to the correct technique than the others.

New Drag-and-Drop Interface for Q-Assessor

We’ve just added a new drag-and-drop Q-sort interface to Q-Assessor, and we’d appreciate feedback about its design and implementation. You can learn more about it, run a demo study, and compare it to our standard interface here.

The new interface runs on all current browsers — Firefox, Safari, Camino, Chrome, Opera, and Internet Explorer versions 9, 8, 7, and 6. Unlike Flash-based systems, it does not require any particular plugins to be installed into participants’ browsers, and it even works (sort of) on iPads and iPhones. Modern hardware with sufficient memory and processor speed insures that the drag-and-drop motions are fluid. Like all web sites, it functions best with standards-compliant browsers (viz., anything besides Internet Explorer before IE9), but we’ve even managed to get IE6 to work, despite the fact that it is so far past its expiry date.

The design scales well to real-sized studies; some of our current users employ concourses with over 100 statements, and our new design fits into a standard screen size without scrolling. Investigators can configure their studies to use either our traditional interface or the new one. We’re gearing up to do another validation study comparing this new interface against our existing, previously-validated interface as well as paper sorts. We think that the drag-and-drop interface probably will replace our original one, but until it’s validated, we won’t know for sure.

The unique strength of Q-Assessor continues to be its integration of the entire Q process, including easy-to-use authoring tools, enrollment management, post-sort interviews, and immediate, online data analysis. The new drag-and-drop Q sort interface merely removes the main objection we’d been hearing about Q-Assessor — that it didn’t “look” like a “standard” Q experience. Now it does.

In addition to running the demo study, please sign up at our site and conduct your own study for free. The best way for you to see what Q-Assessor can do is to use it for a real project.

We’re glad to see the other new offerings under development out there, and the best way for you to evaluate them is to use them head-to-head for real studies. We really want your comparative feedback. Thanks!

Q-Sort Layouts: It's the Sort That Counts, Not the Layout

Some online Q implementations such as FlashQ utilize a two-dimensional grid to structure participants’ sorting. Q-Assessor employs a vertically-oriented, grouped design. Why have these two approaches come to be? Is one superior? Does the layout for a Q sort really matter?

What Happens During a Q Sort

Before addressing these questions directly, let’s step back and consider precisely what the end product of the Q sort is and how it is used in the factor detection algorithms that yield Q findings. This product consists of an ordering of statements from one pole (say, of "agreement") to another pole (say, of "disagreement"). Within this ordering, however, are discrete groupings, defined by the investigator when setting up the study. Within any group, all statements assigned by the participant have the same “value” as the others within that group. Thus, the overall ordering of the statements isn’t, for instance, S1→S2→S3→S4 (for a four-statement concourse) but rather is S1→(S2,S3)→S4 (if the investigator set up three sort “bins” in which the participant places one statement at each pole and two in the middle.

The calculation algorithms take these grouped orderings and treat numerically all statements within each group interchangeably. Thus if one participant submits a sort of S2→S3→S1→S4 and another participant submits a sort of S2→S1→S3→S4, the two participants’ sorts are identical as far as the Q factor algorithms are concerned — because the calculations ignore the orderings of statements within that middle group. (S3→S1) is identical to (S1→S3) as far as the algorithms are concerned. All that matters is that somehow, the Q participant must sort the statements into these ordered groups.

Note also that although Q has traditionally used a two-level sorting process in which the participant initially divides all the statements into three groups, usually “agree,” “uncertain,” and “disagree.” This initial sort is unstructured, in that the participant can place every statement into any one of the groups before going on to the finer-grained ordered sort just described. The results of this initial coarse sort are not preserved in any fashion into the subsequent factor calculations.

Traditional paper methods of Q sorting give participants small pieces of paper with statements typed onto them. A grid layout is typically used to help the participant keep track of the numbers of statements that could go into each group along the sorting axis. Usually in fact a bell-shaped curve is used, with fewest statements at each pole of the axis and most statements in the central groups. However, it actually turns out that the morphology of this distribution is entirely unimportant in the subsequent analysis, and investigators can actually employ any structure of groups that they want. This seems counterintuitive, but studies by Steven Brown and others have confirmed this.

Constraints During Online Sorting

A big advantage of computerized Q tools generally and online Q systems specifically is that the system can direct the participant precisely into placing the correct number of statements in each location within the sort. The big disadvantage is that the limited size of computer screens forces unavoidable tradeoffs for how the sort is displayed.

Any time there are nontrivial numbers of statements, each of which is nontrivial in length, it becomes impossible to display every statement clearly and legibly at the same instant. Either some scrolling must be performed to move amongst the statements, or else the statements must be abbreviated or shrunk and then selectively “zoomed” when read.

Traditional paper methods solve this situation simply by using a bigger desk to lay out the pieces of paper, though at some point, even then the participant has to “scroll” by moving her head side to side around the desk and “zoom” by getting closer and further from the pieces of paper.

There simply is no free lunch here. Given a potentially unlimited amount of information and a limited amount of space within which to display and interact with it, some kind of compromise technique has to be utilized.

To Scroll Or Not to Scroll

Grid-based systems choose the “zoom” approach in which statements all statements are visible but are minimized — so that in essence none of them are visible in the sense that they can be clearly read. The participant must make a specific act — hover the cursor over the statement or click on it — to enlarge the statement so that it can be read. A participant cannot readily scan statements; each view requires an act. Nevertheless, because this approach literally follows the style used with paper sorts, it is frequently preferred by Q traditionalists.

Q-Assessor has chosen to rely on scrolling rather than zooming. This allows a participant to scan related statements easily while moving them into sort “bins” without having to manipulate them to read them. Furthermore, Q-Assessor uses a vertically-oriented layout because computer usability studies consistently show that users hate horizontal scrolling — which is sometimes used by grid Q systems when even the grid won’t fit into the available space. We think that this vertical-scrolling, grouped sorting interface provides the best user experience for accomplishing the cognitive task in Q — producing the grouped ordering of the statements.

What About Drag and Drop?

Q-Assessor’s Javascript code base was initially written way back in 1999, long before the current Javascript libraries that provide drag and drop effects were developed. Since then, we’ve considered changing to this type of interface, but we’ve found some technical limitations with existing Javascript libraries. Even though we presume that we could eventually find satisfactory solutions, we haven’t aggressively pushed in this direction because it’s not entirely clear that this metaphor is better. Instead, we’ve stuck with the button-based “Select the Statement” and “Move It Here” approach you can see in our demo.

Here’s why. Drag and drop requires a certain physical dexterity and precision of cursor manipulation that not all users may have. Further, it is somewhat challenging to communicate clearly to a user what can be dragged at any point and where it can go — and such clear instructions are critical to the Q technique. In contrast, Q-Assessor’s buttons state explicitly what can be manipulated and what will happen when it is. Participants are more explicitly guided through the Q sorting technique.

We continue to review this particular design issue. It may be that we’ll employ a drag and drop interface at some point — in conjunction with our existing vertically-scrolling design.

Validation Is Crucial

Designers of computerized Q implementations can build systems and then hope and assume that they work, or they can build systems and then rigorously evaluate them. Q-Assessor took the latter approach, and in fact we published the first peer-reviewed paper validating the equivalence of our online system to standard paper-based methods. It is noteworthy that our paper came out way back in 2000, yet to date, none of the other online systems have subjected themselves to similar evaluations over all this time.

We think that if anyone is choosing which online Q system to use based on evidence — and not personal preference or guesswork — then the choice is clear.

Enhanced Q-Assessor Demos

Initial feedback about the demo of Q-Assessor’s features we had setup was clear on one point: the analysis of random user data was, well, random — and we needed to fix this.

What we’d created was a study in which anyone visiting Q-Assessor could submit a Q-sort and then play with the accumulating data. However, since most people probably didn’t bother to read the statements — which were just random statements themselves — these data weren’t really worth analyzing. So, we heard loud and clear, this demo of the analytic capabilities didn’t really demo anything other than Q-Assessor’s ability to generate well-formatted numbers.

Fair enough; message heard, and here’s how we’ve fixed this. We split the demo into two parts:

A demo of the Q-sort and interview process itself that a participant performs while doing the study
A demo of the data analysis tools that an administrative Q-Assessor user (aka investigator) uses to explore the data.

These two demos use two different demo studies here at Q-Assessor. The first one collects user data but doesn’t bother to analyze it, since there’s nothing worth analyzing there. The second one uses published responses from the “real” published study — but current users cannot pollute these data with new Q-sort responses, so the analysis is “real” and meaningful.

We’ve added a new demos page that introduces and links to these demos. Check there for more information about these demos, and have a run through them!

Welcome to Members of the Q Method Listserv!

Q-Assessor has broken out of relative stealth to announce its existence to the Q Methodology Listserv. This active group of primarily academic Q scholars has been in existence since 1996, and its archives provide a wealth of information and insight into all things Q.

Already a number of the Q list have signed-up, begun to experiment with Q-Assessor’s tools, and provided interesting and highly useful feedback. Thank you all!

Over time, we’ll address various ideas and suggestions we receive from users here in this space — discussing which innovations make the most sense, what issues others pose, and thinking aloud as to priorities and preferences.

Q-Assessor intends to provide Q investigators with the optimal systems and services they need to do Q on a big scale. We depend on you to let us know how we can best accomplish that! Thank you again!

A Brief History of Q-Assessor

A fitting first topic for these occasional posts is a bit of historical background about how Q-Assessor came to be.

Way back during the summer of 1999, Bryan Reber was working on his PhD dissertation at the University of Missouri School of Journalism. Part of his research utilized Q Methodology, a very popular approach at Mizzou. He happened to describe Q to me at some point, highlighting the multiple, arcane steps involved, the standard need for in-person interviews, and the fact that he needed to study far more and varied people than he could readily reach given his time and budget constraints.

I had been working on a variety of web-based data collection and management projects at that point, so Q seemed to be an ideal candidate for deploying over the Internet. The big implementation challenge, though, was the high degree of interactivity that the Q-sort process requires.

You have to remember that in those ancient times, web pages were pretty much static affairs. Web pages had lots of links and forms, of course, but these merely fetched an entirely new page as a whole with perhaps a back-end database action en route. Manipulating multiple elements within a page simply wasn’t readily accomplished.

That’s not to say it was impossible. A fair bit of preliminary work was going on in the mid-1990s. Java applets were for a while going to provide desktop-like apps within browsers, but they died because they remained too big and bulky for users routinely to download just to hit a page. Microsoft’s ActiveX components were supposed to function similarly, but they were tied to the Windows operating system and posed all sorts of security issues.

But the current crop of AJAX Javascript libraries — notably Prototype and now JQuery (which is what all the Kool Kids use these days) — didn’t come into existence until the mid-200s. Still, in 1999, JavaScript was clearly the tool to use, though, because that has all along been the main scripting language in essentially all browsers.

The somewhat hackish solution that presented itself that summer was to use Javascript to rewrite different HTML frames within a browser page. Frames were developed to allow web pages to be assembled from other web pages. But it was also possible to “write” web pages on the fly with Javascript code triggered by, say, a button click. This allowed a user action in one part of the page to update and redraw another piece of the page. For Q, this was just the ticket.

The original version of Q-Assessor had no back-end machinery. The content of the Q study (the statements, etc) were hard-coded into the page along with the code that implemented the sort interactions. When the sort was done, the results were simply emailed to the investigator. This solution was crude at best, but it still represented a huge leap, as it opened the possibility of doing Q at a distance.

Bryan did a validation study comparing Q-Assessor to the standard in-person technique and found that the new online approach was well-accepted and produced equivalent results, and we published this study in Operant Subjectivity, the premier Q journal. Bryan finished his dissertation, and we wondered whether there was any larger future for Q-Assessor.

A couple years later, I was working on database-backed collaborative websites using OpenACS — one of the first, if not the first, community web application frameworks. Mostly for amusement, I ported Q-Assessor into the Epimetrics web site as a demo. The Q-sort mechanism itself was unchanged, but the much more robust web application capabilities of that site enabled me to expand Q-Assessor’s functionality to include enrollment of participants, online authoring of studies, and database storage of results.

During the last two years, I’ve moved on to Ruby on Rails as my favored development environment. At the point the Epimetrics Group web site was redeveloped, I decided not to port the Q-Assessor demo over. However, there were enough academic inquiries — and more importantly an adequate level of interest in commercial realms — that it seemed propitious to rebirth Q-Assessor as its own entity.

This new iteration of Q-Assessor is still a work in progress. One of the most important developments is the addition of the analytic procedures. Q-Assessor now has a Ruby port of the public domain FORTRAN code from PQMethod that is what nearly all Q studies use to calculate and report factors. Going forward, I expect a number of further enhancements, based on feedback so far:

A new drag-and-drop interface for the Q-sort process, probably using JQuery
Animated visualization of manual rotations, probably using HTML5 and CSS3
Use of AJAX for many of the actions that currently cause entire page fetches
Internationalization of the Q-sort instructions and headers
A major cosmetic redesign of the public portions of the Q-Assessor web site