How the AA surveys Members

We use this at the AA to understand the views of our Members on transport issues and be a voice for them to Government and transport decision makers.

Mathematical formulas have been developed over time determining how large a sample has to be to accurately reflect the wider population to within a certain level of confidence (usually that there is only a one in twenty chance of error).

Those samples can be surprisingly small, which is why political polls quoted in the news are usually based on less than one thousand responses. The AA's own experience over many years of surveying has confirmed this, with proportions changing little once response numbers go beyond 1000, which is in line with the sample sizes we generally seek.

Of course the math only works if the sample of people being surveyed is reasonably random and reflective of the overall population. In general, the bigger the sample the more accurate it will be but it is also vital that the method of drawing the sample doesn’t distort proportions by selecting for any particular quality.

This page has information on:

The AA Membership population
Member Voices Panel
Other survey methods and regional surveys
Distribution system
Question writing
Analysing the responses
Levels of support

The AA Membership population

With more than 1 million personal Members, the AA’s membership is large and broad.

Naturally, AA Members are drivers but they also include users of all other forms of transport.

This means we have Members from right across society but they are not an exact representation of the New Zealand general public.

Comparing our Members to the wider population, they are generally older (about 8 years on average), less likely to live in areas of high deprivation and we have slightly more female than male Members.

AA Members will commonly be middle class, established, somewhat conservative about risk, environmentally conscientious, and consider themselves safe, sensible and reasonable people.

That said, this does not mean they are all the same. Our Members' opinions on issues will still vary widely and this is what our survey work seeks to capture and understand.

Member Voices Panel

The AA Member Voices Panel is a group of up to 10,000 Members who volunteer to take one or two short surveys a month in return for entering a monthly draw for eight $100 Prezzy Cards.

The panel is recruited from the AA Membership and has a range of genders and ages that accurately represents the demographics of AA Members overall.

The panel is large enough to provide insights into not just the views of AA Members overall but also specific groups within it, like whether there are different perspectives in different parts of the country or among different age groups.

Terms and conditions for the Member Voices Panel

Other survey methods and regional surveys

Outside of the Member Voices Panel, Members can be invited to participate in regional or local surveys based on where they live. Our system randomly selects the email addresses to receive a survey and because the AA’s Membership is large we are able to cast a wide net for these random samples.

At 1000 random responses, the margin of error for a survey is about +/- 3%.  

Only one response is possible per email invitation sent out, to prevent an individual from answering a survey multiple times.

On some issues we will not use a random sample approach and instead give all Members in a specific area an equal opportunity to express their view. This is normally done for local issues affecting just one place and/or where the total number of Members in the survey area is small.

In these cases we send invitations to every Member in a certain district offering them the same opportunity to have their views recorded. As previously mentioned, every Member sent the survey will not necessarily see the email or choose to take part but the numbers we get in response are the best measure possible to base a district view on.

Distribution system

We sometimes hear from Members asking why they have never been surveyed. Like many organisations, the AA faces challenges to get our surveys or other communications to people.

Email surveys are very efficient but, like any survey method, there are limitations. While we can track the emails we send we cannot tell whether they go to old or unused email addresses, or if they appear in a recipient's main inbox or get redirected to a sub-folder that makes them less likely to be seen.

Automatic spam classifiers on people's email systems can also prevent our survey invitations from being seen or if Members have joined the NZ Direct Marketing Association's 'do not send' list then the AA will not send anything to that person.

So we know not every survey invite we send out will be seen and obviously not all Members that see it will choose to take part. This means that there is a further random element in whether the Members sent surveys decide to complete it.  

How to confirm we have your correct email details in our survey database

Question writing

Wherever possible, the AA will use questions that are the same as, or very similar to, questions used by other agencies that publish survey results. This means that any differences in response are because of the people answering, not the questions. It often illustrates however that AA Members do not differ much to the rest of the population - probably because a large proportion of the population are in fact AA Members. 

The question writing process is a lot more difficult than many people realise. Ensuring the questions we are asking are neutral and unbiased is always a focus of our survey development process and we go through multiple stages of internal testing for this.

Questions are reviewed so we can be confident they are not leading and where topics are especially sensitive we will sometimes employ an independent consultant to review the questions.

Because the AA has such a large and broad Membership we are used to responses from a wide range of perspectives and try to anticipate these in our questions. 

Sometimes survey results can be surprising because it is natural for people to assume that most others think or do the same things they do. The reality can sometimes be quite different. For example, many of the people  commuting to work every day would likely be surprised to find out that they are a minority within the population overall.

Analysing the responses

When the survey reaches a suitable level of response we gather in the data and analyse the results in a range of ways.

We sense check against other surveys or data sources and also see if there are demographic qualities that make a difference to the responses. Respondents' gender and age frequently makes the most difference - reflecting the fact that different groups can see the world from different perspectives.

If the proportions of these demographic qualities do not match the AA’s Membership we use a mathematical weighting process so that any over-represented group's views are scaled back, and under-represented views scaled up in proportion with the AA Membership's demographics. This means the survey results on the substantive questions end up reflective of what we would have got if the demographic proportions were in line with the AA's overall Membership population.

Levels of support

Because our Membership is so large the AA expects to see conflicting results within our surveys. Just like within any group of people, there will always be a range of views.

Typically unless there is a two thirds majority for any view the AA will not claim the majority of its Membership supports any particular stance. It will instead speak of the Membership being split on that issue.