Using weights

In ideal circumstances, survey data reflects the population it is meant to represent. In the event that it does not, weighting is a technique that can be used to correct differences between sample demographics and known populations when performing data analysis. IntelliSurvey makes applying this technique very simple. Users need only to specify the fields upon which weights will be based and enter the desired percentages for each field. Throughout the reporting tool, these weights may be applied to display data.

Whether to use data weighting is something that really should be considered in advance, during the study design. IntelliSurvey has the tools to utilize data weighting, but weighting often imposes design challenges.

Common design challenges include:

• Does the client have population targets?
• Does the survey ask questions that exactly match the structure of these targets?
• Did the study designers request quotas that ensured that the sample would roughly map to population targets?

If you have the least bit of doubt, don't be surprised! Weighting, while simple in principle, can be very delicate in practice.

More information about weighting can be found here: http://www.applied-survey-methods.com/weight.html.

Overview

To access the Weights applet, click Analytics > Weights button from the Survey Navigation menu. 

This will open the Survey Weighting applet where a table of any previously created weights will be listed. To activate the Actions menu, click on a weight in the grid first. To view the details of a weight, double click on it, or select Results from the Actions menu once the weight is highlighted in the grid (see below for more information on Results).

The following columns are included in the Browse grid.

• Weight ID - The name given to the weight when created.
• Weight Desc - A brief description of the weight. By default, the system will use the question or variable's text for this field.
• Weighted Fields - The field(s) the weight has been applied to.
• Datacut - The datacut selected for use when creating the weight. System-defined options of All records and All completed records are also available.
• Min Collar - The minimum weight value that may be assigned to any of the data for the weighted field (optional). Usually mins and maxes are applied to alter the scores for demographic balance; commonly, 0.3 is used for the min.
• Max Collar - The maximum weight value that may be assigned to any of the data for the weighted field (optional). Usually mins and maxes are applied to alter the scores for demographic balance; commonly, 3.0 is used for the max.
• Rescaled - An indication of whether weights have been scaled to maintain a total number of respondents. If min and max weights were not applied, this will read N/A. If min and max weights were applied, this will read either true or false, depending on whether the weights were scaled or not.
• Uploaded - Indicates whether the weight was created/edited by upload (true) or not (false).

Creating weights

From the Weighting applet's Browse screen, select Create from the menu. Users may elect to create the weight within the platform (Build New) or to create a weight outside the platform and upload it (Upload From File).

Note: The Dev/Pub status does play a role in a user's ability to create weights for a survey. If a survey has never been published, the Create button will be disabled. Furthermore, the Weighting applet only uses published fields, therefore users should publish the latest version of a survey before creating any weights to ensure that all current fields are available, and that removed or edited fields are not accidentally being used.

Building weights within the application

First, select the field(s) from which the weights are to be generated from the Weighting dropdown menu. Then select which records to use from the Record Selector dropdown menu. Any user-created or system-created data cut filters will be included in this menu, including All records and All completed records.

If a new data cut is needed, press the Wrench icon and the Expression Builder modal will appear. Select the field(s) and responses necessary to define the data cut, pressing Apply after each field is selected. After the filter has been defined, an optional description may be added in the text field at the bottom of the modal. Press Done to save the filter and it will be available in the Record Selector pulldown menu for future usage.

After selecting the desired field(s) and record filter, click Next to continue.

 

On the next form, enter a short name for the weights in the Weights Description field (optional) and specify the target percentages by editing the values in the Target % column. The system will provide a default description if one is not entered, using the text associated with the question(s) or variable(s). Often the percents used for the Target % will be based off of census results. Any percentages that are edited will display a red flag in the cell's upper left hand corner. 

Users may also specify collars to apply to the weight. Minimum and Maximum may be added as factors for weighting different responses or demographics, and if need be, users may also opt to scale the weights used to maintain the total number of respondents.

Clicking Next will calculate the weight values. A summary of the results will be displayed on the next screen.

After weights have been created, they can be recalculated (to include new data if it is available) or edited by selecting the weight in the dropdown list on the Weighting/Browse page and then clicking Recalc or Edit. Weights may also be downloaded and uploaded.

Uploading custom weights calculated elsewhere

If weights have been calculated elsewhere, select Upload From File from the Create menu. The system will prompt for a file to be uploaded. A name or description that will help identify the weight should be provided in the Weight Description text field.

Any uploaded files should contain an 'id' column, and a weight column. The 'id' column must contain the record IDs of respondents in the data set. The weight column must be labeled with a field name which identifies the weight and cannot have spaces in it (e.g., 'WEIGHT1'). The data in this column should contain the corresponding weight values for each respondent ID in the adjacent column.

After selecting the file, click Next to continue.

The next screen will verify whether the data was successfully updated. The number of new, existing, and missing completed records will be listed. Press Confirm Add Data to apply the weighting.

Applying weights in a report

Once created, weights may be utilized in both Topline and Crosstab Reports.

Topline reports

Weights may be applied as a filter in the Record Selector in the Topline reports applet. To apply a weight, select Weights in the Record Selector. This will activate the Weights dropdown menu. Select the weights you wish to use, then click Apply to implement the selected weights for all tiles in the Topline Report.

Like other Record Selector options, weights may be combined with additional filter options.

Crosstab reports

Weights may also be applied when creating a Crosstab report under the Report Configuration settings. By default, the Weights field is set to Unweighted, but any existing weights will be listed in the dropdown menu. This field will be grayed out/unavailable if there are no weights in the data set.

When weights are applied to a Crosstab report, both the weighted and unweighted base sizes will be displayed.

Actions

The following options are available from the Weighting/Browse applet Actions menu.

Downloading weights

The Download option allows users to export an Excel file of the data a specific weight has been applied to. Selecting this option brings the user to the Weighting applet's Download screen in which the platform creates a file for the user to download. When the file has been created, users will see a message letting them know the file is ready. Click Download file to download the details of the weight and fields to which it was applied.

Uploading weights

Uploading weights allows users to overwrite existing data for a set of system-generated weights. To upload, select the weighting field from the list in the Weights/Browse grid to replace and click Upload.

The system prompts the user to select a file and enter a Weight Description in the text field provided.

The uploaded file should be an Excel, tab-separated text, or CSV file containing two columns: an 'id' column (referring to respondent record ID) and the relevant weight field (e.g., WEIGHT1). The values in the weighting field should be any non-negative value, including '0'. Select Next and IntelliSurvey will process the file.

When satisfied, click on Confirm Add Data and the new values will be merged into the existing field. The weighting app will show a value of "true" in the Uploaded column for any weights that are uploaded.

Viewing results

To view information about weights that have already been calculated, click on the weight you wish to view and select Results from the Actions menu. This option brings users to the Survey Weighting/Result screen where a table of the Variables and Key Weighting Stats is displayed for the chosen weight.

Recalculate weights

The Recalc option allows users to update the selected weight's values. Selecting this option will bring the user to the Weighting applet's Recalculate screen where they will be shown the results from reapplying the weight to the data set.

Edit weights

To edit an existing weight, click on it, then select Edit from the Actions menu. Users will be taken to the Weighting applet's Edit screen where they can edit the Weighting field and/or which data cut to use in the Record Selector. After the necessary changes are made, press Next to continue.

On the next screen, edit any Target % values and the Weights Description as if you were creating a new weight. Apply or edit any necessary collars then press Next to continue.

 

A final summary of the results of the edited weight will be displayed, including the Targeted %, Weighted %, and Min(s) and Max(s) that have been calculated.

Delete weights

To remove a saved weight from the survey, select Delete from the Actions menu. A confirmation pop-up will appear to confirm the weight's deletion.