3.10. Validation Results

This section discusses damage and loss results from three comparative and validation studies with different emphases:

1. The Influence of Year Built (Zillow vs. NSI and Parametric Study) first explores the implications of utilizing year built based on two data resources (NSI and Zillow). As discussed in Asset Description, the Zillow-based model tends to predict higher year predictions (more recent constructions) than the NSI-based model, especially for the southern edge on the city. This is reflected by the fact that Zillow-based damage and loss in that region is lower than the NSI-based results. Following the parametric study in Verification of Rulesets, the observed trend in damage and loss with respect to year built is discussed.

2. The Validation: StEER contrasts the testbeds damage estimates versus the visually-assigned damage states from StEER members and explores influence of uncertainty in building attributes and peak wind speed. Several notable trends between building damage and building attribute type will be discussed and reasoned.

3. The Validation: FEMA Historical Geospatial Damage Assessment focuses on validating the testbed damage estimates against the FEMA Historical Damage Assessment Database (535 records associated with this testbed). The testbed results are found to slightly overestimate the damage for the undamaged state (DS-0) which suggests that future investigations are still needed given uncertainties around the methodology used to generate the FEMA data.

3.10.1. Influence of Year Built (Zillow vs. NSI and Parametric Study)

The year built is a critical attribute as it triggers attribute assignments by various rulesets. As discussed in Asset Description, different sources of data were used to predict the YearBuilt engaging both the National Structural Inventory (NSI) (with SURF correcting some misalignments and gaps) and Zillow, which also provides the year built information for many of the residual buildings in the studied region. The implications of using different sources of year built information to project damage is explored in Fig. 3.10.1.1, noting the contrast map that depicts the difference between the damage state projected by SURF-NSI and that projected by Zillow. Note that the difference is not prominent in downtown Lake Charles, although divergence is noted southwest of the city where the Zillow set predicts relatively lower damage states. Compared in Fig. 3.10.1.2 are the the CDF distributions of expected damage states (NSI vs. Zillow) as well as the probability of exceeding DS-2 and DS-3. Fig. 3.10.1.3 plots the cumulative distributions of expected damage states for four different year built bins: Pre 1960, 1960-1979, 1980-1999, and Post 2000. In general, we see improved building performance (low damage states) as code revisions by years as recalling the parametric study in Verification of Rulesets, the dominate building type is WSF1 (Fig. 3.1.4.3) whose wind-induced loss is particularly sensitive to nail spacing requirements varied by years.

Fig. 3.10.1.1 Comparison of expected damage states (NSI vs. Zillow).

Fig. 3.10.1.2 CDFs of expected DS and probability of exceeding DS-2 and DS-3.

Fig. 3.10.1.3 CDFs of expected DS by year bins (NSI results).

3.10.2. Validation: StEER

For Hurricane Laura, the Structural Extreme Reconnaissance (StEER) Network has released a dataset of the visual inspection and damage estimation for sample buildings in Lake Charles ([Roueche21], refereed as StEER buildings). There are 99 StEER buildings included in this testbed, which offers an opportunity to validate the workflow. In the meantime, the relatively small sample size also offers the convenience of investigating the influence of different modeling parameters on the estimated damage states and relating them to corresponding field observations to help understand the dominant trends in the data. As will be illustrated, some of these trends validate the effectiveness of the methodology and others indicate potential measures to improve the wind resistance.

Before presenting results, it is important to note, as discussed in StEER’s report on Hurricane Laura ([Roueche21]), that StEER’s assessments for Laura were conducted virtually from streetview imagery due to COVID-19 restrictions on travel; thus this “ground truth” is less reliable in discerning some aspects of the damage than a traditional StEER mission with in-person, up-close forensic assessments. Also note that since rulesets from New Jersey are applied to assign attributes and are not consistent with the historical regulatory environment in Louisiana, the predicted damage states are also likely to be lower, particularly for construction older than 2006 (the year Louisiana first implemented its statewide building codes linked to IBC/IRC). Finally, as also discussed in StEER’s report ([Roueche21]), there were low rates of compliance with shuttering requirements in the affected area (based on observations immediately after landfall). Thus it is likely that the shutter attributes are assigned in this workflow at a rate higher than actually observed in the event, leading to lower rates of predicted damage than those observed in the field.

Fig. 3.10.2.1 plots the estimated damage states against key building attributes that are found to influence the resulting damage state under the hurricane wind hazard. Major observed trends are as follows:

1. In general, the damage state is negatively correlated to the year built. Buildings with built year after 2000 are found to have prominent improvement in the wind resistance and have lower damage states. This is reasoned by the authors to be led by the assumption in the ruleset that the post-2000 buildings would have shutter measures and stronger roof deck attachment. It is import to note, however, that this effect predicted based on New Jersey rulesets will not be observed in Louisiana until at least 2006 when statewise codes were enacted, and further, is likely to not be pronounced given the earlier observation about low rates of compliance with shuttering requirements.

2. The surface roughness, i.e., terrain feature, is found to be a key variable when considering the potential damage from the hurricane to the building. Even moving from open terrain to light suburban, following the damage functions in HAZUS, we see about 50% reduction in the average damage state. This is associated with the reduction in mean wind speed as surface roughness increases.

3. Shutters could significantly help reduce the potential wind-induced damage: average damage state is reduced from 2.2 to 1.2. This observation makes the low compliance noted by StEER particularly noteworthy.

4. Garage type also would influence the building performance against the wind hazard: garages per SFBC 1994 are found to have much better behaviors than the standard and weak garages. This effect is not expected to be pronounced in Lake Charles given the low incidence of attached garages noted in the samples discussed in Asset Description.

5. As expected, roof deck attachment using a tighter nail spacing (6 in vs. 12 in) is found to perform much better.

6. As also expected, gable roof is found to have slightly worse performance than the hip and flat roofs.

Fig. 3.10.2.1 Influential building attributes on the wind-induced damage state.

It is noted that the trends observed in Fig. 3.10.2.1 are based on a so-called “median” model representing our best estimates of building attributes. However, in reality, because of the lack of high-resolution or multiple data resources, we may not have 100% confidence about certain building attributes or hazard fields, e.g., year built or peak wind speed at a specific site. Hence, the estimated “median” damage state from the “median” model may not give the full representation of the real damage potential. As illustrated in Fig. 3.10.2.2, if considering the uncertainty in the year built (instead of a deterministic value, assuming it follows a normal distribution with a standard deviation of 10 years), we could sample year built from the distribution and repeat the damage assessment by sufficient number of times to estimate the distribution of the damage state. This however assumes differences are attributable to uncertainties in the year built and neglects the fact that the predictors are biased by attribute assignments using rulesets that are derived from a difference regulatory environment (New Jersey) with a longer history of adopting and enforcing model building codes statewide. This fact may explain why the predicted damage states bias toward lower levels of damage (assuming less vulnerable structures than was the case in practice).

Fig. 3.10.2.2 Uncertainty in the estimated damage state.

Following this idea, for each of the 99 StEER buildings, we sampled 100 year built values from a distribution with the mean of the given year built value and a standard deviation of 10 years, and ran the damage assessment for each sampled case. Fig. 3.10.2.3 summarizes the comparison between the mean and 95-percentile of estimated damage state and the StEER damage state. Similar exercises (Fig. 3.10.2.4) are conducted for the peak wind speed where we sampled 100 PWS from a distribution with a standard deviation of 20 mph.

Fig. 3.10.2.3 Influence of uncertainty in year built on the estimated damage state.

Fig. 3.10.2.4 Influence of uncertainty in PWS on the estimated damage state.

3.10.3. Validation: FEMA Historical Geospatial Damage Assessment

FEMA and the Department of Interior lead the development of the Geospatial Platform, an internet-based service environment that provides a suite of well-manged geospatial data, services, applications, and tools. The FEMA Historical Damage Assessment Database is a repository of geospatial damage assessments from past National Disaster events where damage assessments were conducted either using high-resolution imagery or by means of geospatial modeling.

For Hurricane Laura, the damage categories of 112,571 buildings in the Louisiana were modeled from flood depths at the structure as characterized based on modeled wind, flood or surge data. Five damage categories are used: No Damage (NOD), Affected (AFF), Minor (MIN), Major (MAJ), and Destroyed (DES). Out of the 112,571 buildings, 535 buildings, located in the west bound of the Lake Charles city at the waterfront of the Prein Lake, are also investigated in this testbed. Table 3.10.3.1 is a subset of data obtained from FEMA Public Data. Table 3.10.3.2 is the building inventory for this subset. The full inventory and FEMA damage assessment results can be accessed by FEMA Damage Assessment of the 535 buildings and BIM inventory of the 535 buildings.

Fig. 3.10.3.1 FEMA Historical Geospatial Damage Assessment Database (2020 Hurricane Laura).

Table 3.10.3.1 FEMA Damage Assessment example buildings in Lake Charles.

ID	DMG_LEVEL	COUNTY	STATE	LONGITUDE	LATITUDE
1	DES	Calcasieu	LA	-93.29	30.17
2	NOD	Calcasieu	LA	-93.30	30.17
3	NOD	Calcasieu	LA	-93.29	30.17
4	NOD	Calcasieu	LA	-93.29	30.17
5	NOD	Calcasieu	LA	-93.28	30.16
6	NOD	Calcasieu	LA	-93.29	30.17
7	NOD	Calcasieu	LA	-93.29	30.17
8	NOD	Calcasieu	LA	-93.29	30.17
9	NOD	Calcasieu	LA	-93.30	30.17
10	AFF	Calcasieu	LA	-93.28	30.17
11	NOD	Calcasieu	LA	-93.29	30.16
12	NOD	Calcasieu	LA	-93.30	30.17
13	NOD	Calcasieu	LA	-93.29	30.17
14	NOD	Calcasieu	LA	-93.28	30.17
15	NOD	Calcasieu	LA	-93.28	30.17
16	NOD	Calcasieu	LA	-93.29	30.16
17	NOD	Calcasieu	LA	-93.29	30.17
18	NOD	Calcasieu	LA	-93.29	30.17
19	NOD	Calcasieu	LA	-93.29	30.17
20	AFF	Calcasieu	LA	-93.28	30.16

Table 3.10.3.2 BIM inventory of example buildings in this testbed overlapped with the FEMA Damage Assessment Database.

ID	RoofShape	FloodZone	Area	Longitude	Latitude	Terrain	DSWII	BuildingType	AvgJanTemp	Garage	NumberOfStories	YearBuilt	RoofSlope	MeanRoofHt	OccupancyClass
1	Gable	X	112	-93.29	30.18	51	132	Wood	Above	0	1	1994	0	12	RES3
2	Hip	X	148	-93.30	30.17	42	132	Wood	Above	0	1	1998	0.06	13.1	RES3
3	Gable	X	65	-93.29	30.17	11	132	Wood	Above	1	1	1994	0.08	13.5	RES3
4	Gable	X	110	-93.29	30.18	52	132	Wood	Above	0	1	1995	0.05	12.9	RES3
5	Hip	X	581	-93.29	30.17	11	132	Wood	Above	0	1	1996	0.05	17.3	RES1
6	Gable	X	83	-93.29	30.17	11	132	Wood	Above	0	1	1999	0.17	15.1	RES1
7	Hip	X	148	-93.30	30.17	42	132	Wood	Above	0	1	1998	0.06	13.1	RES3
8	Gable	X	55	-93.28	30.17	11	132	Wood	Above	0	1	1997	0.16	14.9	RES3
9	Hip	X	487	-93.29	30.16	11	132	Wood	Above	1	1	1996	0.07	13.2	RES1
10	Hip	X	107	-93.30	30.17	42	132	Wood	Above	0	1	1998	0	12	RES3
11	Gable	X	423	-93.29	30.17	11	132	Wood	Above	0	1	1998	0.13	11.3	RES1
12	Flat	X	144	-93.28	30.17	52	132	Wood	Above	1	1	1995	0.2	12	RES3
13	Gable	X	101	-93.29	30.17	42	132	Wood	Above	1	2	1997	0.24	16.4	RES1
14	Gable	X	86	-93.29	30.17	11	132	Wood	Above	0	1	1998	0.51	17.2	RES1
15	Hip	X	797	-93.29	30.17	51	132	Wood	Above	1	1	1995	0.04	12.8	RES3
16	Flat	X	90	-93.30	30.17	11	132	Wood	Above	0	2	1999	0.65	33.3	RES1
17	Gable	X	129	-93.29	30.17	11	132	Wood	Above	1	1	1997	0.24	16.4	RES3
18	Gable	X	704	-93.29	30.17	42	132	Wood	Above	1	2	1998	0.07	13.3	RES1
19	Gable	X	112	-93.29	30.17	11	132	Wood	Above	0	1	1998	0.11	16.3	RES3
20	Gable	X	137	-93.28	30.17	11	132	Wood	Above	0	1	1997	0	12	RES3

Fig. 3.10.3.2 shows the damage categories collected from the FEMA Damage Assessment Database, where most buildings are assessed as having No Damage or Affected. Also noted from Table 3.10.3.1 is that all these buildings are subject to CAT-4 gust wind speed, which is consistent with our input wind field where the 3-s gust peak wind speed is about 130 \(mph\) (please see Hazard Characterization).

To compare these damage categories with the estimated damage states by the workflow in this testbed, we assume that the five damage categories are mapped to the five damage states in the HAZUS wind damage functions, assuming the inundation does not cause prominent flood-induced damage to the buildings. While this assumption will be the subject of future validation, Fig. 3.10.3.3 summarizes (1) the expected damage states of the 535 buildings, (2) the spatial distribution of difference between the FEMA Damage Categories and estimated damage states, and (3) the statistical distribution of the difference between the two. Given the FEMA set is based on a discrete (category-based) description while the testbed set is based on probabilistic estimates, the two results have fairly good agreement except the FEMA set has buildings predicted as no damage (DS-0). This over-prediction of DS-0 is also seen in StEER set results (Fig. 3.10.2.3 and Fig. 3.10.2.4). The uncertainty in the year built and wind speed, to some extent, can help explain this over-prediction, as can the fact that the rulesets applied are translated directly from New Jersey. It is further important to note that FEMA assessments also consider the effects of storm surge. Future investigations are still needed to consider other possible factors causing this mismatch in predicted DS-0.

Fig. 3.10.3.2 FEMA Damage Category in the Geospatial Damage Assessment Database (for the 535 buildings).

Fig. 3.10.3.3 Comparison: FEMA Damage Category vs. expected damage state by the workflow (for the 535 buildings).

Roueche21(1,2,3)

Roueche, D. Kameshwar, S. Vorce, M. Kijewski-Correa, T. Marshall, J. Mashrur, N. Ambrose, K. Brown, C. Childress, O. Fox, D. Morris, K. Rawajfih, H. Rodriguez, L. (2021) “Field Assessment Structural Teams: FAST-1, FAST-2, FAST-3”, in StEER - Hurricane Laura. DesignSafe-CI. https://doi.org/10.17603/ds2-dha4-g845