XR_ANDROID_light_estimation_cubemap

Name String

XR_ANDROID_light_estimation_cubemap

Extension Type

Instance extension

Registered Extension Number

722

Revision

1

Ratification Status

Not ratified

Extension and Version Dependencies

XR_ANDROID_light_estimation

Last Modified Date

2025-08-06

IP Status

No known IP claims.

Contributors

Salar Khan, Google
Scott Chung, Google
Jared Finder, Google
Spencer Quin, Google
Levana Chen, Google
Nihav Jain, Google
Jürgen Sturm, Google

Overview

This extension builds upon the basic XR_ANDROID_light_estimation extension. It adds support for getting cubemap lighting estimates, which provides more detailed estimates about the lighting in the physical environment.

Note

The mechanism for getting the light estimate data is the same as the basic extension, except that XrCubemapLightEstimatorCreateInfoANDROID has to be chained to the XrLightEstimatorCreateInfoANDROID when creating the light estimator handle.

Inspect system capability

typedef struct XrSystemCubemapLightEstimationPropertiesANDROID {
    XrStructureType    type;
    void*              next;
    XrBool32           supportsCubemapLightEstimation;
} XrSystemCubemapLightEstimationPropertiesANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
  • supportsCubemapLightEstimation is an XrBool32 , indicating if the current system supports cubemap light estimation.

An application can inspect whether the system is capable of supporting cubemap light estimation by extending XrSystemProperties with XrSystemCubemapLightEstimationPropertiesANDROID structure when calling xrGetSystemProperties .

If the runtime returns XR_FALSE for supportsCubemapLightEstimation and XrCubemapLightEstimatorCreateInfoANDROID has been chained to XrLightEstimatorCreateInfoANDROID , the runtime must return XR_ERROR_FEATURE_UNSUPPORTED from xrCreateLightEstimatorANDROID .

Valid Usage (Implicit)

Getting supported cubemap resolutions

XrResult xrEnumerateCubemapLightingResolutionsANDROID(
    XrInstance                                  instance,
    XrSystemId                                  systemId,
    uint32_t                                    resolutionCapacityInput,
    uint32_t*                                   resolutionCountOutput,
    uint32_t*                                   resolutions);

Parameter Descriptions

  • instance is an XrInstance created previously.
  • systemId is the XrSystemId retrieved previously by xrGetSystem for which to get the supported cubemap resolutions.
  • resolutionCapacityInput is a uint32_t indicating the maximum number of elements that can be stored in the resolutions array.
  • resolutionCountOutput is a pointer to a uint32_t which is set by the runtime indicating the number of elements written to the resolutions array by the runtime.
  • resolutions is an array of uint32_t which is populated by the runtime with the supported cubemap resolutions.

A cubemap resolution indicates the width and height of each face of the cubemap in pixels. 2-call idiom The application can then choose to use one of the supported resolutions in XrCubemapLightEstimatorCreateInfoANDROID :: cubemapResolution when creating the light estimator handle. The application must allocate the appropriate amount of memory for the image buffer members of XrCubemapLightingDataANDROID based on the resolution chosen and the color format.

Valid Usage (Implicit)

  • The XR_ANDROID_light_estimation_cubemap extension must be enabled prior to calling xrEnumerateCubemapLightingResolutionsANDROID
  • instance must be a valid XrInstance handle
  • resolutionCountOutput must be a pointer to a uint32_t value
  • If resolutionCapacityInput is not 0 , resolutions must be a pointer to an array of resolutionCapacityInput uint32_t values

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SIZE_INSUFFICIENT
  • XR_ERROR_SYSTEM_INVALID
  • XR_ERROR_VALIDATION_FAILURE

Getting supported cubemap color formats

The XrCubemapLightingColorFormatANDROID enumeration identifies to the runtime, the color format of cubemap lighting to use.

typedef enum XrCubemapLightingColorFormatANDROID {
    XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R32G32B32_SFLOAT_ANDROID = 1,
    XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R32G32B32A32_SFLOAT_ANDROID = 2,
    XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R16G16B16A16_SFLOAT_ANDROID = 3,
    XR_CUBEMAP_LIGHTING_COLOR_FORMAT_MAX_ENUM_ANDROID = 0x7FFFFFFF
} XrCubemapLightingColorFormatANDROID;

The enums have the following meanings:

Enum Description

XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R32G32B32_SFLOAT_ANDROID

A color format with 3 channels where each channel is a 32-bit floating point value.

XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R32G32B32A32_SFLOAT_ANDROID

A color format with 4 channels where each channel is a 32-bit floating point value.

XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R16G16B16A16_SFLOAT_ANDROID

A color format with 4 channels where each channel is a 16-bit floating point value.

XrResult xrEnumerateCubemapLightingColorFormatsANDROID(
    XrInstance                                  instance,
    XrSystemId                                  systemId,
    uint32_t                                    colorFormatCapacityInput,
    uint32_t*                                   colorFormatCountOutput,
    XrCubemapLightingColorFormatANDROID*        colorFormats);

Parameter Descriptions

  • instance is an XrInstance created previously.
  • systemId is the XrSystemId retrieved previously by xrGetSystem for which to get the supported cubemap resolutions.
  • colorFormatCapacityInput is a uint32_t indicating the maximum number of elements that can be stored in the colorFormats array.
  • colorFormatCountOutput is a pointer to a uint32_t which is set by the runtime indicating the number of elements written to the colorFormats array by the runtime.
  • colorFormats is an array of XrCubemapLightingColorFormatANDROID which is populated by the runtime with the supported cubemap color formats.

2-call idiom The application can then choose to use one of the supported color formats in XrCubemapLightEstimatorCreateInfoANDROID :: colorFormat when creating the light estimator handle. The application must allocate the appropriate amount of memory for the image buffer members of XrCubemapLightingDataANDROID based on the color format chosen.

Valid Usage (Implicit)

Return Codes

Success

  • XR_SUCCESS

Failure

  • XR_ERROR_FUNCTION_UNSUPPORTED
  • XR_ERROR_HANDLE_INVALID
  • XR_ERROR_INSTANCE_LOST
  • XR_ERROR_RUNTIME_FAILURE
  • XR_ERROR_SIZE_INSUFFICIENT
  • XR_ERROR_SYSTEM_INVALID
  • XR_ERROR_VALIDATION_FAILURE

Create a cubemap light estimator handle

typedef struct XrCubemapLightEstimatorCreateInfoANDROID {
    XrStructureType                        type;
    const void*                            next;
    uint32_t                               cubemapResolution;
    XrCubemapLightingColorFormatANDROID    colorFormat;
    XrBool32                               reproject;
} XrCubemapLightEstimatorCreateInfoANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain.
  • cubemapResolution is a uint32_t indicating the resolution of the cubemap lighting to use.
  • colorFormat is an XrCubemapLightingColorFormatANDROID indicating the color format of the cubemap lighting data to use.
  • reproject is an XrBool32 indicating whether the cubemap lighting should be reprojected to the application base space.

The XrCubemapLightEstimatorCreateInfoANDROID structure describes the information to create an XrLightEstimatorANDROID handle to be capable of providing cubemap lighting estimates. The XrCubemapLightEstimatorCreateInfoANDROID :: cubemapResolution member must be set to one of the resolutions returned by xrEnumerateCubemapLightingResolutionsANDROID . The XrCubemapLightEstimatorCreateInfoANDROID :: colorFormat member must be set to one of the color formats returned by xrEnumerateCubemapLightingColorFormatsANDROID . If the application does not set to the resolution to one of the supported resolutions or the color format to one of the supported color formats, the runtime must return XR_ERROR_FEATURE_UNSUPPORTED from xrCreateLightEstimatorANDROID .

Valid Usage (Implicit)

Cubemap light estimates

typedef struct XrCubemapLightingDataANDROID {
    XrStructureType                type;
    void*                          next;
    XrLightEstimateStateANDROID    state;
    uint32_t                       imageBufferSize;
    uint8_t*                       imageBufferRight;
    uint8_t*                       imageBufferLeft;
    uint8_t*                       imageBufferTop;
    uint8_t*                       imageBufferBottom;
    uint8_t*                       imageBufferFront;
    uint8_t*                       imageBufferBack;
    XrQuaternionf                  rotation;
    XrTime                         centerExposureTime;
} XrCubemapLightingDataANDROID;

Member Descriptions

  • type is the XrStructureType of this structure.
  • next is NULL or a pointer to the next structure in a structure chain. Valid structures are XrAmbientLightANDROID , XrSphericalHarmonicsANDROID , XrDirectionalLightANDROID .
  • state is the XrLightEstimateStateANDROID representing the state of the light estimate.
  • imageBufferSize is a uint32_t indicating the byte size of each face image buffer in the cubemap.
  • imageBufferRight is a uint8_t buffer containing the right face image of the cubemap.
  • imageBufferLeft is a uint8_t buffer containing the left face image of the cubemap.
  • imageBufferTop is a uint8_t buffer containing the top face image of the cubemap.
  • imageBufferBottom is a uint8_t buffer containing the bottom face image of the cubemap.
  • imageBufferFront is a uint8_t buffer containing the front face image of the cubemap.
  • imageBufferBack is a uint8_t buffer containing the back face image of the cubemap.
  • rotation is an XrQuaternionf indicating the rotation of the cubemap.
  • centerExposureTime is an XrTime indicating the time the cubemap was captured.

This structure can be chained to the XrLightEstimateANDROID . The runtime must only populate this structure in xrGetLightEstimateANDROID if the XrCubemapLightEstimatorCreateInfoANDROID was used to create the light estimator handle. The application must allocate the appropriate amount of memory for the image buffers which is dependent on the values set in XrCubemapLightEstimatorCreateInfoANDROID :: cubemapResolution and XrCubemapLightEstimatorCreateInfoANDROID :: colorFormat when creating the light estimator handle. The application must set XrCubemapLightingDataANDROID :: imageBufferSize to the capacity of each face image buffer in bytes. If the application is not using cubemap light estimation or if XrCubemapLightingDataANDROID :: imageBufferSize is not large enough for the runtime to populate the image buffers, the runtime must set XrCubemapLightingDataANDROID :: state to XR_LIGHT_ESTIMATE_STATE_INVALID_ANDROID .

If the application set XrCubemapLightEstimatorCreateInfoANDROID :: reproject to XR_TRUE when creating the light estimator handle, the runtime must set XrCubemapLightingDataANDROID :: rotation to the identity rotation and ensure the internal rotated cubemap is reprojected onto the faces of an identity cubemap in the application base space.

The layout of the lighting cubemap is same as OpenGL cubemap layout, as shown in the following image

XR ANDROID light estimation cubemap layout

Figure 24. Cubemap layout.

Valid Usage (Implicit)

  • The XR_ANDROID_light_estimation_cubemap extension must be enabled prior to using XrCubemapLightingDataANDROID
  • type must be XR_TYPE_CUBEMAP_LIGHTING_DATA_ANDROID
  • next must be NULL or a valid pointer to the next structure in a structure chain
  • state must be a valid XrLightEstimateStateANDROID value
  • imageBufferRight must be a pointer to an array of imageBufferSize uint8_t values
  • imageBufferLeft must be a pointer to an array of imageBufferSize uint8_t values
  • imageBufferTop must be a pointer to an array of imageBufferSize uint8_t values
  • imageBufferBottom must be a pointer to an array of imageBufferSize uint8_t values
  • imageBufferFront must be a pointer to an array of imageBufferSize uint8_t values
  • imageBufferBack must be a pointer to an array of imageBufferSize uint8_t values
  • The imageBufferSize parameter must be greater than 0

Example code for light estimation

The following example code demonstrates how to get all possible light estimation quantities from the runtime

XrSession session;  // Created at app startup
XrInstance instance; // Created at app startup
XrSpace appSpace;   // Created previously.
XrSystemId systemId; // Retrieved previously by xrGetSystem
PFN_xrCreateLightEstimatorANDROID xrCreateLightEstimatorANDROID; // Created previously.
PFN_xrDestroyLightEstimatorANDROID xrDestroyLightEstimatorANDROID; // Created previously.
PFN_xrGetLightEstimateANDROID xrGetLightEstimateANDROID; // Created previously.
PFN_xrEnumerateCubemapLightingResolutionsANDROID xrEnumerateCubemapLightingResolutionsANDROID; // Created previously.
PFN_xrEnumerateCubemapLightingColorFormatsANDROID xrEnumerateCubemapLightingColorFormatsANDROID; // Created previously.

XrSystemCubemapLightEstimationPropertiesANDROID props = {
  .type = XR_TYPE_SYSTEM_CUBEMAP_LIGHT_ESTIMATION_PROPERTIES_ANDROID};
XrSystemProperties base = {.type = XR_TYPE_SYSTEM_PROPERTIES,
                           .next = &props};
CHK_XR(xrGetSystemProperties(instance, systemId, &base));
if (!props.supportsCubemapLightEstimation) {
   // Cubemap light estimation is not supported
}

uint32_t cubemapResolution = 0;
std::vector<uint32_t> supportedCubemapResolutions;
uint32_t resolutionCount;
CHK_XR(xrEnumerateCubemapLightingResolutionsANDROID(
  instance, systemId, 0, &resolutionCount, nullptr));
supportedCubemapResolutions.resize(resolutionCount);
if (resolutionCount == 0) {
  // No cubemap lighting supported
} else {
  CHK_XR(xrEnumerateCubemapLightingResolutionsANDROID(
    instance, systemId, 0, &resolutionCount, supportedCubemapResolutions.data()));
  cubemapResolution = supportedCubemapResolutions[0];
}

uint32_t pixelCount = cubemapResolution * cubemapResolution;

XrCubemapLightingColorFormatANDROID colorFormat;
std::vector<XrCubemapLightingColorFormatANDROID> supportedColorFormats;
uint32_t colorFormatCount;
CHK_XR(xrEnumerateCubemapLightingColorFormatsANDROID(
  instance, systemId, 0, &colorFormatCount, nullptr));
supportedColorFormats.resize(colorFormatCount);
if (colorFormatCount == 0) {
  // No supported color formats for cubemap lighting. Cannot use cubemap
  // light estimation.
} else {
  CHK_XR(xrEnumerateCubemapLightingColorFormatsANDROID(
    instance, systemId, 0, &colorFormatCount, supportedColorFormats.data()));
  colorFormat = supportedColorFormats[0];
}

uint32_t pixelSize = 0;
switch (colorFormat) {
  case XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R32G32B32_SFLOAT_ANDROID:
    pixelSize = 3 * sizeof(float);
    break;
  case XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R32G32B32A32_SFLOAT_ANDROID:
    pixelSize = 4 * sizeof(float);
    break;
  case XR_CUBEMAP_LIGHTING_COLOR_FORMAT_R16G16B16A16_SFLOAT_ANDROID:
    pixelSize = 4 * sizeof(uint16_t);
    break;
  default:
    // Should not happen since the color format was validated previously.
    break;
}

uint32_t perFaceImageBufferSize = pixelCount * pixelSize;

XrLightEstimatorANDROID estimator;
XrCubemapLightEstimatorCreateInfoANDROID cubemapCreateInfo = {
    .type = XR_TYPE_CUBEMAP_LIGHT_ESTIMATOR_CREATE_INFO_ANDROID,
    .cubemapResolution = cubemapResolution,
    .colorFormat = colorFormat,
    .reproject = XR_TRUE
};
XrLightEstimatorCreateInfoANDROID basicCreateInfo = {
    .type = XR_TYPE_LIGHT_ESTIMATOR_CREATE_INFO_ANDROID,
    .next = &cubemapCreateInfo};
CHK_XR(xrCreateLightEstimatorANDROID(session, &basicCreateInfo, &estimator));

std::vector<uint8_t> cubemapBuffer(perFaceImageBufferSize * 6); // 6 faces * perFaceImageBufferSize

// Every frame
XrTime updateTime;  // Time used for the current frame's simulation update.

XrLightEstimateGetInfoANDROID info = {
    .type = XR_TYPE_LIGHT_ESTIMATE_GET_INFO_ANDROID,
    .space = appSpace,
    .time = updateTime,
};

XrCubemapLightingDataANDROID cubemap = {
    .type = XR_TYPE_CUBEMAP_LIGHTING_DATA_ANDROID,
    .next = nullptr,
    .imageBufferSize = perFaceImageBufferSize,
    .imageBufferRight = cubemapBuffer.data() + 0 * perFaceImageBufferSize,
    .imageBufferLeft = cubemapBuffer.data() + 1 * perFaceImageBufferSize,
    .imageBufferTop = cubemapBuffer.data() + 2 * perFaceImageBufferSize,
    .imageBufferBottom = cubemapBuffer.data() + 3 * perFaceImageBufferSize,
    .imageBufferFront = cubemapBuffer.data() + 4 * perFaceImageBufferSize,
    .imageBufferBack = cubemapBuffer.data() + 5 * perFaceImageBufferSize,
};

XrDirectionalLightANDROID directionalLight = {
    .type = XR_TYPE_DIRECTIONAL_LIGHT_ANDROID,
    .next = &cubemap,
};

XrSphericalHarmonicsANDROID totalSh = {
    .type = XR_TYPE_SPHERICAL_HARMONICS_ANDROID,
    .next = &directionalLight,
    .kind = XR_SPHERICAL_HARMONICS_KIND_TOTAL_ANDROID,
};

XrSphericalHarmonicsANDROID ambientSh = {
    .type = XR_TYPE_SPHERICAL_HARMONICS_ANDROID,
    .next = &totalSh,
    .kind = XR_SPHERICAL_HARMONICS_KIND_AMBIENT_ANDROID,
};

XrAmbientLightANDROID ambientLight = {
    .type = XR_TYPE_AMBIENT_LIGHT_ANDROID,
    .next = &ambientSh,
};

XrLightEstimateANDROID estimate = {
    .type = XR_TYPE_LIGHT_ESTIMATE_ANDROID,
    .next = &ambientLight,
};

XrResult result = xrGetLightEstimateANDROID(estimator, &info, &estimate);
if (result == XR_SUCCESS &&
    estimate.state == XR_LIGHT_ESTIMATE_STATE_VALID_ANDROID) {
  // use cubemap, directionalLight, totalSh, ambientSh, and
  // ambientLight if each struct has a valid state field

  if (cubemap.state == XR_LIGHT_ESTIMATE_STATE_VALID_ANDROID) {
    // use cubemap
    if (cubemapCreateInfo.reproject == XR_TRUE) {
      XrQuaternionf identityQuaternion = {0.0f, 0.0f, 0.0f, 1.0f};
      assert(memcmp(&cubemap.rotation, &identityQuaternion, sizeof(XrQuaternionf)) == 0);
    }
  }
}

// When you want to disable light estimation
CHK_XR(xrDestroyLightEstimatorANDROID(estimator));

New Commands

New Structures

New Enums

New Enum Constants

  • XR_ANDROID_LIGHT_ESTIMATION_CUBEMAP_EXTENSION_NAME
  • XR_ANDROID_light_estimation_cubemap_SPEC_VERSION

  • Extending XrStructureType :

    • XR_TYPE_CUBEMAP_LIGHTING_DATA_ANDROID
    • XR_TYPE_CUBEMAP_LIGHT_ESTIMATOR_CREATE_INFO_ANDROID
    • XR_TYPE_SYSTEM_CUBEMAP_LIGHT_ESTIMATION_PROPERTIES_ANDROID

Issues

Version History

  • Revision 1, 2025-12-05 (Salar Khan)

    • Initial extension description